Guidelines for Storing Tables in Off-Heap Memory

This topic provides some guidelines on how and when to use off-heap memory feature.

Using off-heap memory is recommended for any deployment where you need to keep a large amount of data in-memory. If you are running servers with at least 150GB of RAM or you typically run JVMs with more than 32GB of user data, you may benefit from keeping your table data in off-heap memory.

You can use off-heap memory on either 32-bit or 64-bit architectures.

In addition, storing table data in off-heap memory may most benefit deployments that have rows that are either uniform in size or have a pattern of common sized values. Off-heap memory may not perform as well if your tables have drastically different and unpredictably sized row values, or if you have frequent table updates that change the size of row values. These conditions can cause fragmentation of off-heap storage space, which can lead to poor query performances.

Additional Guidelines for Using Off-Heap Storage

When using off-heap memory, keep in mind the following guidelines:
  • When using off-heap tables, GemFire XD only stores table rows in off-heap memory. Table rows include BLOB and CLOB data. Each table row stored in off-heap memory also consumes some heap memory and should be accounted for when calculating JVM heap overhead. See Estimating GemFire XD Heap Overhead and Table Memory Requirements for details.
  • GemFire XD also requires off-heap memory to perform certain DML statements and queries against off-heap tables. This off-heap memory is required in addition to memory needed to store the row data. See Calculating Off-Heap Memory Requirements for DML and Query Execution for details.
  • The maximum row size that GemFire XD can store in off-heap memory is equivalent to the maximum size for table rows stored in the heap. In essence, you cannot store bigger rows by using off-heap memory, but you do not need to use smaller rows. If you are storing BLOBs or CLOBS in your off-heap table, the BLOBs or CLOBs will be stored in a table row in off-heap memory. The maximum size for a BLOB/CLOB stored in a table row is ~2GB, however the size of the BLOB or CLOB does not count towards the overall maximum size for your table row.
  • GemFire XD stores indexes for all tables in the JVM heap; therefore, indexes will use the same amount of heap memory regardless if you are storing the table rows in the heap or off-heap.
  • When you allocate off-heap memory on a GemFire XD data store, you should allocate the same amount of off-heap memory on all data stores in the same cluster.
  • For best performance in production deployments, use the gfxd server -lock-memory option to lock off-heap memory pages (and on-heap memory pages) in memory, especially when running GemFire XD on the same machine as Hadoop. See Supported Configurations and System Requirements for more information.
  • You cannot modify an existing table to use off-heap memory. To move table data that is stored in JVM heap memory to off-heap memory, you will need to drop and recreate the table. See Moving Table Data to Off-Heap Memory.
  • You can store both heap and off-heap tables on a single data store, but each table must be stored entirely off-heap or on-heap on all data stores that host the table in the cluster.
  • If you have created an off-heap table that is distributed across the cluster, any data store that subsequently joins the cluster will need to have off-heap memory allocated. If you do not allocate off-heap memory on the data store, the node will fail to start up with an error message similar to the following:
    GemFireXD Server pid: 8831 status: stopped
    Error starting server process: 
    IllegalStateException: The region /APP/AIRLINES was configured to use off heap memory but no off heap memory was configured. - See log file for details.
  • Keep in mind that any eviction settings configured for an off-heap table may also affect (and in some cases, increase) JVM heap memory usage. For example, if you use the eviction action of destroy, you can free up off-heap memory and heap memory. If you use the eviction action of overflow, you will free up off-heap memory as expected, however overflow eviction can also cause heap memory usage to increase. Some off-heap data is used as a key on the table, and when removed from off-heap memory by overflow eviction, that part of the data must be copied back to the heap.
  • By default, no off-heap memory is allocated upon the start up of a data store, and you cannot allocate off-heap memory on an already running data store. To allocate off-heap memory, you need to restart the data store.
  • If you try to create a table that specifies the OFFHEAP clause on a data store without allocated off-heap memory, you will get an error similar to the following:
    ERROR 38000: SQLSTATE=38000,SEVERITY=-1: (Server=localhost[1529],Thread[DRDAConnThread_20,5,gemfirexd.daemons]) 
    The exception 'java.lang.IllegalStateException: The region /APP/HOTELAVAILABILITY was configured to use off heap memory 
    but no off heap memory was configured.' was thrown while evaluating an expression.