This topic provides some guidelines on how and when to use off-heap memory
Using off-heap memory is recommended for any deployment where you need to keep a large amount
of data in-memory. If you are running servers with at least 150GB of RAM or you typically run
JVMs with more than 32GB of user data, you may benefit from keeping your table data in off-heap
You can use off-heap memory on either 32-bit or 64-bit architectures.
In addition, storing table data in off-heap memory may most benefit deployments that have rows
that are either uniform in size or have a pattern of common sized values. Off-heap memory may not
perform as well if your tables have drastically different and unpredictably sized row values, or
if you have frequent table updates that change the size of row values. These conditions can cause
fragmentation of off-heap storage space, which can lead to poor query performances.
Additional Guidelines for Using Off-Heap Storage
When using off-heap memory, keep in mind the following guidelines:
- When using off-heap tables, GemFire XD only stores table rows in off-heap memory. Table
rows include BLOB and CLOB data. Each table row stored in off-heap memory also consumes some
heap memory and should be accounted for when calculating JVM heap overhead. See Estimating GemFire XD Heap Overhead and Table Memory Requirements for
- GemFire XD also requires off-heap memory to perform certain DML statements and queries
against off-heap tables. This off-heap memory is required in addition to memory needed to
store the row data. See Calculating Off-Heap Memory Requirements for DML and Query Execution for details.
- The maximum row size that GemFire XD can store in off-heap memory is equivalent to the
maximum size for table rows stored in the heap. In essence, you cannot store bigger rows by
using off-heap memory, but you do not need to use smaller rows. If you are storing BLOBs or
CLOBS in your off-heap table, the BLOBs or CLOBs will be stored in a table row in off-heap
memory. The maximum size for a BLOB/CLOB stored in a table row is ~2GB, however the size of
the BLOB or CLOB does not count towards the overall maximum size for your table row.
- GemFire XD stores indexes for all tables in the JVM heap; therefore, indexes will use the
same amount of heap memory regardless if you are storing the table rows in the heap or
- When you allocate off-heap memory on a GemFire XD data store, you should allocate the same
amount of off-heap memory on all data stores in the same cluster.
- For best performance in production deployments, use the gfxd server
-lock-memory option to lock off-heap memory pages (and on-heap memory pages) in
memory, especially when running GemFire XD on the same machine as Hadoop. See Supported Configurations and System Requirements for
- You cannot modify an existing table to use off-heap memory. To move table data that is
stored in JVM heap memory to off-heap memory, you will need to drop and recreate the table.
See Moving Table Data to Off-Heap Memory.
- You can store both heap and off-heap tables on a single data store, but each table must be
stored entirely off-heap or on-heap on all data stores that host the table in the
- If you have created an off-heap table that is distributed across the cluster, any data
store that subsequently joins the cluster will need to have off-heap memory allocated. If you
do not allocate off-heap memory on the data store, the node will fail to start up with an
error message similar to the
GemFireXD Server pid: 8831 status: stopped
Error starting server process:
IllegalStateException: The region /APP/AIRLINES was configured to use off heap memory but no off heap memory was configured. - See log file for details.
- Keep in mind that any eviction settings configured for an off-heap table may also affect
(and in some cases, increase) JVM heap memory usage. For example, if you use the eviction
action of destroy with a partitioned table, you can free up off-heap memory and heap memory. If you use the eviction
action of overflow, you will free up off-heap memory as expected, however overflow eviction
can also cause heap memory usage to increase. Some off-heap data is used as a key on the
table, and when removed from off-heap memory by overflow eviction, that part of the data must
be copied back to the heap.
- By default, no off-heap memory is allocated upon the start up of a data store, and you
cannot allocate off-heap memory on an already running data store. To allocate off-heap memory,
you need to restart the data store.
- If you try to create a table that specifies the OFFHEAP clause on a data store without
allocated off-heap memory, you will get an error similar to the
ERROR 38000: SQLSTATE=38000,SEVERITY=-1: (Server=localhost,Thread[DRDAConnThread_20,5,gemfirexd.daemons])
The exception 'java.lang.IllegalStateException: The region /APP/HOTELAVAILABILITY was configured to use off heap memory
but no off heap memory was configured.' was thrown while evaluating an expression.