Configuring and Using Off-Heap Memory

This topic describes how to allocate off-heap memory on your servers and how to configure tables to use off-heap memory.

Allocating Off-Heap Memory and Creating Off-Heap Tables

  1. Set the off-heap-memory-size boot property for all GemFire XD data stores in the distributed system, either in the gfxd startup command or in the gemfirexd.properties file. For example:
    gfxd server start -dir=server1 -heap-size=10g -off-heap-memory-size=200g -lock-memory
    Some other valid example values:
    -off-heap-memory-size=4096m
    -off-heap-memory-size=10485760k
    -off-heap-memory-size=120g
    If you are specifying a value greater than 2g, the amount of allocated gigabyte memory should be a multiple of 2 since table data is stored in 2g chunks of memory.

    See Estimating GemFire XD Heap Overhead and Table Memory Requirements for guidance on how to estimate the memory allocations for both heap and off-heap memory.

    Note: All data stores that host the same table in off-heap memory should specify the same value for off-heap-memory-size.
    This property both enables off-heap memory storage for the data store and specifies the maximum amount of memory allocated for off-heap storage. The off-heap memory is allocated immediately when each member boots, as you can verify in the log file entry:
    [config 2013/09/10 18:37:25.157 PDT accessor_hs21d_21724 <vm_0_thr_0_accessor1_hs21d_21724> tid=0x2e] 
    Allocating 367001600 bytes of off-heap memory. The maximum size of a single off-heap object is 367001600 bytes.
  2. Specify the OFFHEAP Clause when you create a new table with CREATE TABLE. For example:
    CREATE TABLE HOTELAVAILABILITY
        (HOTEL_ID INT NOT NULL, 
         BOOKING_DATE DATE NOT NULL,
         ROOMS_TAKEN INT DEFAULT 0,
         PRIMARY KEY (HOTEL_ID, BOOKING_DATE))
         OFFHEAP;
    If you create a table with the OFFHEAP clause specified but have not allocated off-heap memory on all the data stores that are receiving this table, you will receive a error message similar to the following:
    ERROR 38000: SQLSTATE=38000,SEVERITY=-1: (Server=localhost[1529],Thread[DRDAConnThread_20,5,gemfirexd.daemons]) 
    The exception 'java.lang.IllegalStateException: The region /APP/HOTELAVAILABILITY was configured to use off heap memory 
    but no off heap memory was configured.' was thrown while evaluating an expression.

Configuring Eviction and Critical Thresholds for Data in Off-Heap Memory

You can still use data eviction including LRU eviction when using off-heap memory to free up space if necessary. Setting the eviction-off-heap-percentage threshold on a data store will trigger eviction on any off-heap table that has been configured with LRU_HEAP.

To set up LRU eviction for table entries stored in off-heap memory, you can either:

If you use the DESTROY eviction action on a partitioned table, you will free up both off-heap and heap memory.

If you use the OVERFLOW eviction action, you will free up off-heap memory as expected, however overflow eviction can also cause heap memory usage to increase. Some off-heap data is used as a key on the table, and when removed from off-heap memory by overflow eviction, that part of the data must be copied back to the heap.

By default, if you specify -off-heap-size when starting up the gfxd server, GemFire XD will set a default critical threshold of 90%. To set a different critical memory threshold when using off-heap memory, you can either:
The following is an example of starting up a server with eviction and critical percentage thresholds:
gfxd server start -dir=server1 -off-heap-memory-size=200g -critical-off-heap-percentage=99.99 -eviction-off-heap-percentage=85
See server for more information on starting your servers with gfxd.

Moving Table Data to Off-Heap Memory

When you create a table without specifying the OFFHEAP clause, you cannot modify the table to become a table stored in off-heap memory. Similarly, you cannot move an existing table data stored in off-heap memory to heap memory. If you need to switch where table data is stored, you need to drop and recreate the table (in a similar manner to how you would modify a replicated table into a partitioned table.)

The following is an example procedure for moving an existing table to off-heap memory:
  1. Export your table data using the SYSCS_UTIL.EXPORT_TABLE built-in stored procedure.
  2. After you have successfully exported your table data, drop the existing table.
  3. If you have not already allocated off-heap memory on all the nodes that will host the off-heap table, restart those nodes and specify the -off-heap memory-size boot property. For example:
    gfxd server start -dir=server1 -off-heap-memory-size=2g
  4. Re-create the table and specify the OFFHEAP clause. For example:
    CREATE TABLE HOTELAVAILABILITY
        (HOTEL_ID INT NOT NULL, 
         BOOKING_DATE DATE NOT NULL,
         ROOMS_TAKEN INT DEFAULT 0,
         PRIMARY KEY (HOTEL_ID, BOOKING_DATE))
         OFFHEAP;
  5. Import your table's data into the new off-heap table by using the SYSCS_UTIL.IMPORT_TABLE built-in stored procedure.