|Managing Your Data in GemFire XD / Managing Tables in Memory|
GemFire XD requires different amounts of heap memory overhead per table and index entry depending on whether you persist table data or configure tables for overflow to disk. If you already have representative data, query the SYS.MEMORYANALYTICS table to obtain a more accurate picture of the memory required to store your data.
|Table is persisted?||Overflow is configured?||Approximate heap overhead|
|Type of index entry||Approximate heap overhead|
|New index entry||80 bytes|
|First non-unique index entry||24 bytes|
|Subsequent non-unique index entry||8 bytes to 24 bytes*|
*If there are more than 100 entries for a single index entry, the heap overhead per entry increases from 8 bytes to approximately 24 bytes.
To calculate the total amount of memory that your table will require, use the SYS.MEMORYANALYTICS table on a representative data set that is as close as possible to your production data set.
If you are storing all your table data in the Java heap, multiply TOTAL_SIZE by the total number of rows (NUM_ROWS) for your table. Then add the per-entry and per-index overhead figures described in the tables above. This will provide an estimate of how much memory you need to allocate to the JVM heap.
If you are storing your table data in off-heap memory, you will need to calculate both heap memory requirements and off-heap memory requirements. To calculate your heap memory requirements, use the tables in the previous section to calculate the per-entry and per-index heap overhead. To calculate the amount of off-heap memory that your off-heap table will require, multiply VALUE_SIZE_OFFHEAP by the number of rows (NUM_ROWS) that will be in your table. This provides the amount of off-heap memory you will need to allocate for this table on your data stores.
See Viewing Memory Usage in SYS.MEMORYANALYTICS for more details.
If you store table data in off-heap memory, GemFire XD also consumes off-heap memory to perform UPDATE and DELETE operations against those tables, as well as to perform certain types of queries against the tables. This off-heap memory is required in addition to the amount of memory needed to store the row data off-heap.
The approximate amount of off-heap memory required for a DML operation or query is m * n * 8 bytes, where m is the estimated number of concurrent operations or qualifying queries performed against the table and n is the estimated number of rows modified or accessed during each operation or query. Repeat this calculation for each DML operation or qualifying query and use the maximum value in your capacity planning.
In general, all multi-table queries that have an explicit join (or that the optimizer can convert into an equi-join) do not consume additional off-heap memory during execution.