Monitoring Off-Heap Memory Usage

This topic describes how to monitor off-heap memory usage and the statistics that are available in GemFire XD.

The GemFire MemberMXBean interface exposes several statistics related to off-heap memory usage, including the amount of off-heap memory used and available, and the number of objects stored in off-heap memory. You can also observe the time that GemFire XD spends compacting objects in off-heap memory, as well as the degree of fragmentation of off-heap data. See JMX Manager MBeans.

You can view some of these statistics in the Pulse interface or view all of them by using a standard JMX monitoring tool such as JConsole. In addition, there are several APIs to obtain statistics (MBean attributes) of the MemberMXBean.

List of Available Off-Heap Statistics

Statistic Description
compactions The total number of times off-heap memory has been compacted. If this statistic is a non-zero value, then the member has had to compact off-heap memory. This may indicate that the member is close to running out of off-heap memory.
compactionTime The total time spent compacting off-heap memory. This value will be zero unless you enable time statistics by using the enable-time-statistics boot property.
fragmentation The percentage of off-heap memory fragmentation. Updated every time a compaction is performed. An excessively high Fragmentation percentage indicates that the off-heap memory is sufficiently fragmented to potentially make it difficult to fill up or utilize.
fragments The number of fragments of free off-heap memory. Updated every time a compaction is done.
freeMemory The amount of off-heap memory, in bytes, that is not being used.
largestFragment The largest fragment of memory found by the last compaction of off heap memory. Updated every time a compaction is done.
maxMemory The maximum amount of off-heap memory, in bytes. This is the amount of memory allocated at startup and does not change.
objects The number of objects stored in off-heap memory.

Each row in a GemFire XD table will have one offheap Object for the row itself and one additional off-heap Object for each LOB in the row.

reads The total number of reads of off-heap memory. This statistic is incremented only by reads of a full object. If an object is only partially read, the statistic is not incremented.
usedMemory The amount of off-heap memory, in bytes, that is being used to store data.

MemberMXBean Off-Heap Attributes

MemberMXBean has new attributes which expose several of the off-heap memory statistics for the member. You can use the following APIs to obtain values for off-heap statistics:

To obtain the value of the off-heap maxMemory (the amount of allocated off-heap memory) statistic, combine the values for freeMemory (getOffHeapFreeSize) and usedMemory (getOffHeapUsedSize).

Note: The MemberMXBean operation listGemFireProperties includes the property off-heap-memory-size, which contains the same value as maxMemory.

RegionMXBean Off-Heap Attributes

The RegionMXBean listRegionAttributes API includes a new boolean attribute that indicates if the region is configured to use off-heap memory or regular Java heap memory. One of the attributes returned by the listRegionAttributes operation is enableOffHeapMemory which will have a value of true if that region is configured to use off-heap memory

Viewing Off-Heap Usage and Table Status in Pulse

To view how much off-heap memory is being used on a member, select Cluster View. Then select the List View icon and double-click on the member you wish to view. See Using Pulse Views for additional information.

The following screenshot depicts how Pulse reports off-heap memory usage for a member:

To view whether a table is storing its data in off-heap memory, select Data View and then select the table you wish to examine.

The following screenshot indicates how off-heap tables are represented in Pulse:

Errors

The following exceptions are associated with off-heap memory:
  • OutOfOffHeapMemoryException. When this exception is thrown, the cache will close in order to maintain consistency across the cluster. Each member should be configured to use off-heap eviction and critical off-heap percentage thresholds on the ResourceManager to avoid getting this exception and to prevent data inconsistency or member loss.
  • LowMemoryException. This exception is thrown during local and client put operations when the amount of off-heap memory being used has surpassed the configured critical-off-heap percentage threshold. You can specify this threshold using the gfxd start server command or by using the SYS.SET_CRITICAL_OFFHEAP_PERCENTAGE or SYS.SET_CRITICAL_OFFHEAP_PERCENTAGE_SGsystem procedure.