Supported Configurations and System Requirements

This topic describes the supported configurations and system requirements for GemFire XD.

Before installing GemFire XD, make sure your system meets the minimum system requirements for installing and running the product.

Pivotal HD Requirement

This release of GemFire XD is not provided as a standalone product. It is instead delivered as part of Pivotal HD Enterprise, and it requires Pivotal HD to be installed on the same machine in order to provide HDFS and HAWQ integration features. HDFS support is limited to Pivotal HD Hadoop.

Use the Pivotal Command Center CLI (Command Line Interface) to install GemFire XD. The Pivotal Command Center CLI installation process installs multiple instances of GemFire XD on different servers, along with HDFS, MapReduce, and other components you choose to install. See the Pivotal HD Enterprise Installation and Administrator Guide for more information.

Supported Configurations

The following table shows all supported configurations for GemFire XD.

Operating System Processor Architecture JVM
Red Hat EL 6.2, 6.4 x86 (64bit)

Java SE 1.7.0_05

Java SE 1.6.0_26*

CentOS 6.2, 6.4 x86 (64bit)

Java SE 1.7.0_05

Java SE 1.6.0_26*

*Pivotal HD version 2.0.1 does not support Java 1.6. If you intend to use GemFire XD with Pivotal HD version 2.0.1 or later, use Java SE 1.7.0_05.

Note: The GemFire XD product download does not include Java; you must download and install a supported JDK for your system.

Host Machine Requirements

Requirements for each host:
  • A supported Java SE installation.
  • File system that supports long file names.
  • Adequate per-user quota of file handles (ulimit for Linux). Large deployments with HDFS-persistent tables can require several thousand file descriptors to manage the HDFS log files for partitioned table buckets. Ensure that you set a high enough ulimit (up to the hard limit of 81,920) for your system. See How GemFire XD Manages HDFS Data and Compaction for HDFS Log Files.
  • TCP/IP.
  • System clock set to the correct time.
  • For each Linux host, the hostname and host files must be properly configured. See the system manpages for hostname and hosts.
  • For each Linux host, configure the swap volume to be the same size as the physical RAM installed in the computer.
  • Time synchronization service such as Network Time Protocol (NTP).
Note: For troubleshooting, you must run a time synchronization service on all hosts. Synchronized time stamps allow you to merge log messages from different hosts, for an accurate chronological history of a distributed run.
Note: If you deploy GemFire XD to a virtualized host, see also Running GemFire XD in Virtualized Environments.

Increase Unicast Buffer Size on Linux Platforms

On Linux platforms, execute the following commands as the root user to increase the unicast buffer size:
  1. Edit the /etc/sysctl.conf file to include the following lines:
    net.core.rmem_max=1048576
    net.core.wmem_max=1048576
  2. Reload sysctl.conf:
    sysctl -p

Disable SYN Cookies on Linux Platforms

Many default Linux installations use SYN cookies to protect the system against malicious attacks that flood TCP SYN packets. The use of SYN cookies dramatically reduces network bandwidth, and can be triggered by a running GemFire XD distributed system.

If your GemFire XD distributed system is otherwise protected against such attacks, disable SYN cookies to ensure that GemFire XD network throughput is not affected.

To disable SYN cookies permanently:
  1. Edit the /etc/sysctl.conf file to include the following line:
    net.ipv4.tcp_syncookies = 0

    Setting this value to zero disables SYN cookies.

  2. Reload sysctl.conf:
    sysctl -p

Client Requirements

GemFire XD supports two JDBC drivers: a thin client JDBC driver and a peer JDBC driver. GemFire XD server instances and the peer driver are supported only with Java SE 6. You can download Java from http://www.oracle.com/technetwork/java/javase/downloads/index.htm.

In addition, GemFire XD now supports an ODBC driver that responds to ODBC API function calls in client applications. The GemFire XD ODBC driver supports ODBC 3.5 APIs and the unixODBC and iODBC driver managers. The driver is a separate installation download from GemFire XD. SeeUsing ODBC Driver Functions in Client Applications for more information on driver installation requirements.

Requirements for Running GemFire XD and Hadoop on the Same Machine

Severe performance degradation can occur for insert-heavy workloads when GemFire XD and Hadoop run on the same machine and GemFire XD is configured with very large memory sizes (either off-heap or on-heap memory). The performance degradation occurs when the operating system pages out GemFire XD memory to disk.

To prevent this problem, perform these steps when deploying GemFire XD and Hadoop on the same machine:
  1. Increase the operating system's ulimit -l value (the maximum size that may be locked in memory) from the default of 32 KB to at least the total amount of memory used by GemFire XD for on-heap or off-heap storage.

    To permanently change the default limit for all users, edit the /etc/security/limits.conf file and add the line:

    * hard memlock limit-value-in-kb
  2. When you start each GemFire XD data store, include the gfxd -lock-memory option to prevent the operating system from paging out heap or off-heap memory.

Note: If you do not increase the "ulimit -l" value to at least the amount of memory used by GemFire XD, then GemFire XD will fail to start when you specify -lock-memory.

Note: Using locked memory in this manner increases the time required to start GemFire XD. The additional time required to start GemFire XD depends on the total amount of memory used, and can range from several seconds to 10 minutes or more. To improve startup time and reduce the potential of member timeouts, execute the following command before starting a GemFire XD member:
$ echo 1 > /proc/sys/vm/drop_caches

Requirements when using Hadoop without High Availability

If you do not configure your Hadoop system for high availability with redundant failover, then it is important that you set Hadoop timeout properties to these recommended values:
Table 1. Hadoop Client Timeout Properties
Client Timeout Property Recommended Value
ipc.client.ping false
ipc.ping.interval 5000
ipc.client.connect.timeout 5000
ipc.client.connect.max.retries 2
ipc.client.connect.max.retries.on.timeouts 0
dfs.client.socket-timeout 30000

Using these values ensures that GemFire XD read operations to Hadoop quickly fail if the Hadoop node is unavailable (instead of hanging for a long period of time due to the default timeout intervals).

Requirements for Deploying HAWQ with GemFire XD

Due to the memory requirements and memory configuration required for each product, Pivotal does not support production deployments of GemFire XD and HAWQ on the same machine. Deploy GemFire XD and HAWQ on the same machine or virtual machine only for development or testing purposes.

Note: When you deploy GemFire XD with HAWQ on the same machine, you must set the swap volume to the same size as physical RAM. See Adding Swap Space in the RedHat or CentOS documentation for details.