|Getting Started with GemFire XD / Overview of Pivotal GemFire XD|
GemFire XD incorporates core Pivotal GemFire technology and Apache Derby RDBMS components to provide a high-performance, distributed database management system. GemFire XD extends standard SQL statements where necessary for creating and managing tables and configuring the GemFire XD system. GemFire XD also provides Pivotal HD integration components to make HDFS persistence files available to Hadoop tools and HAWQ.
The sections that follow document how GemFire XD utilizes the Pivotal GemFire and Derby component functionality.
The Pivotal community site provides a comparison of Pivotal GemFire XD to other data management systems, including Pivotal GemFire.
If you are currently using GemFire XD 1.3.x with Pivotal HD for HDFS table support, continue to use your installed version of the product. Pivotal will provide maintenance releases for version 1.3.x until GemFire XD 1.4 is certified with an upcoming version of Pivotal HD.
GemFire XD stores table data in HDFS in a indexed format that supports read/write operations to the persisted data. This enables GemFire XD to easily persist large amounts of data to HDFS while providing indexed access that data for quick retrieval when SQL queries cannot be satisfied purely by the data available in memory.
However, GemFire XD also provides an InputFormatter component to enable direct processing of the HDFS data in the Hadoop ecosystem. Pivotal HD tools such as MapReduce can use the InputFormatter to access table data without acting as GemFire XD clients. Similarly, an OutputFormatter enables these tools to push table data back into the in-memory tier in a GemFire XD-compatible format.
GemFire XD also supports a PXF driver that enables HAWQ to query GemFire XD data stored in HDFS, without starting or accessing a GemFire XD distributed system. (The PXF driver is installed with HAWQ.)
GemFire XD modifies and extends the query engine and SQL interface to provide support for partitioned and replicated tables, data-aware procedures, data persistence, data eviction, and other features unique to the distributed GemFire XD architecture. GemFire XD also adds SQL commands, stored procedures, system tables, and functions to help easily manage features of the distributed system, such as persistent disk stores, listeners, and locators.