Understanding High Availability for Partitioned Tables

With high availability, each member that hosts data for the partitioned table gets some primary copies and some redundant (secondary) copies.

With redundancy, if one member fails, operations continue on the partitioned table with no interruption of service:

If the member hosting the primary copy is lost, GemFire XD makes a secondary copy the primary. This might cause a temporary loss of redundancy, but not a loss of data.
Whenever there are not enough secondary copies to satisfy redundancy, the system works to recover redundancy by assigning another member as secondary and copying the data to it.

Note: You can still lose cached data when you are using redundancy if enough members go down in a short enough time span.

You can configure how the system works to recover redundancy when it is not satisfied. You can configure recovery to take place immediately or, if you want to give replacement members a chance to start up, you can configure a wait period. Redundancy recovery is also automatically attempted during any partitioned data rebalancing operation. Without redundancy, the loss of any of the table's data stores causes the loss of some of the table's cached data.

Generally, you should not use redundancy when your applications can directly read from another data source, or when write performance is more important than read performance.

Controlling Where Your Primaries and Secondaries Reside

By default, GemFire XD places your primary and secondary data copies for you, avoiding placement of two copies on the same physical machine. If there are not enough machines to keep different copies separate, GemFire XD places copies on the same physical machine. You can change this behavior, so GemFire XD only places copies on separate machines.

You can also control which members store your primary and secondary data copies by using redundancy zones. Redundancy zones are configured at the member level. Redundancy zones let you separate primary and secondary copies by member groups, or zones. You assign each data host to a zone. Then GemFire XD places redundant copies in different redundancy zones, the same as it places redundant copies on different physical machines. You can use this to split data copies across different machine racks or networks, This option allows you to add members on the fly and use rebalancing to redistribute the data load, with redundant data maintained in separate zones. When you use redundancy zones, GemFire XD will not place two copies of the data in the same zone, so make sure you have enough zones.

Running Processes in VMware Virtual Machines

By default, GemFire XD stores redundant copies on different machines. When you run your processes in VMware virtual machines, the normal view of the machine becomes the VMware VM and not the physical machine. If you run multiple VMWare VMs on the same physical machine, you could end up storing partitioned table primary buckets in separate VMware VMs, but on the same physical machine as your secondaries. If the physical machine fails, you can lose data. When you run in VMware VMs, you can configure GemFire XD to identify the physical machine and store redundant copies on different physical machines.

Reads and Writes in Highly-Available Partitioned Tables

GemFire XD treats reads and writes differently in highly-available partitioned tables than in other tables because the data is available in multiple members:

Write operations (like PUT INTO) go to the primary for the data keys and then are distributed synchronously to the redundant copies.
Read operations go to any member holding a copy of the data, with the local member favored, so a read intensive system can scale much better and handle higher loads.

In this figure, M1 is reading W, Y, and Z. It gets W directly from its local copy. Since it doesn't have a local copy of Y or Z, it goes to a cache that does, picking the source cache at random.