Overview of Multi-site Distribution

A multi-site installation consists of two or more distributed systems that are loosely coupled. Each site manages its own distributed system, but table data is distributed to remote sites using one or more logical connections.

The logical connections consist of a gateway sender in the sending site, and a gateway receiver in the receiving site. The gateway connection between systems is tolerant of weak or slow physical connections between distributed system sites. All data exchange occurs asynchronously without any loss of data.

Gateway senders and receivers are configured in the server layer.

Discovery for Multi-Site Systems

Each GemFire XD cluster in a WAN configuration uses locators to discover remote GemFire XD clusters as well as local GemFire XD members.

Each locator in a WAN configuration defines a unique distributed-system-id property that identifies the local cluster to which it belongs. A locator uses the remote-locators property to define the addresses of one or more locators in remote GemFire XD clusters to use for WAN distribution.

When a locator starts up, it contacts each locator that is configured in the remote-locators property to exchange information about the available locators and gateway receivers in the cluster. The locator also shares information about locators and gateway receivers in any other GemFire XD clusters that have connected to the cluster. Connected clusters can then use the shared gateway receiver information to distribute table updates according to their configured gateway senders.

Each time a new locator starts up or an existing locator shuts down, the changed information is broadcast to other connected GemFire XD clusters.

Note: When you configure a multi-site system, you must use locators for discovery rather than multicast.

Gateway Senders

A GemFire XD cluster uses a gateway sender to distribute table DML events to another, remote GemFire XD cluster. You can create multiple gateway sender configurations to distribute DML events to multiple remote clusters, and/or to distribute DML events concurrently to another remote cluster.

A gateway sender always communicates with a gateway receiver in a remote cluster. Gateway senders do not communicate directly with other cache server instances. See Gateway Receivers.

A gateway sender distributes table DML events from a single GemFire XD server in the local cluster to a remote GemFire XD cluster. Although multiple tables can use the same gateway for distribution, a gateway uses a single logical event queue to dispatch events for all tables that use the gateway sender.

As an application performs DML operations on a WAN-replicated table, GemFire XD writes those operations, in order, to a single gateway sender queue. GemFire XD preserves the ordering of those DML events as they are distributed across the WAN. This can provide a high level of consistency between two tables in a WAN-replicated configuration, especially when multiple distributed systems might update the same table concurrently.

However, keep in mind that a gateway sender provides only a finite amount of throughput for distributing DML operations. As you add more tables and servers to the local cluster, you may need to configure additional gateway senders manually and isolate individual tables on specific gateway senders to handle the increased distribution traffic.

Gateway Sender Queues

The queue that a gateway sender uses to distribute events to a remote site overflows to disk as needed, in order to prevent the GemFire XD member from running out of memory. You can configure the maximum amount of memory that each queue uses, as well as the batch size and frequency for processing batches in the queue. You can also configure these queues to persist to disk, so that a gateway sender can pick up where it left off when its member shuts down and is later restarted.

See Create and Start Gateway Sender.

High Availability for Gateway Senders

When a gateway sender configuration is deployed to a server group that contains multiple GemFire XD datastore members, only one "primary" sender is active at a given time. All other gateway sender instances are standby "secondaries" that are available as backups if the primary sender shuts down. GemFire XD designates the first datastore member that starts up in the server group to run the primary sender, while all other members in the server group host secondary senders. As gateway senders or datastores start and shut down in the distributed system, GemFire XD ensures that the oldest-running gateway sender operates as the primary.

Gateway Receivers

A gateway receiver configures a physical connection for receiving update events from gateway senders in one or more remote GemFire XD clusters.

A gateway receiver applies each update event to the same table (or table partition) that is hosted in the local GemFire XD member. (An exception is thrown if the receiver receives an event for a table that it does not define.)

Gateway senders use any available gateway receiver in the target cluster to send update events. You can deploy gateway receiver configurations to multiple GemFire XD members as needed for high availability and load balancing.

See Create a Gateway Receiver.