Create a GemFire XD Cluster

In this tutorial you set up and start a cluster of two GemFire XD servers.

Procedure
  1. Begin by opening a command prompt or terminal window on the computer or VM where you installed GemFire XD.
  2. The initial GemFire XD cluster in this tutorial contains two standalone GemFire XD server members. Move to your home directory and create a new directory for each server:
    $ cd ~
    $ mkdir server1 server2
    Each server will use its local directory to write log files, backup disk store files, a datadictionary directory for persisting data, and a single status file, .gfxdserver.ser.
  3. To manage client connections to the available GemFire XD server members, the cluster in this tutorial uses a GemFire XD locator member. Create a new directory for the locator member:
    $ mkdir locator
    A locator maintains a list of available servers in the cluster, and updates that list as servers join and leave the cluster. Locators also load balance client connections across all available servers.
  4. To start or stop a GemFire XD locator or a GemFire XD server, you use the gfxd script (for Linux platforms) or gfxd.bat script (for Windows platforms). If you installed GemFire XD using the RPM or your are using the Pivotal HD Single Node VM, the installation process automaticallly sets your path to include the gfxd script, and you can skip this step.

    If you installed using the ZIP file distribution, you must first ensure that the path to the GemFire XD bin directory is part of your PATH environment variable. For example, on a Linux platform enter:

    $ export PATH=$PATH:path-to-gemfirexd-installation/bin
    On a Windows platform enter:
    c:\> set Path=%Path%;path-to-gemfirexd-installation\bin
    Substitute path-to-gemfirexd-installation with the actual path to your GemFire XD installation.
  5. When you start a GemFire XD distributed system, always begin by starting the locator member. Use the gfxd locator command to start the locator in the specified directory:
    $ gfxd locator start -dir=$HOME/locator -peer-discovery-port=10101 -client-port=1527 -jmx-manager-start=true -jmx-manager-http-port=7075
    Starting GemFireXD Locator using peer discovery on: 0.0.0.0[10101]
    Starting network server for GemFireXD Locator at address localhost/127.0.0.1[1527]
    Logs generated in /home/gpadmin/locator/gfxdlocator.log
    GemFireXD Locator pid: 41456 status: running

    The -peer-discovery-port defines a unique connection port that all members of this GemFire XD distributed system use for communicating with one another. Always use a unique -peer-discovery-port number to avoid joining a cluster that is already running on your network. If other people might be evaluating GemFire XD on your network, choose a port number other than 10101.

    The -client-port defines the connection port that client applications use to connect to this locator. In this tutorial, all GemFire XD members run on the localhost address and use different port numbers to define unique connections. In a production environment, you can specify dedicated network interfaces for member discovery or client access by using the -peer-discovery-address and -client-bind-address options.

    Specifying -jmx-manager-start=true starts an embedded JMX Manager within the locator. By starting a JMX Manager, you can monitor and browse the GemFire XD distributed system through the Pulse graphical interface.

    You can verify the peer and client connections in the locator startup messages.

    Note: By starting the locator member first, the locator can manage cluster membership from the start as new servers join and leave the distributed system. The locator member should also be the last process that you shut down when you want to stop a GemFire XD distributed system.
    Note: As an alternative, GemFire XD members can discover each other using multicast messaging. However, important GemFire XD features such as WAN replication and user authentication require that a GemFire XD system use locators rather than multicast for discovery. See Configuring Discovery Mechanisms.
  6. Now use the gfxd server command to start both GemFire XD server members and join them to the distributed system:
    $ gfxd server start -dir=$HOME/server1 -locators=localhost[10101] -client-port=1528
    Starting GemFireXD Server using locators for peer discovery: localhost[10101]
    Starting network server for GemFireXD Server at address localhost/127.0.0.1[1528]
    Logs generated in /home/gpadmin/server1/gfxdserver.log
    GemFireXD Server pid: 42051 status: running
      Distributed system now has 2 members.
      Other members: 192.168.125.140(41456:locator):20431
    $ gfxd server start -dir=$HOME/server2 -locators=localhost[10101] -client-port=1529
    Starting GemFireXD Server using locators for peer discovery: localhost[10101]
    Starting network server for GemFireXD Server at address localhost/127.0.0.1[1529]
    Logs generated in /home/gpadmin/server2/gfxdserver.log
    GemFireXD Server pid: 42528 status: running
      Distributed system now has 3 members.
      Other members: 192.168.125.140(41456:locator):20431, 192.168.125.140(42051:datastore)<v1>:13160

    In each command, the -locators option defines the peer discovery address to use for joining the GemFire XD distributed system. Production deployments generally use multiple locator members, in which case you would specify a comma-separated list of locator host[port] connections when starting a server.)

    Again, the -client-port indicates that each server will listen for thin clients on a unique connection (localhost:1528 and localhost:1529, respectively). However, in the tutorials that follow all clients will connect using the locator instead of making direct connections to servers.

    Startup messages show the cluster membership details.

    By default, new GemFire XD servers host data as data stores, and are automatically added to a default server group. You can optionally specify server group membership at startup using the server-groups boot property.

  7. Before adding any data to the cluster, look at the contents of the new member directories now that the distributed system has started up:
    $ ls $HOME/locator $HOME/server1 $HOME/server2
    /Users/yozie/locator:
    BACKUPGFXD-DEFAULT-DISKSTORE.if     DRLK_IFGFXD-DEFAULT-DISKSTORE.lk    gfxdlocator.log                     start_gfxdlocator.log
    BACKUPGFXD-DEFAULT-DISKSTORE_1.crf  datadictionary/                     locator10101state.dat
    BACKUPGFXD-DEFAULT-DISKSTORE_1.drf  gfxdlocator-01-01.log               locator10101views.log
    
    /Users/yozie/server1:
    BACKUPGFXD-DEFAULT-DISKSTORE.if     BACKUPGFXD-DEFAULT-DISKSTORE_1.drf  datadictionary/                     start_gfxdserver.log
    BACKUPGFXD-DEFAULT-DISKSTORE_1.crf  DRLK_IFGFXD-DEFAULT-DISKSTORE.lk    gfxdserver.log
    
    /Users/yozie/server2:
    BACKUPGFXD-DEFAULT-DISKSTORE.if     BACKUPGFXD-DEFAULT-DISKSTORE_1.drf  datadictionary/                     start_gfxdserver.log
    BACKUPGFXD-DEFAULT-DISKSTORE_1.crf  DRLK_IFGFXD-DEFAULT-DISKSTORE.lk    gfxdserver.log
    Notice the files that begin with BACKUPGFXD-DEFAULT-DISKSTORE. These are GemFire XD disk store files, which can be used to persist table data to disk. The files listed above are created for the default server group, which includes all data stores members that join the distributed system. You can also create named server groups and disk stores, but the default group and disk stores are used for persistence if you do not specifically assign persistent tables and queues to a named server group. Also notice the /datadictionary subdirectory on each member:
    $ ls $HOME/locator/datadictionary $HOME/server1/datadictionary $HOME/server2/datadictionary
    /Users/yozie/locator/datadictionary:
    BACKUPGFXD-DD-DISKSTORE.if     BACKUPGFXD-DD-DISKSTORE_1.crf  BACKUPGFXD-DD-DISKSTORE_1.drf  DRLK_IFGFXD-DD-DISKSTORE.lk
    
    /Users/yozie/server1/datadictionary:
    BACKUPGFXD-DD-DISKSTORE.if     BACKUPGFXD-DD-DISKSTORE_1.crf  BACKUPGFXD-DD-DISKSTORE_1.drf  DRLK_IFGFXD-DD-DISKSTORE.lk
    
    /Users/yozie/server2/datadictionary/:
    BACKUPGFXD-DD-DISKSTORE.if     BACKUPGFXD-DD-DISKSTORE_1.crf  BACKUPGFXD-DD-DISKSTORE_1.drf  DRLK_IFGFXD-DD-DISKSTORE.lk

    A separate set of disk store files are created to maintain the datadictionary for the distributed system. Regardless of whether you intend to persist your table data to disk, GemFire XD always persists the datadictionary, so any Data Definition Language (DDL) that you execute creates objects that exist even through full shutdowns of the distributed system.

    Also notice that the locator also has created disk store files. Like data store members, locators maintain persistence files for the cluster. This is important as you may want to stop or restart data store members for maintenance. Locator process that remain running continue to maintain their persitent data files, and datastores that rejoin the cluster at a later time can bring their persistence files up-to-date (referred to as "recovering" their persistence files) using disk store files from a locator or other data store members.

    More information about persistence and disk store files is provided in the persistence tutorial. The important thing to keep in mind is that you should never delete, move, or rename persistence files or directories, as it is critical that GemFire XD members have access to these files when they restart and join the distributed system.