Persist Tables to Disk

By default, a GemFire XD distributed system persists only the data dictionary for the tables and indexes you create. These persistence files are stored in the datadictionary subdirectory of each locator and data store that joins the distributed system. Table data, however, is not persisted by default; if you shut down all GemFire XD members, the tables are empty on the next startup. In this tutorial you will persist table data to GemFire XD disk store files.

Procedure

In a separate terminal window or GUI editor, examine the contents of the create_persistent_schema.sql script. Notice that this script uses the PERSISTENT keyword in each CREATE TABLE statement. For example, in the script the statement for creating the COUNTRIES table reads:
```
CREATE TABLE COUNTRIES
   (
      COUNTRY VARCHAR(26) NOT NULL CONSTRAINT COUNTRIES_UNQ_NM Unique,
      COUNTRY_ISO_CODE CHAR(2) NOT NULL CONSTRAINT COUNTRIES_PK PRIMARY KEY,
      REGION VARCHAR(26),
      CONSTRAINT COUNTRIES_UC
        CHECK (country_ISO_code = upper(country_ISO_code) )

   ) REPLICATE PERSISTENT;
```
When you execute the script in the next step, the existing schema will be dropped and replaced with a schema that GemFire XD persists to operational log files in the server member directories.
In the gfxd session, execute the create_persistent_schema.sql script, then load table data:
```
gfxd> run 'create_persistent_schema.sql';
gfxd> run 'loadTables.sql';
```
Exit the gfxd session:
```
gfxd> exit;
```
Because the data is persistent, GemFire XD recovers data from disk when you restart the data stores. First use the shut-down-all command to shut down all data stores in the cluster:
```
$ gfxd shut-down-all -locators=localhost[10101]
Connecting to distributed system: locators=localhost[10101]
Successfully shut down 2 members
```

Restart the data stores using the commands:

$ gfxd locator start -dir=$HOME/locator -peer-discovery-port=10101 -client-port=1527 -jmx-manager-start=true -jmx-manager-http-port=7075 -enable-network-partition-detection=trueStarting GemFireXD Locator using peer discovery on: 0.0.0.0[10101]
Starting network server for GemFireXD Locator at address localhost/127.0.0.1[1527]
Logs generated in /Users/dyozie/locator/gfxdlocator.log
GemFireXD Locator pid: 25723 status: waiting
Waiting for table GFXD_PdxTypes (DiskId: fbefd3dd-2195-4936-b454-f4bcb6175037, Location: /Users/dyozie/locator/./datadictionary) on: 
 [DiskId: 0c4f0b73-036e-413a-a05d-bb01a521d3c0, Location: /Users/dyozie/server1/./datadictionary]
$ gfxd server start -dir=$HOME/server1 -locators=localhost[10101] -client-port=1528 -sync=false -enable-network-partition-detection=trueStarting GemFireXD Server using locators for peer discovery: localhost[10101]
Starting network server for GemFireXD Server at address localhost/127.0.0.1[1528]
Logs generated in /Users/dyozie/server1/gfxdserver.log
GemFireXD Server pid: 25734 status: waiting
Waiting for table APP.AIRLINES (DiskId: 9c0ba4d7-d9f8-469b-83d0-f8417977aeb3, Location: /Users/dyozie/server1/.) on: 
 [DiskId: a78adff5-2a96-4ac6-a9cf-09515b6ad385, Location: /Users/dyozie/server2/.]
$ gfxd server start -dir=$HOME/server2 -locators=localhost[10101] -client-port=1529 -sync=false -enable-network-partition-detection=true
Starting GemFireXD Server using locators for peer discovery: localhost[10101]
Starting network server for GemFireXD Server at address localhost/127.0.0.1[1529]
Logs generated in /Users/dyozie/server2/gfxdserver.log
GemFireXD Server pid: 25743 status: running
  Distributed system now has 3 members.
  Other members: 10.84.17.128(25734:datastore)<v1>:31259, 10.84.17.128(25723:locator)<ec><v0>:16099

Note: By default, GemFire XD servers and locators start to a "waiting" state if the member depends on another server or locator for disk store synchronization. The server automatically continues booting after any dependent servers or locators start. If you start servers from within a shell script, you can use the gfxd locator wait or gfxd server wait command to wait for members to reach the running state before continuing. As an alternative, you can monitor the current status of GemFire XD members using STATUS column in the MEMBERS system table.

Notice the startup message:

Waiting for table APP.AIRLINES (DiskId: 9c0ba4d7-d9f8-469b-83d0-f8417977aeb3, Location: /Users/dyozie/server1/.) on: 
 [DiskId: a78adff5-2a96-4ac6-a9cf-09515b6ad385, Location: /Users/dyozie/server2/.]

This is not an error message. It indicates that the GemFire XD member you are starting is waiting for another member to become available online, in order to ensure that the distributed system uses the very latest version of any persisted data.

Also notice that the startup commands include the new option, -enable-network-partition-detection=true. Enabling network partition is a best practice for any system that persists data, even if you only persist the data dictionary. When network partition detection is disabled, a network segmentation can cause different groups of data stores to persist inconsistent data in their disk store files. See Member Startup Problems for more information.

Now verify that the persistent tables were reloaded:

$ gfxd
gfxd> connect client 'localhost:1527';
gfxd> select distinct dsid() as id from flights;
ID                                                                             
-------------------------------------------------------------------------------
10.84.17.128(25734)<v1>:31259                                                                                                   
10.84.17.128(25743)<v2>:21552                                                

2 rows selected
gfxd> select count(*) memberRowCount, dsid() from flights group by dsid();
MEMBERROWCOUNT|2                                                               
-------------------------------------------------------------------------------
279           |10.84.17.128(25734)<v1>:31259                                                                                                   
263           |10.84.17.128(25743)<v2>:21552                                  

2 rows selected

Note: The exact row count may be different for your GemFire XD distributed system.

Because the tutorial cluster uses no server groups, you could not specify a named server group when creating tables with the PERSISTENT clause. When you do not use a named server group, GemFire XD stores persistent data the default server group. As mentioned in an earlier tutorial, the default server group disk store files are located in the member directory of each data store and locator member. The files begin with BACKUPGFXD-DEFAULT-DISKSTORE:

$ ls $HOME/locator $HOME/server1 $HOME/server2
/Users/yozie/locator:
BACKUPGFXD-DEFAULT-DISKSTORE.if     DRLK_IFGFXD-DEFAULT-DISKSTORE.lk    gfxdlocator.log                     start_gfxdlocator.log
BACKUPGFXD-DEFAULT-DISKSTORE_1.crf  datadictionary/                     locator10101state.dat
BACKUPGFXD-DEFAULT-DISKSTORE_1.drf  gfxdlocator-01-01.log               locator10101views.log

/Users/yozie/server1:
BACKUPGFXD-DEFAULT-DISKSTORE.if     BACKUPGFXD-DEFAULT-DISKSTORE_1.drf  datadictionary/                     start_gfxdserver.log
BACKUPGFXD-DEFAULT-DISKSTORE_1.crf  DRLK_IFGFXD-DEFAULT-DISKSTORE.lk    gfxdserver.log

/Users/yozie/server2:
BACKUPGFXD-DEFAULT-DISKSTORE.if     BACKUPGFXD-DEFAULT-DISKSTORE_1.drf  datadictionary/                     start_gfxdserver.log
BACKUPGFXD-DEFAULT-DISKSTORE_1.crf  DRLK_IFGFXD-DEFAULT-DISKSTORE.lk    gfxdserver.log

Disk store files are critical for maintaining the integrity of data in your system. Maintaining a backup of all disk store files (for the data dictionary, as well as table data), you can recover from disk corruption or failed storage devices. You can also use disk store backup files to move GemFire XD members to new hardware as necessary. In order to backup disk store files, first create a dedicated directory for storing the backups:

$ mkdir $HOME/gfxd-backups

Then create a full disk store backup using the gfxd backup command:

$ gfxd backup $HOME/gfxd-backups -locators=localhost[10101]
Connecting to distributed system: locators=localhost[10101]
The following disk stores were backed up:
	20f8d17e-4f23-4615-bc73-950cd639b6f2 [ward-4.local:/Users/yozie/server1/.]
	b2912f8e-3fdd-4fa7-a035-9066a98f2590 [ward-4.local:/Users/yozie/server1/./datadictionary]
	539314c2-e5b5-42fa-a251-9bea69e15849 [ward-4.local:/Users/yozie/locator/.]
	50f25e1f-6441-40ac-9bed-5148dd56c71f [ward-4.local:/Users/yozie/locator/./datadictionary]
	74667f34-e800-4313-bb0d-a64415f52beb [ward-4.local:/Users/yozie/server2/./datadictionary]
	1027151d-33ba-4d71-8ec7-24b6a8cff5e1 [ward-4.local:/Users/yozie/server2/.]
Backup successful.

This command creates a full backup of the disk store files in a directory indicating the time that the backup was made. Subdirectories are created for each locator and data store in the distributed system:

$ ls $HOME/gfxd-backups
2014-12-16-16-33-15/
$ ls $HOME/gfxd-backups/2014-12-16-16-33-15
10_84_17_128_25723_ec_v0_16099/ 10_84_17_128_25734_v1_31259/    10_84_17_128_25743_v2_21552/

Each member directory in the backup directory contains a restore script, that you can use to restore the files to a given directory. The restore script performs a test to ensure that it does not overwrite existing disk store files of the same name. If no files of the same name exist, the script copies the backups to the specified directory. This is the restore script that was created for server2:

$ cd $HOME/gfxd-backups/2014-12-16-16-33-15/10_84_17_128_25723_ec_v0_16099/
$ cat restore.sh 
#!/bin/bash -e
cd `dirname $0`

#Restore a backup of gemfire persistent data to the location it was backed up
#from.
#This script will refuse to restore if the original data still exists.

#This script was automatically generated by the gemfire backup utility.

#Test for existing originals. If they exist, do not restore the backup.
test -e '/Users/dyozie/locator/./datadictionary/BACKUPGFXD-DD-DISKSTORE.if' && echo 'Backup not restored. Refusing to overwrite /Users/dyozie/locator/./datadictionary/BACKUPGFXD-DD-DISKSTORE.if' && exit 1 
test -e '/Users/dyozie/locator/./BACKUPGFXD-DEFAULT-DISKSTORE.if' && echo 'Backup not restored. Refusing to overwrite /Users/dyozie/locator/./BACKUPGFXD-DEFAULT-DISKSTORE.if' && exit 1 

#Restore data
mkdir -p '/Users/dyozie/locator/./datadictionary'
cp -rp 'diskstores/GFXD-DD-DISKSTORE_fbefd3dd-2195-4936-b454-f4bcb6175037/dir0'/* '/Users/dyozie/locator/./datadictionary'
mkdir -p '/Users/dyozie/locator/.'
cp -rp 'diskstores/GFXD-DEFAULT-DISKSTORE_1695f17d-b150-4e06-b744-3b24287a10cf/dir0'/* '/Users/dyozie/locator/.'

If you need to move a GemFire XD member to new hardware, an up-to-date restore script can help you copy the disk store files that are necessary for that member. For example, you can emulate moving the "server2" member by first shutting down the server and then renaming (or removing) its working directory:
```
$ gfxd server stop -dir=$HOME/server2
The GemFireXD Server has stopped.
$ mv $HOME/server2 $HOME/deleted-server-2
```
Moving or deleting the directory is necessary in this tutorial, because the restore script checks for the existing server directory before copying the files. To restore the server2, run its associated restore script:
```
$ cd $HOME/gfxd-backups/2014-12-16-16-33-15/10_84_17_128_25743_v2_21552
$ ./restore.sh 
```
You can then restart the server using the restored disk store files:
```
$ gfxd server start -dir=$HOME/server2 -locators=localhost[10101] -client-port=1529 -enable-network-partition-detection=true
Starting GemFireXD Server using locators for peer discovery: localhost[10101]
Starting network server for GemFireXD Server at address localhost/127.0.0.1[1529]
Logs generated in /Users/dyozie/server2/gfxdserver.log
GemFireXD Server pid: 25903 status: running
  Distributed system now has 3 members.
  Other members: 10.84.17.128(25723:locator)<ec><v0>:16099, 10.84.17.128(25734:datastore)<v1>:31259
```
Note that in a production environment, your working directory may contain other artifacts such as automated startup scripts or the gemfirexd.properties file, used to configure boot properties. You would need to backup and restore those files as well as the disk store files for a complete backup solution.