Solve common problems may be encountered using GemFire XD.
GemFire XD Locator pid: 23537 status: waiting Waiting for DataDictionary (DiskId: 531fc5bb-1720-4836-a468-3d738a21af63, Location: /pivotal/locator2/./datadictionary) on: [DiskId: aa77785a-0f03-4441-84f7-6eb6547d7833, Location: /pivotal/server1/./datadictionary] [DiskId: f417704b-fff4-4b99-81a2-75576d673547, Location: /pivotal/locator1/./datadictionary]Here, the startup messages indicate that locator2 is waiting for the persistent datadictionary files on locator1 and server1 to become available. GemFire XD always persists the data dictionary for indexes and tables that you create, even if you do not configure those tables to persist their stored data. The startup messages above indicate that the locator2 member was shut down before it could gracefully shut down in the a distributed system consisting of itself, locator1, and server2, and that locator1 or locator2 might potentially store a newer copy of the data dictionary for the distributed system.
Starting GemFire XD Server using locators for peer discovery: localhost,localhost Starting network server for GemFire XD Server at address localhost/127.0.0.1 Logs generated in /pivotal/server1/gfxdserver.log The server is still starting. 15 seconds have elapsed since the last log message: Region /_DDL_STMTS_META_REGION has potentially stale data. It is waiting for another member to recover the latest data. My persistent id: DiskStore ID: aa77785a-0f03-4441-84f7-6eb6547d7833 Name: Location: /10.0.1.31:/pivotal/server1/./datadictionary Members with potentially new data: [ DiskStore ID: f417704b-fff4-4b99-81a2-75576d673547 Name: Location: /10.0.1.31:/pivotal/locator1/./datadictionary ] Use the "gfxd list-missing-disk-stores" command to see all disk stores that are being waited on by other members.The data store startup messages indicate that locator1 has "potentially new data" for the data dictionary. In this case, both locator2 and server1 were shut down before locator1 in the system, so those members are waiting on locator1 to ensure that they have the latest version of the data dictionary.
The above messages for data stores and locators can be commonplace when individual members are shut down one-by-one rather than by using gfxd shut-down-all, which allows all members to synchronize and shut down gracefully. However, if the indicated disk store persistence files are available on the missing member, simply start that member and allow the running members to recover. For example, in the above system you would simply start locator1 and allow locator2 and server1 to synchronize their data.
ConflictingPersistentDataException: Region /_DDL_STMTS_META_REGION remote member curwen(23695)<v1>:4505 with persistent data /10.0.1.31:/pivotal/locator1/./datadictionary created at timestamp 1373667883741 version 0 diskStoreId 9cf5aea67c6c4374-9d7205f72fecd47c name was not part of the same distributed system as the local data from /10.0.1.31:/pivotal/server1/./datadictionary created at timestamp 1373649392112 version 0 diskStoreId aa77785a0f034441-84f76eb6547d7833 name - See log file for details.If the datadictionary directory is deleted or moved, then GemFire XD creates a new data dictionary upon startup of that member. (Remember, all GemFire XD locators and data stores maintain a persistent data dictionary for the distributed system, even if you do not persist data in the tables.) However, other members in the distributed system may expect the locator or data store to have the previous, deleted version of the datadictionary, in order to recover more recent operations. If this occurs, the newly-created data dictionary conflicts with other members view of the distributed system, and new members that startup throw a ConflictingPersistendDataException.
If you cannot resolve startup problems associated with missing or conflicting data dictionary files, you can force the GemFire XD member to complete its startup by using the gemfirexd.datadictionary.allow-startup-errors property. This property enables you to startup a GemFire XD member even if the volume or directory in which a disk store was created no longer exists; you can recreate the disk store manually, after forcing the member to restart.
This problem can be caused if you specify null values for the username and password connection properties in the JDBC connection URL. Some third-party tools specify automatically supply null values but include the connection properties if you do not specify user credentials.
If authentication is disabled in your distributed system, then you can specify any temporary user name and password value when connecting. Connecting to GemFire XD with JDBC Tools provides more details.
It is important to monitor the disk usage of GemFire XD members. If a member lacks sufficient disk space for a disk store, the member attempts to shut down the disk store and its associated tables, and logs an error message. After you make sufficient disk space available to the member, you can restart the member. (See Member Startup Problems.) A shutdown due to a member running out of disk space can cause loss of data, data file corruption, log file corruption and other error conditions that can negatively impact your applications.
You can configure the log message frequency with the gemfire.DISKSPACE_WARNING_INTERVAL system property.
If a member of your GemFire XD distributed system fails due to a disk full error condition, add or make additional disk capacity available and attempt to restart the member normally. If the member does not restart and there is a redundant copy of its tables in a disk store on another member, you can restore the member using the following steps: