IPLM Server backup

Describes the backup methods available to back up the Perforce IPLM database. Backing up Perforce IPLM Server can be done online ("hot") or offline ("cold").

Offline ("cold") Backups

An offline backup is the simplest form and requires PiServer to be shutdown to ensure the db doesn't change during the backup period. Often an offline backup is acceptable since the underlying data-management system isn't affected and users can still work in their workspaces during a PiServer outage. Any Perforce IPLM commands run during this period will prompt the user to retry with no loss of integrity in the workspace.

Note that the IPLM Server database is typically in the order of a few gigabytes in a typical enterprise, which shouldn't take more than a minute or 2 to write on performant local storage.

Once a database has been created we recommend you run a data consistency check to ensure the db is in a useable state

RHEL/CentOS

# Create Backup
su - root
service piserver stop
/usr/share/mdx/neo4j/current/bin/neo4j-admin dump --to `date +"%Y%m%d_%H%M".dump`
service piserver start
 

# Restore from Backup
su - root
cd /var/lib/mdx-neo4j/data/databases/
service piserver stop
mv graph.db graph.db.old
/usr/share/mdx/neo4j/current/bin/neo4j-admin load --from /root/20161207_1214.dump
service piserver start

Debian

# Create Backup
su - root
service neo4j stop; systemctl stop piserver
/usr/share/mdx/neo4j/current/bin/neo4j-admin dump --to `date +"%Y%m%d_%H%M".dump
service neo4j start; systemctl start piserver
 

# Restore from Backup
su - root
cd /var/lib/mdx-neo4j/data/databases/
neo4j stop; systemctl stop piserver
mv graph.db graph.db.old
/usr/share/mdx/neo4j/current/bin/neo4j-admin load --from /root/20161207_1214.dump
service neo4j start; systemctl start piserver

Notes:

the target .bak file should be relocated to a backed-up location on the filesystem (typically an NFS mounted storage location)
a simple tar/gzip of the graph.db directory is not the recommended backup strategy. More details can be found here

Example Cron Script

For convenience an example cron script is included in /usr/share/mdx/products/piextras/bin/piserver_backups.sh (requires installation of the piextras package first)

Online ("hot") backups

Online backups do not require the server to be shutdown first, which is often a requirement with continuous integration and long running nightly regression workflows. There are 4 options for implementing online backups

Logical Volume Management (LVM)

Since the underlying Perforce IPLM database is transactional we can configure the PiServer host to use LVM and take advantage of the snapshot mechanism to enable online backups. More discussion on how to configure a filesystem with LVM is discussed here

Storage Area Network (SAN)

Another option for online backups is to write to a SAN based fileserver and use its snapshot based backup mechanism. NFS mounted storage is considered bad practice for database storage and should be avoided however a SAN interface should give acceptable performance. More information on SAN is available here.

Highly Available (HA)

A third option for online backups is to take advantage of an HA configuration and take one of the servers in the HA pair down for the duration of the backup. A typical HA configuration for PiServer is discussed here

Neo4j Enterprise

A fourth option for online backups is via the Neo4j Enterprise server edition, this has an extra license cost however. Perforce is an OEM supplier of Neo4j enterprise edition and customers interested in this should contact sales@methodics.com. More details here.

Backup Database Integrity

Neo4j includes a database consistency checker that can be used to confirm the integrity of our backup databases.

To use this first we must tar/gunzip our backup to a temp directory

Backup to the Temp Directory

# Assuming our backups are kept in the nfs mounted directory /nfx/mdx/backups
  
> mkdir /tmp/20180409_1156
> cd /tmp/20180409_1156
> tar xvfz /nfs/mdx/backups/20180409_1156.graph.db.tgz

Now we can run the Neo4j consistency checker

Neo4j Consistency Database
                                                        
> /usr/share/mdx/neo4j/current/bin/neo4j-admin check-consistency --backup=/tmp/20180409_1156
 
2018-04-09 17:51:50.777+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] Selected RecordFormat:StandardV3_2[v0.A.8] record format from store /var/lib/mdx-neo4j/data/databases/graph.db
2018-04-09 17:51:50.780+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] Format not configured. Selected format from the store: RecordFormat:StandardV3_2[v0.A.8
.................... 10%
.................... 20%
.................... 30%
.................... 40%
.................... 50%
.................... 60%
.................... 70%
.................... 80%
.....Checking node and relationship counts
.................... 10%
.................... 20%
.................... 30%
.................... 40%
.................... 50%
.................... 60%
.................... 70%
.................... 80%
.................... 90%
.................... 100%
                                                        

If you run into any reported inconsistencies, contact Perforce support at https://www.perforce.com/support.