Perforce IPLM High Availability with HAProxy and Neo4j cluster: Final HA configuration steps and maintenance

Previous step: Perforce IPLM High Availability with HAProxy and Neo4j cluster: Deploy HAProxy

Configure IPLM Client to use IPLM Server HA

In IPLM Client command, configure the IPLM Server target as http://<HAProxy node>:8080.

Perform remote backup for Neo4j HA cluster

It is NOT recommended to use an NFS mounted folder as the backup target location. Backing up directly into an NFS mounted location slows down the backup and is more likely to cause data corruption.

Full backup is needed when doing backup for the first time. The following command does a full backup in /tmp/neo4j-backup  from host <Neo4j node 1>. In a typical LAN environment, it may take 5-10 minutes to do a full backup for a ~6.5GB graph DB.

mkdir /tmp/neo4j-backup
<neo4j-enterprise-3.5.3 install dir>/bin/neo4j-admin backup --backup-dir=/tmp/neo4j-backup --name=graph.db-backup --protocol=common --from=<Neo4j node 1>

Future backups can be done through incremental backups.

<neo4j-enterprise-3.5.3 install dir>/bin/neo4j-admin backup --backup-dir=/tmp/neo4j-backup --name=graph.db-backup --protocol=common --from=<Neo4j node 1>

So the command used to do incremental backups is same as that for doing the full backup. According to Neo4j manual, 

An incremental backup is performed whenever an existing backup directory is specified, and the transaction logs are present since the last backup.

To make sure incremental backup can successfully back up the Neo4j DB even if the backup folder somehow doesn't contain valid backup, add option --fallback-to-full=true  to the command.

 It may still take more than a couple of minutes to do incremental backups even though the DB changes between backups are small. A much quicker backup can be done by adding option --check-consistency=false (not recommended).

Add/Replacing Node

With Neo4j running in cluster mode, a node in the cluster can be replaced without interrupting Neo4j service.

  1. Get a new license file from Perforce for the new node.
    • Run mdx_gather.sh script on the new node and send the request to Perforce.
    • The new license is for the IP/BlockID of the new node.
  2. Load the new license file into cluster.
  3. Prepare the new node.
  4. On the node to be replaced, stop Neo4j service or simply shut down the node(disable Neo4j auto-start before shutdown).
    • On HAProxy dashboard(the stats page), make sure the status of the nodes in cluster are changed as expected.
  5. On the new node, start Neo4j service.
  6. On other nodes in the cluster, one by one, update/restart Neo4j configuration.
    • Update Neo4j config file to replace the old node with the new node in ha.initial_hosts
    • Restart Neo4j configuration
      service neo4j restart
    • Monitor neo4j log and make sure Neo4j comes up properly after service restart.
       journalctl -f -u neo4j # watch for message of Remote interface available
    • On HAProxy dashboard(the stats page), make sure the status of the nodes in cluster are changed as expected
  7. On the new node, monitor neo4j log and make sure service comes up.
    •  journalctl -f -u neo4j # watch for message of Remote interface available

  8. On HAProxy node, update HAProxy configuration to replace old node with new node, and reload configuration.

    • service haproxy reload
    • On the HAProxy dashboard (the stats page), make sure the status of the nodes in cluster are changed as expected

Rollback

To rollback IPLM Server and Neo4j cluster to an older version of Pi release, both IPLM Server and Neo4j cluster must be shut down during the rollback. Rollbacks are not always possible because of potential underlying schema changes between versions. Check with Perforce support (https://www.perforce.com/support) to make sure you can safely rollback from your current version to the targeted older version.

The commands below are for RHEL7/CenOS7 platforms. For other platforms, refer to the sections above and the Deployment Guide of the corresponding platform.

  1. Stop IPLM Server and Neo4j cluster.
    • On each IPLM Server node,
      service piserver stop
    • On each Neo4j node in the Neo4j cluster,
      service neo4j stop
  2. Backup the current Neo4j DB.
    Assume /methodics is the backup folder on each node,
    • On each IPLM Server node,
      cd /methodics
      cp -a /etc/mdx/piserver.conf .
      cp -a /etc/mdx/piserver.properties .
      cp -a /etc/mdx/piserver.yml .
    • On each Neo4j node in the Neo4j cluster,
      cd /methodics
      cp -a /var/lib/neo4j .
      cp -a /etc/neo4j/neo4j.conf .
  3. Remove current install of IPLM Server and IPLM Server Neo4j plugin.
    •  On each IPLM Server node,
        rpm -e mdx-piserver mdx-piserver-tools
        rpm -qa | grep mdx  # make sure IPLM Server packages are removed
    • On each Neo4j node in the Neo4j cluster,
       rpm -e mdx-piserer-neo4j-plugin
       rpm -qa | grep mdx  # make sure IPLM Server plugin package is removed
  4. Install IPLM Server and IPLM Server Neo4j plugin of the older version of Pi release.
    • On each IPLM Server node,
      yum install mdx-piserver-tools-<older release of Pi>
      /usr/share/mdx/products/piserver/bin/install-rpm-deps.sh
      yum install mdx-piserver-<older release of Pi>
    • On each Neo4j node in the Neo4j cluster,
       yum install mdx-piserer-neo4j-plugin-<older release of Pi>
  5. Install Neo4j DB for the older Pi release.
    • Use the Neo4j DB saved above(or backed up through other means) and follow the steps in the section of Migrate old Neo4j DB to Neo4j cluster to install old Neo4j DB
  6. Review IPLM Server and Neo4j configurations, and make changes if needed.
    • On each IPLM Server node,
      diff /methodics/piserver.conf /etc/mdx/piserver.conf  # edit /etc/mdx/piserver.conf as needed
       diff /methodics/piserver.properties /etc/mdx/piserver.properties  # edit  /etc/mdx/piserver.properties as needed
       diff /methodics/piserver.yml /etc/mdx/piserver.yml  # edit /etc/mdx/piserver.yml as needed
    • On each Neo4j node in the Neo4j cluster,
        diff /methodics/neo4j.conf /etc/neo4j/neo4j.conf  # edit /etc/neo4j/neo4j.conf as needed
  7. Start Neo4j cluster.
    • On each Neo4j node in the Neo4j cluster,
      service neo4j start
  8. Start IPLM Server.
    • On each IPLM Server node,
      service piserver start