Configuring IPLM Cache for High Availability

IPLM Cache can be configured to run in High Availability mode. Details of this configuration are described below.

IPLM Cache High Availability

IPLM Cache Server can be configured for High Availability (HA). Multiple instances of each component can be started simultaneously on different nodes, and built-in HA support will make sure that there's no single point of failure in any of the components.

IPLM Cache uses a queuing mechanism for Perforce jobs, where a PWM client, like Pi Client, connects to IPLM Cache's Front API component that enqueues perforce jobs into Redis list data structures, and then IPLM Cache Worker processes dequeue the jobs from Redis and execute them. In this manner, multiple IPLM Cache nodes can work in parallel, independently of each other, accessing the same Redis instance to get available jobs. IPLM Cache Server instances have the ability to monitor peer IPLM Cache Server instances and if a peer is seen as down (unavailable) the jobs it was working on when it went down are requeued in Redis by the monitoring IPLM Cache Server instance. In addition to installing the mdx-backend-redis, mdx-backend-sentinel, mdx-backend-mongodb, and mdx-picache packages on a IPLM Cache Server node, the Linux Watchdog Daemon should also be installed for an HA deployment.

Detecting a node that's down is done through IPLM Cache's Watchdog Monitor component, which registers its heartbeat status in a Redis key. Each IPLM Cache Server node has its own Watchdog Monitor component and its own associated heartbeat Redis key. IPLM Cache Server processes register with the Watchdog Monitor on the same node and then periodically ping it to indicate they are healthy. If a process misses an expected ping, the Watchdog Monitor indicates the node is unhealthy and the Linux Watchdog Daemon will reboot the node. At that time, or after the node has been seen as unresponsive for a configured duration, a peer IPLM Cache node will requeue the downed node's jobs that were in progress. A monitoring IPLM Cache Server instance will requeue a downed peer's jobs only if the monitoring instance has a Watchdog Monitor heartbeat Redis key and if that key indicates the monitoring node is healthy,

A IPLM Cache Server can also be configured to requeue its own jobs when it is to terminate and when it starts up (i.e., requeue any "stuck" jobs when the server went down previously).

HA Configuration Options

The following IPLM Cache configuration items are HA related:

[main] Section
- shutdown-worker-control should be set to requeue. This will cause IPLM Cache to requeue its in-progress jobs when it terminates and when it starts up, requeuing downed jobs in Redis when it previously went down and could not requeue its jobs for some reason.
- If Redis is in an HA configuration:
  - Comment out redis-host
  - Uncomment redis-sentinel-instances and edit the line to give the IP addresses and Redis Sentinel port of the nodes running the Redis Sentinel Server instances
  - Uncomment redis-sentinel-master and set it to the name of the Redis Sentinel master service (typically mymaster)
- If MongoDB is in an HA configuration:
  - Comment out the single host:port pair for mongod-host
  - Uncomment the three host:port pairs for mongod-host and edit the line to give the IP addresses and MongoDB port of the nodes running the MongoDB Server instances
  - Uncomment mongod-rs and set it to the name of the MongoDB replication set (typically rs0)
- heartbeat-interval (in seconds) should be set to a non-zero positive value. A value of 10 has been seen to be reasonable. This controls how often the IPLM Cache instance monitors peer instances.
- heartbeat-loss-max should be set to something reasonable, like 5. Should a peer not go down but becomes unresponsive (no heartbeat updates) for a duration of heartbeat-interval times heartbeat-loss-max the IPLM Cache instance will infer the peer node is down and requeue that node's in-progress jobs.
[watchdog] Section
- ping-period should be set to something reasonable, like 20. This is the maximum period between pings from registered clients, unless overridden by client.

Additional HA Deployment Information and Configuration Files

Redis

See Events Administration for information on deploying Redis in an HA configuration with the mdx-backend-redis and mdx-backend-sentinel packages.

Also refer to Redis documentation on the various other directives that you may need in your Redis configuration files, for example

bind: by default, if no "bind" configuration directive is specified, Redis listens for connections from all the network interfaces available on the server, which could be a security concern if the Redis instance is directly exposed to the internet. Use bind to explicitly set the interface(s) the Redis server is to bind to.
port: the default port Redis uses is 6379. Use the port directive to override this value.
min-replicas-to-write and min-replicas-max-lag: by default, the master Redis instance will continue to accept writes even without any slave instances to replicate to. This could cause a problem if the master is then lost or stopped and a slave instance later becomes available, with a different set of keys, and is promoted to master. As discussed in the Redis Sentinel Documentation, to help prevent this scenario ensure the Redis Server and Redis Sentinel instances are in machines that are believed to fail in an independent way and the min-replicas-to-write and min-replicas-max-lag directives can be used so the master will stop accepting writes if there are not enough replicas available after a certain number of seconds.

MongoDB

As of IPLM Cache release v1.6.0, use of MongoDB by IPLM Cache is optional. See picache.conf Configuration file for the mongod-host setting.

To deploy MongoDB in an HA configuration, follow the instructions in the appropriate Methodics package installation section to install, but not yet run, the mdx-backend-mongodb package on each node of the MongoDB cluster using apt-get, yum, or zypper.

mdx-backend-mongodb installs the MongoDB binaries within /usr/share/mdx/mongodb. As of this writing, the MongoDB version packaged is v3.2.20 and the MongoDB binaries are located in /usr/share/mdx/mongodb/mongodb-linux-x86_64-3.2.20/bin.
The MongoDB shell utility, mongo, is also in the binaries directory, so to use this (as is done below) your shell's PATH environment variable needs to reference the binaries directory or you need to use the full path to mongo.
mdx-backend-mongodb installs a supplementary script in /usr/share/mdx/mongodb/scripts that is used by the mdx-backend-mongodb service.
The MongoDB configuration file is installed at /etc/mdx/mdx-backend-mongodb.conf.

Afterwards, edit /etc/mdx/mdx-backend-mongodb.conf as follows:

In the net section:
- Edit the line bindIp by adding an externally accessible interface to it, and in the unixDomainSocket subsection, set enabled to false, for example
  Editing bindIP
```
net:
   bindIp: 127.0.0.1,10.211.55.6
   port: 27017
   unixDomainSocket:
      enabled: false
      pathPrefix: "/var/run/mdx-backend-mongodb"
```
Add a new replication section giving a replication set name of rs0:
Adding New Replication Section
```
replication:
   replSetName: rs0
```
If the MongoDB configuration file is changed while the mdx-backend-mongodb service is running, restart the mdx-backend-mongodb service in order for the MongoDB Server to get the updated configuration:
Restart the mdx-backend-mongodb Service
```
# service mdx-backend-mongodb restart
```

The first time the MongoDB instances are started, the replica set needs to be initiated, which can be done by logging onto one of the node's (and only one of the node's) MongoDB shell and executing the rs.initiate command, referencing the three hosts running the MongoDB instances, for example:

Referencing the Three Hosts Running the MongoDB Instances

$ mongo 10.211.55.6
> rs.initiate({
      _id: "rs0",
       members: [
         {_id: 0, host: "centos6-picache1:27017"},
         {_id: 1, host: "centos6-picache2:27017"},
         {_id: 2, host: "centos6-picache3:27017"}
       ]
  })

Shortly after this point, one of the nodes will become the Primary MongoDB instance and the other two Secondary MongoDB instances.