Perforce IPLM High Availability with HAProxy and Neo4j cluster: Deploy HAProxy

Previous step: Perforce IPLM High Availability with HAProxy and Neo4j cluster: Deploy IPLM Server

Deploy HAProxy

HAProxy needs to be deployed on a separate node. It’s recommended to use the haproxy software package that comes with the Linux distribution.

Install HAProxy Software

For RHEL/CentOS distro, use the following command to install haproxy.

yum install haproxy

Configure HAProxy

Delete or rename HAProxy configuration file /etc/haproxy/haproxy.cfg. Edit the file with following content and in sections of neo4j_all and piserver_all, replace Neo4j/PiServer host name and IP with the actual host names and IP addresses.

global
 log /dev/log local0
 stats socket /var/run/haproxy/admin.sock mode 660 level admin
 stats timeout 30s
 user haproxy
 group haproxy
 daemon
 ca-base /etc/ssl/certs
 crt-base /etc/ssl/private
 ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
 ssl-default-bind-options no-sslv3
defaults
 log global
 mode http
 option httplog
 option dontlognull
 timeout connect 5000
 timeout client 10m
 timeout server 10m
 errorfile 400 /usr/share/haproxy/400.http
 errorfile 403 /usr/share/haproxy/403.http
 errorfile 408 /usr/share/haproxy/408.http
 errorfile 500 /usr/share/haproxy/500.http
 errorfile 502 /usr/share/haproxy/502.http
 errorfile 503 /usr/share/haproxy/503.http
 errorfile 504 /usr/share/haproxy/504.http
frontend neo4j-in
 bind *:7474
 default_backend neo4j_all
frontend piserver-in
 bind *:8080
 default_backend piserver_all
backend neo4j_all
 option httpchk HEAD /db/manage/server/ha/available HTTP/1.0
 server <Neo4j node 1 host> <Neo4j node 1 IP>:7474 check port 7474
 server <Neo4j node 2 host> <Neo4j node 2 IP>:7474 check port 7474
 server <Neo4j node 3 host> <Neo4j node 3 IP>:7474 check port 7474
backend piserver_all
 mode http
 balance roundrobin
 option forwardfor
 default-server inter 1s fall 2
 option httpchk HEAD /public/v1/system/info HTTP/1.0
 server <PiServer node hostname> <PiServer node IP>:8080 check
listen admin
  bind *:8081
  stats enable

Create folder /var/run/haproxy.

mkdir -p /var/run/haproxy/
chown haproxy:haproxy /var/run/haproxy/

Some systems (RHEL7 for example) clean up the /var/run folder when the system reboots. Make sure /var/run/haproxy is created when the system reboots.

echo "d /var/lib/haproxy/ 0750 haproxy haproxy - -" > /etc/tmpfiles.d/haproxy.conf

Additional configuration recommendations

Configure Master Nodes and Slave Nodes

Sometimes, it's desired to separate master nodes (read-write nodes) from slave nodes (read-only nodes). To accomplish this, replace the backend neo4j_all section above with the below two sections.

backend neo4j_master
 option httpchk HEAD /db/manage/server/ha/master HTTP/1.0
 server <Neo4j node 1 host> <Neo4j node 1 IP>:7474 check port 7474
 server <Neo4j node 2 host> <Neo4j node 2 IP>:7474 check port 7474
 server <Neo4j node 3 host> <Neo4j node 3 IP>:7474 check port 7474

backend neo4j_slaves
 option httpchk HEAD /db/manage/server/ha/slave HTTP/1.0
 server <Neo4j node 1 host> <Neo4j node 1 IP>:7474 check port 7474
 server <Neo4j node 2 host> <Neo4j node 2 IP>:7474 check port 7474
 server <Neo4j node 3 host> <Neo4j node 3 IP>:7474 check port 7474

And then reference the new sections in frontend neo4j-in,

frontend neo4j-in
bind *:7474
acl is_get method GET
use_backend neo4j_slaves if is_get
default_backend neo4j_master

Add Fallback to Master

If Neo4j cluster nodes are configured as master nodes and slave nodes, we also suggest adding a fallback to the master, to cover the use case where all your slaves are dead causing all the read traffic to fail until at least one slave comes back.

With the modification below, all read traffic will fallback to the master when all the slaves are dead. It’s not ideal but better than nothing assuming the master can handle the load.

#Add an ACL that will test if slaves are available or not
acl neo4j_slaves_dead nbsrv(neo4j_slaves) lt 1
#modify use_backend slave routes to conditionally route to master
use_backend neo4j_slaves if is_get !neo4j_slaves_dead
use_backend neo4j_master if is_get neo4j_slaves_dead

Start HAProxy

To start HAProxy,

service haproxy start

Replace the start argument with stop or status to stop or check the status of the haproxy service.

Monitoring

Open URL http://<HAProxy host>:8081/haproxy?stats to check the status of the all the nodes configured in HAProxy configuration file. Keep refreshing the page to get the latest status info.

HAProxy logs

HAProxy daemon maintains logs for each request coming in and out of the HAProxy node.

On RHEL6/CentOS6, the log file is /var/log/haproxy.log
On RHEL7/CentOS7, the log can be viewed by running following command
- journalctl -u harpy

Protect HAProxy

Ideally, HAProxy should also be protected so that there is no any single point of failure in the Perforce IPLM HA structure. Protecting HAProxy is no different from protecting any other critical network services. It's actually a bit easier to protect HAProxy because data synchronization is not required among multiple HAProxy nodes.

Detailed discussion on protecting HAProxy from being single point of failure is beyond main topic of this document. Below are some of the techniques that are generally used.

Round-Robin DNS - This is simple to implement but requires SysAdmin to set up extra mechanism to promptly remove failed host from DNS name records.
See https://www.linux.com/topic/networking/simple-load-balancing-dns-linux/
Linux Virtual Server - If Active/Backup LVS Routers (communicated via pulse service) have already been set up by IT department, a separate lvs daemon can be configured on Active LVS Router to monitor multiple HAProxy nodes.
See https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/virtual_server_administration/s1-lvs-block-diagram-vsa#s2-lvs-components-VSA
Pacemaker+Corosync - With Pacemaker and Corosync, multiple HAProxy nodes can establish a quorum and a backup node can be configured to automatically become active with same VIP when the active node is down.
See https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/high_availability_add-on_overview/s1-pacemakerarchitecture-haao

It is recommended to first consult your own IT department for suggestions before setting up multiple HAProxy nodes.