IPLM Cache logging

See the log files available with IPLM Cache and how to access the logging information.

Accessing IPLM Cache logs and debugging information

IPLM Cache server log files

There are a several log files associated with the IPLM Cache Server:

  • http logs showing interaction with the clients:

    • /var/log/mdx-picache/picache-access.log

    • /var/log/mdx-picache/picache-error.log

  • Main worker log (mainly for IPLM Cache server startup/shutdown)

    • /var/log/mdx-picache/picache.log

See the picache.conf Configuration file page for the log-related configuration items: log-file-backup-count, and log-file-maxbytes. 

MongoDB logs

IPLM Cache Worker processes and its Maintenance and HA Node Handling processes log messages to the MongoDB database when the configuration item log-target-backend is set to mongodb.

See the picache.conf Configuration file page for the MongoDB-related configuration items: log-target-backend, log-timestamp-utc, log-mongodb-ttl-days, mongod-logs-write-ack, mongod-host, and mongod-rs. 

The logs will expire in 30 days, depending on the log-mongodb-ttl-days setting, and the associated space will be reused by MongoDB database.

To extract logs from MongoDB, use the picache-query-mongo-log.sh tool, as of v1.6.0, or the query_mongo_log.pyc tool for versions prior to v1.6.0:

Extracting logs

$ /usr/share/mdx/products/picache/bin/picache-query-mongo-log.sh --help
or
$ /usr/share/mdx/products/picache/local/bin/python /usr/share/mdx/products/picache/local/lib/python2.7/site-packages/methodics/picache_server/tools/query_mongo_log.pyc --help
usage: query_mongo_log.py [-h] [-c CONF] [-e [PATTERN [PATTERN ...]]]
                          [-f [FUNCTION [FUNCTION ...]]] [-i [IP [IP ...]]]
                          [-j [JOBID [JOBID ...]]]
                          [--level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                          [-L [LOGNAME [LOGNAME ...]]]
                          [-m [MODULE [MODULE ...]]] [-n [NODE [NODE ...]]]
                          [-p [PID [PID ...]]] [-t MINTIME] [-T MAXTIME] [-v]
                          [--count] [--exc]

Tool for extracting IPLM Cache Server logs from MongoDB.

optional arguments:
  -h, --help            show this help message and exit
  -c CONF, --conf CONF  IPLM Cache config file, default: /etc/mdx/picached.conf
  -e [PATTERN [PATTERN ...]], --regexp [PATTERN [PATTERN ...]]
                        Regular expressions to search log records for
  -f [FUNCTION [FUNCTION ...]], --function [FUNCTION [FUNCTION ...]]
                        Functions log records are for
  -i [IP [IP ...]], --ip [IP [IP ...]]
                        IPs log records are for
  -j [JOBID [JOBID ...]], --jobid [JOBID [JOBID ...]]
                        Job IDs log records are for
  --level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Log level (and above) log records have
  -L [LOGNAME [LOGNAME ...]], --logname [LOGNAME [LOGNAME ...]]
                        Logger names log records are for
  -m [MODULE [MODULE ...]], --module [MODULE [MODULE ...]]
                        Modules log records are for
  -n [NODE [NODE ...]], --node [NODE [NODE ...]]
                        Nodes (hostnames) log records are from
  -p [PID [PID ...]], --pid [PID [PID ...]]
                        Process IDs log records are for
  -t MINTIME, --mintime MINTIME
                        Gets log records newer than or equal to this datetime
  -T MAXTIME, --maxtime MAXTIME
                        Gets log records older than or equal to this datetime
  -v, --verbose         Verbose output
  --count               Prints the number of log records and exits
  --exc                 Gets log records only showing exceptions

Use an egrep expression for a logical OR search of the logs, for example:
  -e 'job_build|job_update'
will get log records containing 'job_build' or 'job_update'.
Use multiple expressions for a logical AND search of the logs, for example:
  -e 'enter' 'job_build'
will get log records containing both 'enter' and 'job_build'.
MINTIME and MAXTIME are formatted as '2018-07-02T14:59:13.365'
(fields 'T' and after are optional).

For the LOGNAME values, the following can be used:

  • mdx.picache for the Worker Processes
  • mdx.picache.maintenance for the Maintenance Process
  • mdx.picache.ha_node_handler for the HA Node Handling Process

Rotate mongod.log when needed

Over time, the MongoDB daemon log file, /var/log/mdx-backend-mongodb/mongod.log, may become very big. It's recommended to rotate the file once a month. To rotate the log file, with the MongoDB daemon still running, run the command below:

sudo kill -SIGUSR1 $(</var/run/mdx-backend-mongodb/mongod.pid)

The log file will be rotated and saved as /var/log/mdx-backend-mongodb/mongod.log.<current datetime>. The saved log file can be simply removed in future. 

Recommended mdx-backend-mongodb.conf modifications

Setting the parameters verbosity and quiet to the settings below can reduce the size of the log files to prevent excessive output.

systemLog:
     destination: file
     path: "/var/log/mdx-backend-mongodb/mongod.log"
     logAppend: true
     verbosity: 0
     quiet: true
storage:
     dbPath: "/var/log/mdx-backend-mongodb/mongodb"
     journal:
          enabled: true
processManagement:
     fork: true
     pidFilePath: "/var/run/mdx-backend-mongodb/mongod.pid"
net:
     bindIp: 127.0.0.1
     port: 27017
     unixDomainSocket:
          enabled: true
          pathPrefix: "/var/run/mdx-backend-mongodb"

Syslog logging

Instead of being logged into MongoDB, IPLM Cache Worker processes and its Maintenance and HA Node Handling processes can be configured to log messages to a Syslog server when the configuration item log-target-backend is set to syslog.

See the picache.conf Configuration file page for the Syslog-related configuration items: log-target-backend, syslog-address, syslog-socktype, mongod-logs-write-ack, and syslog-facility. 

IPV information

To get information about all the IPVs in the cache (MongoDB must be used), use the picache-ipv-usage.sh tool, as of v1.6.0, or the ipv_usage.pyc tool for versions prior to v1.6.0:

/usr/share/mdx/products/picache/bin/picache-ipv-usage.sh

or

/usr/share/mdx/products/picache/local/bin/python /usr/share/mdx/products/picache/local/lib/python2.7/site-packages/methodics/picache_server/ipv_usage.pyc

Job information

To get information about recent jobs (MongoDB must be used), use the picache-ipv-logger.sh tool, as of v1.6.0, or the ipv_logger.pyc tool for versions prior to v1.6.0:

/usr/share/mdx/products/picache/bin/picache-ipv-logger.sh

or

/usr/share/mdx/products/picache/local/bin/python /usr/share/mdx/products/picache/local/lib/python2.7/site-packages/methodics/picache_server/ipv_logger.pyc

To get information about a specific job (MongoDB must be used):

/usr/share/mdx/products/picache/bin/picache-ipv-logger.sh -j <job ID>

or

/usr/share/mdx/products/picache/local/bin/python /usr/share/mdx/products/picache/local/lib/python2.7/site-packages/methodics/picache_server/ipv_logger.pyc -j <job ID>

To get job information in the job queues (MongoDB must be used):

/usr/share/mdx/products/picache/bin/picache-ipv-logger.sh -j mdx.picache
or

/usr/share/mdx/products/picache/local/bin/python /usr/share/mdx/products/picache/local/lib/python2.7/site-packages/methodics/picache_server/ipv_logger.pyc -j mdx.picache

To get information about jobs processed by the maintenance process (MongoDB must be used):

/usr/share/mdx/products/picache/bin/picache-ipv-logger.sh -j mdx.picache.maintenance
or

/usr/share/mdx/products/picache/local/bin/python /usr/share/mdx/products/picache/local/lib/python2.7/site-packages/methodics/picache_server/ipv_logger.pyc -j mdx.picache.maintenance