IPLM Cache administration tools

IPLM Cache comes with tools that are helpful for monitoring and troubleshooting. IPLM Cache uses the MongoDB document storage database to store log data and IPV usage metadata. IPLM Cache uses the Redis database as a job queue, in which the front-end enqueues jobs that the backend dequeues and executes. Most of the tools operate on either MongoDB or Redis.

To configure logging to use Syslog, see the log-target-backend and the various syslog-related settings in the picache.conf Configuration file page.

The tools are installed in the /usr/share/mdx/products/picache/bin directory:

picache-db-admin.sh, for exporting the IPLM Cache database from MongoDB and for pruning IPLM Cache logs in MongoDB
picache-ipv-admin.sh, for managing IPVs in the cache
picache-ipv-logger.sh, for extracting IPV logs in MongoDB (mostly replaced by picache-query-mongo-log.sh)
picache-ipv-usage.sh, for extracting IPV metadata from MongoDB (mostly replaced by picache-ipv-admin.sh)
picache-job-purge.sh, for deleting all jobs from IPLM Cache Redis queues
picache-monitor.sh, for monitoring IPLM Cache
picache-query-job.sh, for obtaining job information
picache-query-mongo-log.sh, for extracting logs from MongoDB
picache-query-queue.sh, for printing the job IDs in queues
picache-wdog-unregister.sh, for unregistering clients from the IPLM Cache Watchdog Monitor

These are all bash script wrappers for the underlying tool implementations that reside in /usr/share/mdx/products/picache/local/lib/python2.7/site-packages/methodics/picache_server and /usr/share/mdx/products/picache/local/lib/python2.7/site-packages/methodics/picache_server/tools.

To easily access the tools, add /usr/share/mdx/products/picache/bin to your shell's PATH environment variable.

picache-db-admin.sh

The picache-db-admin.sh tool is used to prune (remove) logs from the MongoDB database and to save the MongoDB database to a tar file.

$ picache-db-admin.sh --help
usage: picache-db-admin.sh [-h] (--export | --prune) [-c CONF]
                           [--all | --age AGE] [-v] [-m MONGODB_DIR]

IPLM Cache administration tool for exporting IPLM Cache information from MongoDB and for pruning IPLM Cache logs in MongoDB.

optional arguments:
  -h, --help            show this help message and exit
  --export              export logs and usage collections to picache-db-export.tar.gz
  --prune               prune the logs collection, use --all or --age
  -c CONF, --conf CONF  IPLM Cache config file, default: /etc/mdx/picache.conf
  --all                 when --prune is used, delete all records in logs collection
  --age AGE             when --prune is used, delete records in logs collection older than AGE
  -v, --verbose         verbose output
  -m MONGODB_DIR, --mongodb-dir MONGODB_DIR
                        MongoDB directory, default: /usr/share/mdx/mongodb

AGE is formatted as '2018-07-02T14:59:13.365'
(fields 'T' and after are optional).

In the picache.conf configuration file, /etc/mdx/picache.conf, there are several options related to the size of MongoDB logs:

debug-enable, when set to yes IPLM Cache does verbose logging, typically used to troubleshoot a particular issue
log-mongodb-ttl-days, gives the Time-To-Live, in days, for IPLM Cache logs, after which time a log will automatically be removed from the MongoDB database

The prune option is useful for removing unneeded logs and reducing the size of the MongoDB database. To remove all logs, for example when transitioning from an evaluation phase to a production phase, the --prune option can be used with the --all option, for example:

$ picache-db-admin.sh --prune --all

Number of log records before pruning: 90548

Number of log records to delete: 90548

Number of log records after pruning: 0

To remove logs older than a certain age, the --prune option is used with the --age option.

The --export option is used to save the MongoDB IPLM Cache database into a compressed tar file.

picache-ipv-admin.sh

The picache-ipv-admin.sh tool is used to list the IPVs in the cache, remove one or more IPVs from the cache, perform a cache integrity check, and edit certain characteristics of one or more IPVs for IPLM Cache use.

$ picache-ipv-admin.sh --help
usage: picache-ipv-admin.sh [-h] [-v] [-c CONF] [--project PROJECT] [--all]
                            [--colorize]
                            (--list | --remove | --expire | --no-expire | --clear-consistency-check | --fix-aliases | --integrity-check)
                            [-t TIMEOUT]
                            [ipv [ipv ...]]

IPLM Cache administration tool for managing IPVs in the cache.

positional arguments:
  ipv one or more IPV names

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         verbose output
  -c CONF, --conf CONF  IPLM Cache config file, default: /etc/mdx/picache.conf
  --project PROJECT     IPV project
  --all                 select all IPVs recorded in cache
  --colorize            use with --integrity-check to give green (pass), red (fail), yellow (skip) text colors
  --list                list IPV in the cache
  --remove              remove IPV from cache
  --expire              mark IPV subject to auto-removal
  --no-expire           mark IPV to not be subject to auto-removal
  --clear-consistency-check
                        clear IPV data consistency check in progress indication
  --fix-aliases         replace non-HEAD aliases in IPV names with corresponding version number
  --integrity-check     P4 DM type only: verify cache contents with PiServer, MongoDB, p4 have, file system, and data consistency check
  -t TIMEOUT, --timeout TIMEOUT
                        number of seconds to wait for a remove or check job to finish, default: 300

For example, to get a listing of all IPVs in the cache, the picache-ipv-admin.sh can be used as:

$ picache-ipv-admin.sh --list --all

tutorial.MS90G@1.TRUNK
tutorial.io5v@1.TRUNK
tutorial.io_tsmc18@1.TRUNK
tutorial.padring@1.L1

Use -v/--verbose to get full info for each IPV

Number of IPVs in cache: 4

The --expire and --no-expire options refer to IPLM Cache's IPV Automatic Cleanup feature. In picache.conf, if the ipv-cleanup setting is set to True, use picache-ipv-admin.sh with the --no-expire option to configure one or more IPVs to not be subject to being automatically cleaned up so they remain in the cache until manually removed. Use the --expire option to reverse the expiration setting to configure one or more IPVs back to being subject to auto-removal.

The --clear-consistency-check option refers to IPLM Cache's Data Consistency Check feature. In picache.conf, if the cache-check-period-hrs setting is non-zero, under an exceptional condition it may be that IPLM Cache was stopped with one or more IPVs undergoing data consistency checking. Use picache-ipv-admin.sh with the --clear-consistency-check option to reset the check so the IPVs can later undergo data consistency checking again.

To remove an IPV from the cache, the IPV Automatic Cleanup feature can be used or the pi-admin picache remove command can be used. However, if automatic cleanup is disabled and one or more versions of an IP are loaded into the cache and then the IP is renamed, pi-admin picache remove will not remove the previously named IPVs because the previous name is no longer known by PiServer. For this situation, picache-ipv-admin.sh --remove can be used to remove the IPVs with the previous name.

IPLM Cache's treatment of IPVs using aliases was corrected in IPLM Cache Version 1.8.4, in which aliases, other than the alias HEAD, are replaced with the specific version number the alias corresponds to. For IPVs loaded previous to v1.8.4, the --fix-aliases option can be used to change the names of the IPVs in IPLM Cache's MongoDB database to match how IPLM Cache v1.8.4 and later would store the IPV names. Doing this or not will not affect how the IPV with an alias is treated by IPLM Cache; using --fix-aliases is only for logging and presentation purposes, making the IPV names with aliases consistent with IPV names stored by IPLM Cache v1.8.4 and later.

Cache integrity checking feature

IPLM Cache Version 1.9.1 introduced the picache-ipv-admin.sh cache integrity check feature with the --integrity-check and --colorize options. The integrity check is only done for P4 Data Manager type IPVs. Five integrity checks are performed:

Comparing the PiServer file list for an IPV and the IPV's file list as stored in IPLM Cache's MongoDB database
Comparing the PiServer file list for an IPV and the file list as returned by the p4 have command run in the IPV's cache directory
Comparing the file list as stored in IPLM Cache's MongoDB database for an IPV and the file list as returned by the p4 have command run in the IPV's cache directory
Comparing the file list as returned by the p4 have command run in the IPV's cache directory with the file list contents actually in the IPV's cache directory, looking for removed files
Performing a Data Consistency Check on the IPV's cache directory, as described in Data Consistency Checking

picache-ipv-admin.sh exits with a value of one if any integrity check can be run, is run, and fails; otherwise it will exit with a value of zero. If an integrity check cannot be performed, for example if PiServer cannot be connected to, PiServer does not return a file list for an IPV, or the IPV's file list is not stored in MongoDB, that will not by itself cause picache-ipv-admin.sh to exit with a value of one.

A successful integrity check outputs the text PASS ipv-name plus any project information and the integrity check run. A failed integrity check outputs the text FAIL ipv-name plus any project information and the integrity check run. A skipped integrity check (a check that could not be run) outputs the text SKIP ipv-name plus any project information and the reason the check was skipped. A customer script using picache-ipv-admin.sh integrity check feature can use the exit value and output of picache-ipv-admin.sh as needed for its activities.

This integrity check feature cannot be used as part of a IPLM Cache post-build hook because this hook is called after the cache is populated but before the IPV usage information is stored in MongoDB and picache-ipv-admin.sh requires the MongoDB IPV usage information. However, it is possible to use this feature as part of a Library's post_load and post_update hook. See Managing IP Libraries.

Other integrity check usage notes:

MongoDB must be used so that picache-ipv-admin.sh can extract IPV usage information for the IPV(s) being checked
The pi-server and pi-server-credentials-file settings in picache.conf must be used so that picache-ipv-admin.sh can retrieve the file list for the IPV(s) being checked from a PiServer
An integrity check on a HEAD IPV can be performed, but since there is no file list stored in MongoDB for a HEAD IPV, the PiServer and MongoDB file lists check and the MongoDB and p4 have file lists check will both the skipped. And the PiServer and p4 have file lists check may or may not pass, depending on if there have been recent changes to the HEAD IPV not in the latest release of the IPV.
Much like Pi Client, picache-ipv-admin.sh depends on certain environment variables to be set to successfully run the integrity checks, specifically to run the p4 have command:
- PATH must contain the path to the Perforce p4 client executable
- P4PORT must be set to Perforce Helix server access information. See Configuring Perforce.
- P4CONFIG must be set to .p4config to match IPLM Cache's storage of the P4CONFIG file. See Configuring Perforce.
The --colorize option is used for presentation only. If --colorize is used, when picache-ipv-admin.sh outputs to a terminal, passed integrity checks show in green text, failed integrity checks show in red text, and skipped integrity checks show in yellow text. This is so they are obvious to the end-user running the check. If the output to picache-ipv-admin.sh is redirected to a file, colored text is not output, whether --colorize is used or not.

An example run is:

$ picache-ipv-admin.sh --integrity-check --all

Performing integrity check for tutorial.MS90G@HEAD.TRUNK. For large IPV this could take a while.
Comparing PiServer and MongoDB file lists...
Getting file list from PiServer...
Getting file list from MongoDB...
SKIP tutorial.MS90G@HEAD.TRUNK: PiServer and MongoDB file lists comparison skipped, no file list stored in MongoDB
Comparing PiServer and p4 have file lists...
Getting p4 have file list...
Performing comparison...
PASS tutorial.MS90G@HEAD.TRUNK: PiServer used file list is the same as p4 have file list
Comparing MongoDB and p4 have file lists...
SKIP tutorial.MS90G@HEAD.TRUNK: MongoDB and p4 have file lists comparison skipped, no file list stored in MongoDB
Comparing p4 have and file system file lists...
Performing comparison...
PASS tutorial.MS90G@HEAD.TRUNK: p4 have file list is the same as file system file list
Performing data consistency check...
SKIP tutorial.MS90G@HEAD.TRUNK: Data consistency check skipped, no file list

Performing integrity check for tutorial.verif_config@2.TRUNK. For large IPV this could take a while.
Comparing PiServer and MongoDB file lists...
Getting file list from PiServer...
Getting file list from MongoDB...
Performing comparison...
PASS tutorial.verif_config@2.TRUNK: PiServer file list is the same as MongoDB file list
Comparing PiServer and p4 have file lists...
Getting p4 have file list...
Performing comparison...
PASS tutorial.verif_config@2.TRUNK: PiServer used file list is the same as p4 have file list
Comparing MongoDB and p4 have file lists...
Removing unused files from MongoDB file list using PiServer file list...
PASS tutorial.verif_config@2.TRUNK: MongoDB used file list is the same as p4 have file list
Comparing p4 have and file system file lists...
Performing comparison...
PASS tutorial.verif_config@2.TRUNK: p4 have file list is the same as file system file list
Performing data consistency check...
Enqueuing check job...
Waiting for check job to finish...
PASS tutorial.verif_config@2.TRUNK: data consistency check passed

picache-ipv-logger.sh

The picache-ipv-logger.sh tool is used to extract logs from MongoDB.

picache-ipv-logger.sh

$ picache-ipv-logger.sh -h
usage: picache-ipv-logger.sh [-h] [-c CONF] [-t TEST] [-j JOB_ID] [-n NUM]

PiCache administration tool for showing IPV logs.

optional arguments:
  -h, --help            show this help message and exit
  -c CONF, --conf CONF  PiCache config file, default: /etc/mdx/picache.conf
  -t TEST, --test TEST  test logging using host TEST then exit
  -j JOB_ID, --job-id JOB_ID
                        job ID to get logs for on localhost
  -n NUM, --num NUM     number of log messages to output (from end)

With no arguments, all the log messages for the last 10 jobs are output.

This tool tries to connect to MongoDB on localhost on port 27017.

It is recommended to use the picache-query-mongo-log.sh tool, however, picache-ipv-logger.sh is handy for a quick check on what PiCache has recently been working on.

This tool tries to connect to MongoDB on localhost on port 27017.

It is recommended to use the picache-query-mongo-log.sh tool, however, picache-ipv-logger.sh is handy for a quick check on what IPLM Cache has recently been working on.

picache-ipv-usage.sh

The picache-ipv-usage.sh tool is used to extract IPV metadata from MongoDB.

picache-ipv-usage.sh

$ picache-ipv-usage.sh --help
usage: picache-ipv-usage.sh [-h] [-c CONF] [-v]

PiCache administration tool for showing the IPVs in the cache.

optional arguments:
  -h, --help            show this help message and exit
  -c CONF, --conf CONF  PiCache config file, default: /etc/mdx/picache.conf
  -v, --verbose         verbose output (extracts file list information)

This tool is similar to picache-ipv-admin.sh --list, except this tool dumps all the metadata, unformatted, for example:

$ picache-ipv-usage.sh 
Connected to mongoDB...
{u'dm_type': u'Perforce', u'project': None, u'is_published': False, u'last': datetime.datetime(2021, 3, 2, 9, 49, 41, 212000), u'uuid': u'6dbdf9a8-6064-4648-b259-23fd8837890a', u'filelist': None, u'repo': u'//mdx_test/tutorial/io5v/TRUNK', u'version': 2, u'expire': True, u'last_consistency_check_time': [datetime.datetime(2021, 3, 2, 9, 49, 41, 212000)], u'_id': ObjectId('603e0a35879bdc1ba348bc2f'), u'relpath': u'tutorial/io5v/TRUNK/1/tutorial.io5v', u'name': u'tutorial.io5v@1.TRUNK'}
{u'dm_type': u'Perforce', u'project': None, u'is_published': False, u'last': datetime.datetime(2021, 3, 2, 9, 49, 41, 445000), u'uuid': u'328691f6-ef1f-4557-9985-af966042dbef', u'filelist': None, u'repo': u'//mdx_test/tutorial/padring/L1', u'version': 2, u'expire': True, u'last_consistency_check_time': [datetime.datetime(2021, 3, 2, 9, 49, 41, 445000)], u'_id': ObjectId('603e0a35879bdc1bb948bc94'), u'relpath': u'tutorial/padring/L1/1/tutorial.padring', u'name': u'tutorial.padring@1.L1'}
...

It is recommended to use picache-ipv-admin.sh --list unless file list information is needed.

picache-job-purge.sh

The picache-job-purge.sh tool is used to delete all jobs from Redis queues.

$ picache-job-purge.sh --help
usage: picache-job-purge.sh [-h] [-c CONF] [-n]

IPLM Cache  administration tool for deleting all jobs from IPLM Cache Redis queues.

optional arguments:
  -h, --help       show this help message and exit
  -c CONF, --conf  CONF IPLM Cache config file, default: /etc/mdx/picache.conf
  -n, --dry-run    run the tool without deleting any job

This may be needed to reset the Redis queues to clean up an exceptional condition in order to get IPLM Cache to operate normally again.

picache-monitor.sh

The picache-monitor.sh tool is used to monitor various aspects of IPLM Cache. See the IPLM Cache administration overview page.

$ picache-monitor.sh --help
usage: picache-monitor.sh [-h] [-v] [-c CONF] [-p PERIOD] [-i] [-q] [-l]
                          [--hb] [-a] [-n] [-j]

IPLM Cache administration tool for monitoring IPLM Cache.

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         verbose output
  -c CONF, --conf CONF  IPLM Cache config file, default: /etc/mdx/picache.conf
  -p PERIOD, --period PERIOD
                        causes repeated output with this period in seconds between outputs (0 = continuous output)
  -i, --ipvs            show IPVs in cache
  -q, --queues          show queue details
  -l, --locks           show lock information
  --hb                  show heartbeat information
  -a, --all             show all information
  -n, --no_summary      don't output Summary section (overridden by --all)
  -j, --json            output in JSON format (used for automation purposes)

To get a summary of what is being monitored, picache-monitor.sh can be used without any arguments, for example:

$ picache-monitor.sh
...

__Summary__
Number of IPVs in cache: 7
Number of jobs in recovery queue(s): 11
Number of jobs in progress: 4
Number of master jobs enqueued: 93
Number of user jobs enqueued: 150
Number of locks: 7
Number of jobs finished: 119
Number of jobs failed: 0
Number of heartbeat keys: 1
Number of Redis clients: 45

To get detailed IPV information, similar to picache-ipv-admin.sh --list, picache-monitor.sh can be used as;

$ picache-monitor.sh -niv
...

__IPVs in Cache__
tutorial.io5v@1.TRUNK
    project: None
    UUID: 6dbdf9a8-6064-4648-b259-23fd8837890a
    relpath: tutorial/io5v/TRUNK/1/tutorial.io5v
    expire: True
    last: 2021-03-02 09:49:41.212000
    repo: //mdx_test/tutorial/io5v/TRUNK
    last_consistency_check_time: [datetime.datetime(2021, 3, 2, 9, 49, 41, 212000)]
tutorial.io_tsmc18@1.TRUNK
    project: None
    UUID: 2dfccb11-13b9-474e-841f-97e02cdf9f4f
    relpath: tutorial/io_tsmc18/TRUNK/1/tutorial.io_tsmc18
    expire: True
    last: 2021-03-02 09:49:41.633000
    repo: //mdx_test/tutorial/io_tsmc18/TRUNK
    last_consistency_check_time: [datetime.datetime(2021, 3, 2, 9, 49, 41, 633000)]
...

To look at the queues of IPLM Cache once per second, picache-monitor.sh can be used as:

$ picache-monitor.sh -nqvp 1
...

__Queue Information__
b=build job, u=update job, c=check job, r=remove job,
D=user_rr master dummy job, X=user_rr master execute job
d=ipv master dispatch job,
user_rr jobs executed right-to-left
mdx:picache:recovery:suse-1: 10
    charlie: b
    master: X
    alice: u
    master: X
    bob: r
    master: X
    dongwu: b
    otis: r
    master: X
    master: X
picache:ipv:wait:mdx:picache:tutorial.CADenv@2.TRUNK: 34
    rcubrcubrcubrcubrcubrcubrcubrcubrc
picache:ipv:wait:mdx:picache:tutorial.MS90G@1.TRUNK: 36
    rcubrcubrcubrcubrcubrcubrcubrcubrcub
picache:ipv:wait:mdx:picache:tutorial.acells_tsmc18@1.TRUNK: 35
    rcubrcubrcubrcubrcubrcubrcubrcubrcu
picache:ipv:wait:mdx:picache:tutorial.aes512@1.TRUNK: 35
    rcubrcubrcubrcubrcubrcubrcubrcubrcu
picache:ipv:wait:mdx:picache:tutorial.t0@1.TRUNK: 16
    rbrbrbrbrbrbrbrb
No job queues
rq:wip:mdx:picache:alice: 1
    u
rq:wip:mdx:picache:bob: 1
    r
rq:wip:mdx:picache:dongwu: 1
    b
rq:wip:mdx:picache:otis: 1
    r

...

The --json option causes picache-monitor.sh to output its data in JSON-formatted text so that picache-monitor.sh can be used for test automation purposes. For example:

$ picache-monitor.sh -j
{"summary": {"num_failed_jobs": 0, "num_finished_jobs": 472, "num_heartbeat_keys": 1, "num_ipvs_in_cache": 3, "num_jobs_in_progress": 0, "num_jobs_in_recovery_qs": 0, "num_locks": 0, "num_master_jobs_enq": 0, "num_redis_clients": 35, "num_user_jobs_enq": 0}}

picache-query-job.sh

The picache-query-job.sh tool is used to extract information from a Job key in Redis.

$ picache-query-job.sh --help
usage: picache-query-job.sh [-h] [-c CONF] id [id ...]

IPLM Cache administration tool for obtaining job information.

positional arguments:
  id                    one or more job IDs

optional arguments:
  -h, --help            show this help message and exit
  -c CONF, --conf CONF  IPLM Cache config file, default: /etc/mdx/picache.conf

This tool can be used for troubleshooting purposes for a job ID found in a log file, for example:

$ picache-query-job.sh e06904d6-e056-4b03-a579-7842bef962e3

For job e06904d6-e056-4b03-a579-7842bef962e3
...
{u'created_at': '2021-03-02T11:25:07.301253Z',
u'data': "x\x9cm\x92\xcbo\xd40\x10\x87\xb7\xa5Ohy\x1c\x10= \xc4\xebP\t\xc5!\x9b\x90\xddr\xe2^i\x85\x10H\xbe\xa0\xc8\xb1g\x13\x97$N\xfc\xd8n\x91*q\xe2\xef\xee\xc4i\xb5\r\xe2bM\xe6\xf7\xcd+3\x7f\xb6O\xe9\x87\xc9dR\x83-\x95\x90\xdc\x90Vr\xc6K\xc8\x0c\xe8\x15h\xa2\xbb\xecB\xe5\x86\xe0\x93\xb9V0\x0b\xdd\xd6\xe2\xba\xc3\xb0]\x0c+\xb4r\xed\x82\xee\xa0\xd9\xb0\x1a\xe8\x0b4\xac\xb3JKV\x11\x06\xe6S4\xfd\x12\x91\xef\xdf~,\xce\xbb\x07\xf4\x08U\xcci\xa4j2{\xd5\x02=@\x87\x86\x95\xec=\xfec)+\xa8\xa4\xb1?\xbb\x9dS\xfa\x16\x1dB\xf1\xb0\xbc\xcc\x04\x18Y4a^\x15\x19\xf6\xc6\x7f\xf5\x0cYWf\xfd.\xeav\xe9\xcb\r\x88\xe9\xe5\xf2\xca\x83\xb9+\xee\x90=\xfa\n\x11\x94\xb9\x12\x10\xfa\xa7\xb6\xc9\xbc\xe2\xf1\xb4\x8eR6%+\x84\xf6\xe9\xf3\x7f!\x0bX\xa6\xd7\x0e\xe8\xc9=\xed\xae\xcb\xbe\x0cY\xa2|8\x925\xb4J\x0f\xa2n-\xca\x0fG\xe5oec\x99\xadd\x03\\\xb9\xa6\x87\x1e\x8d\xca\xf7\x957\xe5\x8f|\x02s\xab\t-qL?\xe3`\x92\xd9o\x84\x8e\xe9\x9b{\x90O\xb0A2\x9f\xcbs\x8f\x81\x1e\xfb\xff^1\x8bR\xd62[\x8e\x16\x17\x0e\x8b\x0b\xfd\xda\xc2\xc8o\xb7o\x9a\xbeF#\x0ck\xb1\xce\x86\xf6\xfe\xc7{\xbaT\xc6\xfaq84V\x994\x10\xb0\xb2JU\xe6s\x94\xa6)\xdd\xef\xf7U\x0f'\xb0\x8d\xf6\xd7\xc4G9<9\xba\xe7w\xd9\x14\x97\xcec\xd8%0\x03\xe7[\xa3[\x19.\xce9)\xe8{4\x92x\x16\xe7gy\x1cp>\xcb\x83\x84\xa7,\x98'|\x1a\xf0t\xce\xc4r\x1e\x83\x98}t\x8b\xbf\xdd\x93\xeb\xee\xa9\xed\x9e\x91\x1b)\x12\xe5-",
 u'description': u"methodics.picache_server.rq_jobs.job_update({u'group': None, u'name': u'tutorial.aes512@1.TRUNK', u'version_type': u'revision', u'filelist': [u'doc/hw_design/blg_checklist.xlsx#1', u'doc/hw_verify/blg_bug.xlsx#1', u'hw_code/code/mt48lc32m16a2.v#1', u'hw_code/code/test.v#1', u'hw_code/filelist/blg.f#1', u'hw_code/report/blg.rpt#1', u'hw_code/report/statlinecount#1', u'hw_code/test/test.v#1', u'sw_code/driver/blg_driver.7z#1',  u'sw_code/test/blg_driver_test.7z#1'], u'relative_path': u'tutorial/aes512/TRUNK/1', u'repo': u'//mdx_test/tutorial/aes512/TRUNK', u'host': u'centos6-devtools:1666', u'dm_type': u'P4', u'user': u'dongwu', u'release': 1, u'revision': None, u'uuid': u'4373b9b3-cc7b-4c6a-84c2-c68adf83ed70'}, None)",
 u'ended_at': '2021-03-02T11:25:30.910988Z',
 u'enqueued_at': '2021-03-02T11:25:07.301369Z',
 u'failure_ttl': 2592000,
 u'meta': '\x80\x02}q\x01U\x04nameX\x17\x00\x00\x00tutorial.aes512@1.TRUNKs.',
 u'origin': u'mdx:picache:dongwu',
 u'result': '\x80\x02\x88.',
 u'result_ttl': 345600,
 u'started_at': '2021-03-02T11:25:30.097701Z',
 u'status': 'finished',
 u'timeout': 86400,
 u'ttl': 86400}

picache-query-mongo-log.sh

The picache-query-mongo-log.sh tool is used to extract logs from MongoDB.

$ picache-query-mongo-log.sh --help
usage: picache-query-mongo-log.sh [-h] [-c CONF] [-e [PATTERN [PATTERN ...]]]
                                  [-f [FUNCTION [FUNCTION ...]]]
                                  [-i [IP [IP ...]]] [-j [JOBID [JOBID ...]]]
                                  [--level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                                  [-L [LOGNAME [LOGNAME ...]]]
                                  [-m [MODULE [MODULE ...]]]
                                  [-n [NODE [NODE ...]]] [-p [PID [PID ...]]]
                                  [-t MINTIME] [-T MAXTIME] [-v] [--count]
                                  [--exc]

IPLM Cache administration tool for extracting logs from MongoDB.

optional arguments:
  -h, --help            show this help message and exit
  -c CONF, --conf CONF  IPLM Cache config file, default: /etc/mdx/picache.conf
  -e [PATTERN [PATTERN ...]], --regexp [PATTERN [PATTERN ...]]
                        Regular expressions to search log records for
  -f [FUNCTION [FUNCTION ...]], --function [FUNCTION [FUNCTION ...]]
                        Functions log records are for
  -i [IP [IP ...]], --ip [IP [IP ...]]
                        IPs log records are for
  -j [JOBID [JOBID ...]], --jobid [JOBID [JOBID ...]]
                        Job IDs log records are for
  --level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Log level (and above) log records have
  -L [LOGNAME [LOGNAME ...]], --logname [LOGNAME [LOGNAME ...]]
                        Logger names log records are for
  -m [MODULE [MODULE ...]], --module [MODULE [MODULE ...]]
                        Modules log records are for
  -n [NODE [NODE ...]], --node [NODE [NODE ...]]
                        Nodes (hostnames) log records are from
  -p [PID [PID ...]], --pid [PID [PID ...]]
                        Process IDs log records are for
  -t MINTIME, --mintime MINTIME
                        Gets log records newer than or equal to this datetime
  -T MAXTIME, --maxtime MAXTIME
                        Gets log records older than or equal to this datetime
  -v, --verbose         Verbose output
  --count               Prints the number of log records and exits
  --exc                 Gets log records only showing exceptions

Use an egrep expression for a logical OR search of the logs, for example:
  -e 'job_build|job_update'
will get log records containing 'job_build' or 'job_update'.
Use multiple expressions for a logical AND search of the logs, for example:
  -e 'enter' 'job_build'
will get log records containing both 'enter' and 'job_build'.
MINTIME and MAXTIME are formatted as '2018-07-02T14:59:13.365'
(fields 'T' and after are optional).

The --exc option is used to find any exceptions raised by the IPLM Cache backend processes. This can be used to find failed jobs, for example:

$ picache-query-mongo-log.sh --exc

Number of total log records: 554

Log records with search criteria:

...

Log record 2:
{u'_id': ObjectId('603fd827879bdc5a11e1c979'),
 u'exception': {u'fileName': u'/mdx/tools/methodics/devtools/2019-12-03/env/local/lib/python2.7/site-packages/rq/worker.py',
                u'stackTrace': u'Traceback (most recent call last):\n File "/mdx/tools/methodics/devtools/2019-12-03/env/local/lib/python2.7/site-packages/rq/worker.py", line 822, in perform_job\n rv = job.perform()\n File "/mdx/tools/methodics/devtools/2019-12-03/env/local/lib/python2.7/site-packages/rq/job.py", line 605, in perform\n self._result = self._execute()\n File "/mdx/tools/methodics/devtools/2019-12-03/env/local/lib/python2.7/site-packages/rq/job.py", line 611, in _execute\n return self.func(*self.args, **self.kwargs)\n File "/home/bob/develop/picache/src/methodics/picache_server/rq_jobs.py", line 232, in job_build\n reply = ipinst.build()\n File "/home/bob/develop/picache/src/methodics/picache_server/ip.py", line 323, in build\n % (type(e).__name__, str(e)))\nPICacheError: PICacheError exception occurred, build removed: User \'mdxadmin\' is not a member of group \'video\'\n'},
 u'funcName': u'handle_exception',
 u'ipv_name': u'tutorial.perm_test@28.TRUNK',
 u'job-args': [{u'dm_type': u'P4',
                u'filelist': [u'test1.txt#2',
                              ...],
               u'group': u'video',
               u'host': u'centos6-devtools:1666',
               u'name': u'tutorial.perm_test@28.TRUNK',
               u'relative_path': u'tutorial/perm_test/TRUNK/28/tutorial.perm_test',
               u'release': 28,
               u'repo': u'//mdx_test/tutorial/PIC-260/TRUNK',
               u'revision': None,
               u'signature': u'88 10 152',
               u'user': u'mdxadmin',
               u'uuid': u'f0d61285-b317-4840-a5f1-e466d3dce81f',
               u'version_type': u'revision'},
              None],
 u'job-func': u'methodics.picache_server.rq_jobs.job_build',
 u'job-kwargs': {},
 u'job-queue': u'mdx:picache:mdxadmin',
 u'jobId_or_name': u'b25d4b48-0f71-45b0-95e3-4d4779b20771',
 u'level': u'ERROR',
 u'lineno': 871,
 u'message': u'PICacheError: PICacheError exception occurred, build removed: User \'mdxadmin\' is not a member of group \'video\'\nTraceback (most recent call last):\n File "/mdx/tools/methodics/devtools/2019-12-03/env/local/lib/python2.7/site-packages/rq/worker.py", line 822, in perform_job\n rv = job.perform()\n File "/mdx/tools/methodics/devtools/2019-12-03/env/local/lib/python2.7/site-packages/rq/job.py", line 605, in perform\n self._result = self._execute()\n File "/mdx/tools/methodics/devtools/2019-12-03/env/local/lib/python2.7/site-packages/rq/job.py", line 611, in _execute\n return self.func(*self.args, **self.kwargs)\n File "/home/bob/develop/picache/src/methodics/picache_server/rq_jobs.py", line 232, in job_build\n reply = ipinst.build()\n File "/home/bob/develop/picache/src/methodics/picache_server/ip.py", line 323, in build\n % (type(e).__name__, str(e)))\nPICacheError: PICacheError exception occurred, build removed: User \'mdxadmin\' is not a member of group \'video\'\n',
 u'module': u'worker',
 u'node': u'suse-1',
 u'process': 23057,
 u'timestamp': datetime.datetime(2021, 3, 3, 10, 40, 39, 228000)}

Knowing the ID of the process in which an exception occurred, it is sometimes useful to get the logs for that process using the -p/--pid argument, for example:

$ picache-query-mongo-log.sh -p 23057

To get the logs specific to an IP, the -i/--ip argument can be used, for example:

$ picache-query-mongo-log.sh --ip tutorial

$ picache-query-mongo-log.sh --ip tutorial.perm_test

$ picache-query-mongo-log.sh --ip tutorial.perm_test@28.TRUNK

In a high-availability deployment with multiple nodes running IPLM Cache instances, the -n/–-node option can be used to obtain logs from one or more specific nodes.

The -t/--mintime and/or -T/–-maxtime options can be used to refine the search by bounding the search to specific times, for example the following command searches for logs within a specific second:

$ picache-query-mongo-log.sh -p 17965 -t 2021-3-3T11:33:24.124000 -T 2021-3-3T11:33:25.124000

Searching the logs for a specific message or part of a message is done with the -e/--regexp option. Other options are used to search for log messages from a particular module and function. Ordinarily, a Perforce engineer will ask that you run a specific query using these for troubleshooting purposes.

picache-query-queue.sh

The picache-query-queue.sh tool is used to extract the Job IDs in one or more queues in Redis.

$ picache-query-queue.sh --help
usage: picache-query-queue.sh [-h] [-c CONF] [queue [queue ...]]

IPLM Cache administration tool for printing the job IDs in queues.

positional arguments:
  queue                 one or more queue names (none to print the queue names)

optional arguments:
  -h, --help            show this help message and exit
  -c CONF, --conf CONF  IPLM Cache config file, default: /etc/mdx/picache.conf

Queue names can be obtained by picache-query-queue.sh with no arguments (or by picache-monitor.sh) and then one or more queue names passed to picache-query-queue.sh to get the job IDs in the queue, which could then be used by picache-query-job.sh to get information on a specific job, for example:

$ picache-query-queue.sh
...

Number of queues: 6
Queues:
rq:queue:mdx:picache:charlie
rq:queue:mdx:picache:bob
rq:queue:mdx:picache:dongwu
rq:queue:mdx:picache:master
rq:queue:mdx:picache:alice
rq:queue:mdx:picache:otis

$ picache-monitor.sh -nq
...

__Queue Information__
No recovery queues
No IPV wait queues
rq:queue:mdx:picache:alice: 4
rq:queue:mdx:picache:bob: 4
rq:queue:mdx:picache:charlie: 4
rq:queue:mdx:picache:dongwu: 4
rq:queue:mdx:picache:master: 18
rq:queue:mdx:picache:otis: 2
No jobs in progress

$ picache-query-queue.sh rq:queue:mdx:picache:otis
...

For queue rq:queue:mdx:picache:otis
  Number of jobs in queue: 2
  Job IDs:
    15b82208-c929-431b-b5ae-df8429d5761c
    0d09d999-a9f1-4c22-ae03-7e4cadca101e

$ picache-query-job.sh 15b82208-c929-431b-b5ae-df8429d5761c

For job 15b82208-c929-431b-b5ae-df8429d5761c
...
{u'created_at': '2021-03-03T20:13:28.820358Z',
 u'data': "x\x9c]\x92Mo\xd40\x10\x86\xb7\xd0\xd2\x02-\xb4BH\x1c\x10\x12\x1f\x87J(N\xb2\x1f\xe9\xb6'\xeeH+\x84@\xf2\x05EYg\xb218qbO\xc2\x16\xa9RO\xfd\xdd\x8c\xbd\xad\xb6\xe9%\x9a\xcc\xfb\xcc\xbcc\x8f\xaf\x1f\x9d\xf2\xcf\xa3\xd1\xa8\x02,u.\x85e\x8d\x14\x99(!\xb5`z0\xcc\xb4\xe9o\xbd\xb4\x8c>\xa9\x81J\xf7\xd0\xee,\xaeZ*\xdb\xa3\xb2\x95\xd1]\xe3\xa3\x8ex\xcbw)\xaa\xb3\n\xf8+\n\xb0Cmd\xa6\x18F_b\xf6\xe3\xfb\xcf\xc5\xd7\xf61?$\x85\x1a[\xa9\xeb\x14/\x1b\xe0\x07\x940\xd0K\x97\xf1?\x85T\xa0\xa4\xc5_\xed\xee)\xff@\x89\\\x8b\xb0\xfc\x9b\xe6`\xe5\xaa\x0e\x97j\x95\xd2\x80\xe2\x8fc\xd8Z\xd9\xf5\xc7\xb8\xdd\xe3o\xb7 \xb5\x97\xc5\xa5\x07\x97\xdd\xea\x0ey\xc2\xdf\x11B\xb2\xd09\x84\xfeS\xe1t\xae\xc4d\\\xc5I6f=A\xfb\xfc\xf5C\x08\x81l\x9cv\xc0\xdf\xdc\xd3\xee\xa6t6\xac \xf9\xe9@6\xd0h\xb3\x11M\x83$?\x1b\xd8\xdf\xca\x163T\xb2\x06\xa1\xbb\xdaA\xcf\x07\xf6\xceyk\x7f\xe8\x1b\xd8[-7\x92\x8e\xe9\xcf\xb8\t\xd9\xd9?\x82\x8e\xf8\xfb{\x90o\xb0ER\xdf\xcbs/\x80\x1f\xf9{W\x19\x92\x946\x19\x96\x83\xa5\x85\x18\x85~ea\xec\xb7\xea\x06\xf6w\x1c\x86U\xbeN7\xa3=d=Yj\x8b\xfe\x18\x02j\xd46\tr\xe8Qke/\xe2$I\xf8\xbe\xdbS\xb5]\xfd70\x856\x02|\xad{E>\xd0(\xadGiB\xc8,\xf0\x1d\x8a\xe3\xe1[q\xa9hS\xd6\xc9\x9c\x7fr\x96\xd3D\xc4\xf3\x18\x021^\x9e\x07\xd3x\x92\x04\xf3(*\x82bV$\xa2\x98M\xe6\xb33\xd1-n\xda\x97W\xed1\xb6'\xec?\xba<\xe5\xb4",
 u'description': u"methodics.picache_server.rq_jobs.job_remove({u'group': u'users', u'name': u'tutorial.t0@1.TRUNK', u'version_type': u'revision', u'filelist': [u'doc/hw_design/blg_checklist.xlsx#1', u'doc/hw_verify/blg_bug.xlsx#1', u'hw_code/code/mt48lc32m16a2.v#1', u'hw_code/code/test.v#1', u'hw_code/filelist/blg.f#1', u'hw_code/report/blg.rpt#1',  u'hw_code/report/statlinecount#1', u'hw_code/test/test.v#1', u'sw_code/driver/blg_driver.7z#1', u'sw_code/test/blg_driver_test.7z#1'], u'relative_path': u'tutorial/t0/TRUNK/1', u'repo': u'//mdx_test/tutorial/t0/TRUNK', u'host': u'centos6-devtools:1666', u'dm_type': u'Perforce', u'user': u'otis', u'release': u'1', u'revision': u'0', u'uuid': u'c46c181e-c2b9-4136-800f-f5f6cf53857c'}, None)",
 u'enqueued_at': '2021-03-03T20:13:28.820442Z',
 u'failure_ttl': 2592000,
 u'meta': '\x80\x02}q\x01U\x04nameX\x13\x00\x00\x00tutorial.t0@1.TRUNKs.',
 u'origin': u'mdx:picache:otis',
 u'result_ttl': 345600,
 u'status': 'queued',
 u'timeout': 86400,
 u'ttl': 86400}

picache-wdog-unregister.sh

The picache-wdog-unregister.sh tool is used to unregister one or more clients from the IPLM Cache Watchdog Monitor.

$ picache-wdog-unregister.sh --help
usage: picache-wdog-unregister.sh [-h] [-v] [-c CONF]
                                  entity_name [entity_name ...]

IPLM Cache administration tool for unregistering clients from the IPLM Cache Watchdog Monitor.

positional arguments:
  entity_name           one or more entity names to unregister

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         verbose output
  -c CONF, --conf CONF  IPLM Cache config file, default: /etc/mdx/picache.conf

The IPLM Cache Watchdog Monitor is used to monitor the health of the IPLM Cache instance running on the same node as the monitor. As discussed in the IPLM Cache Watchdog section of the IPLM Cache administration overview page, the Watchdog Monitor can be used with the Linux Watchdog Daemon to reboot the node should the IPLM Cache instance become unhealthy. If the Linux Watchdog Daemon is not used and a IPLM Cache process registers with the Watchdog Monitor but stops pinging for some reason, the message 'entity-name' has stopped pinging will be logged continuously to the Watchdog Monitor log at /var/log/mdx-picache/picache-wdog.log. In order to unregister the entity with the Watchdog Monitor and stop the logging, use the picache-wdog-unregister.sh tool using the name of the entity as its argument.