Data consistency checking

It is important for the files of the static IPVs to remain unchanged. The IPLM Cache Data Consistency Checking feature detects any changes to the static IPVs and logs any inconsistencies. This currently only applies to using Perforce as the data manager.

Background

IPLM Cachecan be setup to periodically check the contents in the cache, as compared to the Perforce repository for the IPV, and log any data inconsistencies found. This feature requires IPLM Cache use of MongoDB, which as of release v1.6.0 is optional (see picache.conf configuration file for the mongod-host setting).

IPLM Cache configuration setting for data consistency checking

The related configuration settings in the IPLM Cache configuration file, /etc/mdx/picache.conf, are:

cache-check-period-hrs 
# The interval at which the cache will be gone through and static IPVs checked
# for consistency. Defaults to 24 hours. A value of zero disables the check.
#cache-check-period-hrs = 24

# The maximum number of concurrent IPV data consistency check jobs that can run
# at any one time. A value of zero means no limit. Defaults to zero. Give a
# non-zero value to balance IPV data consistency checking with responsiveness to
# end-user jobs.
#max-num-concurrent-check-jobs = 4

# The interval, in hours, at which IPLM Cache maintenance activities (cleanup
# and data consistency checks) will sleep for before running again. Defaults to
# 1 hour. A value of zero disables maintenance activities.
#maintenance-period-hrs = 1

# The period of time, in minutes, IPLM Cache will wait for the remove and check
# jobs enqueued by the cleanup and data consistency check activities.
# Defaults to 10 minutes. A value of zero means no waiting is done.
#maintenance-job-wait-mins = 10

To use the Data Consistency Check feature, you must use the following settings:

  • MongoDB is required. See picache.conf Configuration file for more details.
    • mongod-host setting - Defines one MongoDB host:port pair for a single instance MongoDB deployment or three host:port pairs (separated by commas) for an HA MongoDB deployment.
    • For HA deployments, the mongod-rs setting is also required.
  • The cache-check-period-hrs setting must be non-zero.

The cache-check-period-hrs setting determines the interval at which an individual IPV will undergo a data consistency check. For example, if cache-check-period-hrs were set to 24 hours, after an IPV is checked for consistency, it will be checked again 24 hours later.

As of IPLM Cache v1.7.0, the max-num-concurrent-check-jobs, maintenance-period-hrs, and maintenance-job-wait-mins settings can be used to provide a smooth flow of check jobs without IPLM Cache becoming unresponsive to end users' jobs. For example, given a cache with thousands of static IPVs, if all of the IPVs' data consistency checks happened at the same time, IPLM Cache could be busy for an extended period of time processing the IPVs' check jobs and a user's job (build, update, or remove) could be considerably delayed. One approach to handling this scenario could be to use the following settings:

  • max-num-concurrent-check-jobs = 4
  • maintenance-period-hrs = 0.25
  • maintenance-job-wait-mins = 0

These settings would configure IPLM Cache's maintenance task to not wait for the check (and remove) jobs the maintenance task enqueues (maintenance-job-wait-mins = 0); after enqueuing all of its check and remove jobs, the maintenance task would sleep for 15 minutes (maintenance-period-hrs = 0.25); and no more than four check jobs would be enqueued at any one time (max-num-concurrent-check-jobs = 4). Consequently, the thousands of IPVs undergoing data consistency checks would be checked at most four at a time, with more check jobs possibly being added about every 15 minutes, with the idea of providing a smooth, constant flow of IPV data consistency checking while leaving additional IPLM Cacheworkers available to handle users' jobs.

Some experimentation may be needed to find optimum values for these settings. For example, if IPV data consistency check job execution times are on average short, the maintenance-period-hrs setting could be made shorter, like 10 minutes (maintenance-period-hrs = 0.1667). If IPLM Cachee is used in a High Availability deployment, say with eight workers per instance (worker-number = 8), if you wanted half of the workers in each instance to handle data consistency check jobs you could set max-num-concurrent-check-jobs in each instance's picache.conf file to four times the number of instances, so with two instances max-num-concurrent-check-jobs = 8. You could experiment with the value for max-num-concurrent-check-jobs, too, making it higher or lower, depending on user responsiveness.

What is Checked?

Static IPVs are checked for consistency. These are IPVs denoted with a specific version, like tutorial.verif_config@2.TRUNK, which have a known filelist. Non-static IPVs, like tutorial.verif_config@HEAD.TRUNK, do not have a known filelist and are not checked.

Static IPV contents are checked for the following:

  1. Changes to the DM repository/depot, not reflected in the cache
  2. Files in the depot missing from the cache
  3. Files in the cache missing from the depot
  4. Files modified in the cache that are not opened for edit by Perforce
  5. Files which are opened by Perforce
  6. Differences between the PiServer file list for the IPV and the actual files
  7. The IPV's Perforce Client exists and is as expected

Accessing Data Consistency Check Results

Data Consistency Check results can be obtained from the IPLM Cache MongoDB log by getting the logs for the job_check function using the picache-query-mongo-log.sh tool, as of v1.6.0, or the query_mongo_log.pyc tool for versions prior to v1.6.0:

job_check Function 
/usr/share/mdx/products/picache/bin/picache-query-mongo-log.sh -f job_check
or
/usr/share/mdx/products/picache/local/bin/python /usr/share/mdx/products/picache/local/lib/python2.7/site-packages/methodics/picache_server/tools/query_mongo_log.pyc -f job_check

Examples

An example of an IPV being checked and no inconsistencies being found:

Checking an IPV (No Inconsistencies) 
/usr/share/mdx/products/picache/bin/picache-query-mongo-log.sh -f job_check
or
$ /usr/share/mdx/products/picache/local/bin/python /usr/share/mdx/products/picache/local/lib/python2.7/site-packages/methodics/picache_server/tools/query_mongo_log.pyc -f job_check

Number of total log records: 374

Log records with search criteria:

Log record 1:
{u'_id': ObjectId('5e68409a5e58cc705ad0b339'),
 u'funcName': u'job_check',
 u'ipv_name': u'tutorial.verif_config@2.TRUNK',
 u'jobId_or_name': u'147c4e37-d9dc-449b-909f-0b4542910f81',
 u'level': u'INFO',
 u'lineno': 227,
 u'message': u"enter job_check:\n{u'_id': ObjectId('5e6816725e58cc4e33d0b34b'),\n u'dm_type': u'Perforce',\n u'expire': True,\n u'filelist': [u'doc/README.txt#1',\n               u'doc/hw_design/blg_checklist.xlsx#1',\n               u'doc/hw_design/readme#0',\n               u'doc/hw_verify/blg_bug.xlsx#1',\n               u'hw_code/code/mt48lc32m16a2.v#1',\n               u'hw_code/code/test.v#1',\n               u'hw_code/filelist/blg.f#1',\n               u'hw_code/report/blg.rpt#1',\n               u'hw_code/report/statlinecount#1',\n               u'hw_code/test/test.v#1',\n               u'sw_code/driver/blg_driver.7z#1',\n               u'sw_code/test/blg_driver_test.7z#1'],\n u'last': datetime.datetime(2020, 3, 10, 22, 36, 34),\n u'last_consistency_check_time': datetime.datetime(2020, 3, 10, 22, 36, 34),\n u'name': u'tutorial.verif_config@2.TRUNK',\n u'project': None,\n 'relative_path': u'tutorial/verif_config/TRUNK/2/tutorial.verif_config',\n u'repo': u'//mdx_test/tutorial/verif_config/TRUNK',\n u'uuid': u'7332f652-ad39-4e3c-bbad-2ee14ab5c287',\n u'version': 1}",
 u'module': u'rq_jobs',
 u'node': u'centos6-ipcache3',
 u'process': 28762,
 u'timestamp': datetime.datetime(2020, 3, 10, 18, 36, 26, 550000)}

Log record 2:
{u'_id': ObjectId('5e68409a5e58cc705ad0b348'),
 u'funcName': u'job_check',
 u'ipv_name': u'tutorial.verif_config@2.TRUNK',
 u'jobId_or_name': u'147c4e37-d9dc-449b-909f-0b4542910f81',
 u'level': u'INFO',
 u'lineno': 236,
 u'message': u'job_check return: True',
 u'module': u'rq_jobs',
 u'node': u'centos6-ipcache3',
 u'process': 28762,
 u'timestamp': datetime.datetime(2020, 3, 10, 18, 36, 26, 723000)}

...


An example of an IPV being checked with a missing file, an unexpected added file, and an unexpected modified file:

Checking an IPV (Inconsistencies) 
$ /usr/share/mdx/products/picache/bin/picache-query-mongo-log.sh -f job_check --ip tutorial.check_test@2.TRUNK
or
$ /usr/share/mdx/products/picache/local/bin/python /usr/share/mdx/products/picache/local/lib/python2.7/site-packages/methodics/picache_server/tools/query_mongo_log.pyc -f job_check --ip tutorial.check_test@2.TRUNK


Number of total log records: 10451

Log records with search criteria:

...
 
Log record 41:
{u'_id': ObjectId('5e69077c5e58cc4bfb5d68f3'),
 u'funcName': u'job_check',
 u'ipv_name': u'tutorial.check_test@2.TRUNK',
 u'jobId_or_name': u'b714a928-3687-49b3-bab8-347351778662',
 u'level': u'INFO',
 u'lineno': 227,
 u'message': u"enter job_check:\n{u'_id': ObjectId('5e6904ec5e58cc244d5d68d6'),\n u'dm_type': u'Perforce',\n u'expire': True,\n u'filelist': [u'test1.txt#1', u'test2.txt#1'],\n u'last': datetime.datetime(2020, 3, 11, 15, 34, 4, 284000),\n u'last_consistency_check_time': datetime.datetime(2020, 3, 11, 15, 44, 30, 382000),\n u'name': u'tutorial.check_test@2.TRUNK',\n u'project': None,\n 'relative_path': u'tutorial/check_test/TRUNK/2/tutorial.check_test',\n u'repo': u'//mdx_test/tutorial/check_test/TRUNK',\n u'uuid': u'4c5952ba-146b-439f-8149-499b4af9a362',\n u'version': 1}",
 u'module': u'rq_jobs',
 u'node': u'centos6-ipcache3',
 u'process': 19451,
 u'timestamp': datetime.datetime(2020, 3, 11, 8, 45, 0, 646000)}

Log record 42:
{u'_id': ObjectId('5e69077c5e58cc4bfb5d6904'),
 u'funcName': u'job_check',
 u'ipv_name': u'tutorial.check_test@2.TRUNK',
 u'jobId_or_name': u'b714a928-3687-49b3-bab8-347351778662',
 u'level': u'INFO',
 u'lineno': 236,
 u'message': u'job_check return message: Project None IPV tutorial.check_test@2.TRUNK: Status: move/add: /picache-root/tutorial/check_test/TRUNK/2/tutorial.check_test/test1-1.txt; Project None IPV tutorial.check_test@2.TRUNK: Status: move/delete: /picache-root/tutorial/check_test/TRUNK/2/tutorial.check_test/test1.txt; Project None IPV tutorial.check_test@2.TRUNK: Status: edit: /picache-root/tutorial/check_test/TRUNK/2/tutorial.check_test/test2.txt',
 u'module': u'rq_jobs',
 u'node': u'centos6-ipcache3',
 u'process': 19451,
 u'timestamp': datetime.datetime(2020, 3, 11, 8, 45, 0, 860000)}

To just find the IPVs with data inconsistencies, the following search can be done (as of IPLM Cache v1.7.0):

$ picache-query-mongo-log.sh -e "job_check return message"