vscNetworkMonitoring

Monitoring your Perforce or Subversion server network activity

There maybe situations where accessing the repository takes longer than expected and it's useful to have some tools in place to help identify potential issues.

  • Nagios is a useful network monitoring tool
  • RDDTOOL is a great logging and graphing tool for your network performance

Some Example Perforce Checks

Perform a p4 info against the P4PORT of each Perforce server and proxy

  • This can be applied to RDD and the history can be recorded over a certain period
  • Note that if p4 checkpoints take longer than a few minutes, you'll need to make sure the monitor doesn't alert during the checkpoint period.

Disk space monitors on all the server and proxy hosts.

Check for the number of p4d processes running, the max time run-time, CPU and memory use

  • If the number of processes is above a threshold (eg: 35), the max run-time is greater than 10 min and the CPU is greater than 35% then send an alert to kill that process.

Check the number of commands that are running on Perforce

  • p4 monitor show | wc -l
  • Any more than a certain threshold send an alert

Monitor with 'p4 depots' or 'p4 print' (on a specific file) and test the output.

  • 'p4 info' will return successfully even when there are a number of problems on the server. For example, you can delete all your db.* files while Perforce is running, and p4 info will return successfully.

Nightly checks of the repositories for pending changes

  • Pending changes are usually a sign that a commit failed, VersIC commits should never go into a pending state

Nightly checks for unmanaged cells in managed libraries and other anomalies

  • These kinds of situations are rare but maybe caused by users not following the methodology guidelines for creating library data, eg. copying a cell to a library via Unix rather than through the library manager, stream-in to a library without running "Add to VersIC".

Updating the users local workspace in a controlled fashion to reduce multiple simultaneous library updates

  • This would be an alternative to setting VSC_AUTO_UPDATE = "Library" in versic.conf.

eg:

 05 1 * * * simon svn up /proj/svn/workareas/simon
 15 1 * * * fergus svn up /proj/svn/workareas/fergus
 25 1 * * * christy svn up /proj/svn/workareas/christy

Cron may also be used for data-mining techniques to deduce heuristics on the state of the design repository

  • VersIC ensures commits are tagged with state information (LVS Clean, DRC Clean, Work In Progress etc) and this maybe used to determine the state of a project at any point in the design cycle