Automated testing for reviews

Integrating P4 Code Review with a test suite involves enabling Automated Tests in your project's configuration and providing a trigger URL. When the trigger URL is requested, P4 Code Review expects your test suite to be executed. When the tests complete, P4 Code Review expects an update callback URL, a pass callback URL, or a fail callback URL to be requested by your test suite.

1. Navigate to the Project page.
2. Click the project Settings tab to display the Project Settings page.
3. Ensure that paths in each named branch configured for the project do not overlap with paths in other named branches.

4. Automated tests checkbox: select Enable to display the configuration fields:

   

5. Provide a URL that triggers your test suite execution.

   Special arguments are available to inform your test suite of various details from P4 Code Review. P4 Code Review automatically replaces the arguments with the relevant P4 Code Review information when it calls the test:

   P4 for Jenkins 1.10.11 and later: P4 Code Review must send the parameters for the build to Jenkins as a POST request. To do this, enter the parameters in the Post Body and select URL Encoded.

   {test}

   The name of the test

   {testRunId}

   The test run id

   {change}

   The change number

   If your CI system supports change=now, you can use this instead of the change={change} to make sure your review is always tested against the latest version of the submitted code.

   {status}

   Status of the change, shelved or submitted

   {review}

   The review's identifier

   {version}

   The version of the review

   {description}

   The change description of the change used to generate this update. {description} cannot be used in the URL, it can only be used in the POST Body.

   {project}

   The project's identifier

   {projectName}

   The project's name

   {branch}

   The branch identifier(s) impacted by the review, comma-separated

   {branchName}

   The branch name(s) impacted by the review, comma-separated

   {update}

   The update callback URL. You can include any or all of the following when calling the update url to update the test run: status, messages, and a url in the body that links to the CI system for that run. They should be formatted in JSON in the body of the POST request. Status: valid status values are running, pass, and fail. Messages: you can pass a maximum 10 messages, if you provide more than 10 messages only the first 10 are saved. Each message can contain a maximum of 80 characters, any messages with more than 80 characters will be automatically truncated. {update} is the preferred option for P4 Code Review 2019.3. For more details, see the note below.

   {pass}

   Tests pass callback URL. From P4 Code Review 2019.3, {update} is preferred. For more details, see the note below.

   {fail}

   Tests fail callback URL. From P4 Code Review 2019.3, {update} is preferred. For more details, see the note below.

   • P4 Code Review 2019.3 and later still supports {pass} and {fail}, however {update} is preferred because you can also include a message with the test status.
   • {update}, {pass}, and {fail} are composed automatically by P4 Code Review. They include P4 Code Review's own per-review authentication tokens.

6. Optional: specify any parameters that your automated tests require that must be sent via HTTP POST in the POST Body field. The POST parameters can include the special arguments listed above.

   Select the format of the POST parameters, either URL Encoded or JSON Encoded.

   • URL Encoded: POST parameters are parsed into name=value pairs.
   • JSON Encoded: parameters are passed raw in the POST body.

Configuring Jenkins for P4 Code Review integration

1. Install the p4-plugin for Jenkins:

   For instructions on installing the p4-plugin, see in the Installation chapter of the P4 for Jenkins Documentation.

2. Configure a Jenkins project:

   P4 for Jenkins 1.10.11 and later: P4 Code Review must send the parameters for the build to Jenkins as a POST request. To do this, enter the parameters in the Post Body and select URL Encoded.

   1. Specify the job name so that it matches the project identifier used in the trigger URL, as defined below.

      For example, the computed value of {projectName}_{branchName}.

      Or, edit the trigger URL to use the Jenkins job name you specify.

   2. Make the build parameterized to accept these parameters (note that these are named to match up with the script that is called):

      {test}

      The name of the test

      {testRunId}

      The test run id

      {status}

      Whether the changelist to be tested is shelved or submitted

      {change}

      Changelist # to run tests against

      {review}

      The review's identifier

      {version}

      The version of the review

      {branchName}

      The branch name(s) impacted by the review, comma-separated

      {update}

      The URL to POST to update the test run. You can include any or all of the following when calling the update url to update the test run: status, messages, and a url in the body that links to the CI system for that run. They should be formatted in JSON in the body of the POST request. Status: valid status values are running, pass, and fail. Messages: you can pass a maximum 10 messages, if you provide more than 10 messages only the first 10 are saved. Each message can contain a maximum of 80 characters, any messages with more than 80 characters will be automatically truncated. {update} is the recommended way of reporting test status.

      If your test system cannot POST to P4 Code Review, you cannot use update and you must use pass and fail instead.

      When using {update}:

      • If your build script has access to any messages related to the test execution, pass the messages to P4 Code Review using the {update} URL. P4 Code Review uses the provided message(s) to add to the test results.
      • If your build script has access to the results of test execution, include a POST parameter called url when calling the update URL. P4 Code Review uses the provided url to link reviews to the test results. Valid test status values are running, pass, and fail.
      • The {update} callback url accepts a JSON body where you can specify any or all of the following: messages, url, and status.

      {
          "messages" : ["My First Message", "My Second Message"],
          "url" : "http://jenkins_host:8080/link_to_run",
          "status": "pass"
      } 

      {pass}

      The URL to wget if the build succeeds. From P4 Code Review 2019.3, {update} is preferred. For more details, see the note below.

      {fail}

      The URL to wget if the build fails. From P4 Code Review 2019.3, {update} is preferred. For more details, see the note below.

      When using {pass} and {fail}: if your build script has access to the results of test execution, include a GET or POST parameter called url when calling the pass or fail URLs. P4 Code Review uses the provided url to link reviews to the test results.

      For example: ?url=https://jenkins.example.com/view/job/<jobname>/<jobid>/

      P4 Code Review 2019.3 and later still supports {pass} and {fail}, however {update} is preferred because you can also include a message with the test status.

   3. Select Perforce Software for the Source Code Management section.

      You might see Perforce in the Source Code Management section. This represents an earlier community-provided Perforce plugin that does not include support for P4 Code Review.

   4. Set up credentials and workspace behavior as needed.

      For instructions on configuring credentials and workspaces, see the Credentials and Workspaces sections of the P4 for Jenkins Documentation.

      • If your P4 Server is configured for P4 AS, the service user credentials used for automated testing must not use P4 AS.
      • The client workspace configured in Jenkins must have a view that includes the paths defined for that branch in P4 Code Review.

3. Configure your P4 Code Review project to run automated tests with a URL and parameters like this:

   In this example, the build system is passed the changelist number of the latest version of the review in {change}. If the test is rerun at a later date, the build system will rerun the test against the same {change} number, and this will probably not be the latest version of the codebase. If you want the review to always be tested against the latest versions of submitted code, replace change={change} with change=now. Your CI system must support change=now to use this feature.

   

   For Jenkins, the job name needs to match the job identifier in the URL. In the example above, this is the computed value of {projectName}_{branchName}.

   If you prefer a different naming scheme in Jenkins, replace {projectName}_{branchName} in the URL above with the project name actually defined in Jenkins.

   If security is enabled in Jenkins, the trigger URL needs to include credentials. Follow these steps:

   1. Create a Jenkins user that will trigger P4 Code Review builds. For example swarm.
   2. Log into Jenkins as the new user.
   3. Click on the user's username in the Jenkins toolbar.
   4. Scroll down to API Token.
   5. Click Show API Token.

   6. Incorporate the value of the API Token into the P4 Code Review trigger URL.

      For example, if the username is swarm and the API Token value is 832a5db7e5500c1288324c1441460610, the P4 Code Review trigger URL and parameters should be: