Submitting build and automated test results from Jenkins to Helix ALM

To get build and automated test result data submitted to Helix ALM, you must use the Jenkins plugin. See Using the Helix ALM Test Management Jenkins plugin. You must also make sure to add post-build steps in Jenkins to submit results data to Helix ALM. See Adding post-build steps to automatically submit build and automated test results from Jenkins to Helix ALM.

You can configure Jenkins to generate a report file after a build completes that contains information about build and automated tests, including results. The report format must be JUnit or XUnit. Other report formats may not be XML.

The plugin parses data from the XML file and then uploads data to Helix ALM. You can then view the build and automated test data in the Helix ALM web client.

Your automated test scripts need to write the correct data out to the XML report file. The following information explains how data in XML report files is parsed and submitted to Helix ALM so you can understand the data that your scripts need to provide.

Helix ALM supports JUnit and xUnit XML result report formats. Some automated testing tools may not generate a report file that Helix ALM can parse or the JUnit XML file may not contain all of the data you want to upload to Helix ALM. Specifically, if you are using a testing framework that runs tests using Selenium, detailed information about the test environment, such as operating system and browser information, will not be added to Helix ALM. If it is important to include this information in Helix ALM, you may need to handle reading the test report yourself and submit results using the Helix ALM REST API. See Submitting builds and automated test results to Helix ALM using the REST API.

XML report elements that Helix ALM parses

The plugin parses data from the following elements in the XML report file:

  • <testsuite> – Contains one or more <testcase> elements and information about the timing and results of those tests. A report may contain multiple <testsuite> elements, but only one build is added to Helix ALM.
  • <testcase> – Contains all information about a single automated test. One automated test result is added to Helix ALM for each <testcase> element in the XML report file. A <testcase> element can contain child elements that specify extra information for tests that fail, have errors, or are skipped.

Build data added to Helix ALM

One build is added to Helix ALM for each build, even if the XML report file contains more than one <testsuite> element. The following information about builds is added to Helix ALM from Jenkins.

Required data

If the following data is must be specified in Jenkins and in the XML report file.

  • Build number

Optional data

If other optional data is specified in Jenkins and in the XML report file, it is displayed in the build details in Helix ALM. See Viewing build summary information in automation suites.

The following values are configured by the Jenkins plugin user per project. If you are using a Freestyle project, they are text values entered in text fields. If you are using Pipeline, you can optionally provide these values when invoking the halm_report pipeline step.

  • Description
  • Branch
  • Test run set

The following values are automatically populated by the Jenkins plugin and will always have data.

Build Start date and Duration fields

The build Start date and Duration fields are populated in Helix ALM using the following methods:

  • If the XML contains one <testsuite> element and has timestamp and time attributes, the build Start date field is populated with timestamp and the Duration field is populated with time.
  • If the XML contains multiple <testsuite> elements with timestamp and time attributes, the build Start date field is populated with the earliest reported timestamp and time value, and the Duration field is calculated by adding the difference between the earliest and latest reported timestamp plus the duration value specified in the latest reported timestamp.
  • If the <testsuite> elements in the XML file do not contain a timestamp attribute, the build Start date field is not set in Helix ALM, but Helix ALM infers the date from the <testcase> elements. <testcase> elements optionally include a time value to indicate the test duration. Helix ALM uses the total of all reported time values and uses it for the build Duration field value.

Automated test result data added to Helix ALM

One automated test result is created for each <testcase> element in the XML report file. Helix ALM automatically maps the following XML attributes to the automated test result fields in Helix ALM. You can view this data on the Automated Test Result Details dialog box in Helix ALM. See Viewing automated test result details.

Automated test result values in Helix ALM Value from XML file used to populate Helix ALM values More information
Test name name or uniqueName Automated test results in Helix ALM must have unique names to ensure that results are automatically associated with test cases correctly. If names are not unique, automated test result and associated test cases could be mismatched when running subsequent tests. The name attribute in the <testcase> element in the XML file is not required to be unique, so we recommend also specifying the uniqueName attribute in the <testcase> element.

If uniqueName is not specified, Helix ALM uses SuiteName:ClassName:TestName instead. If uniqueName is already used on a result, a number is appended to the existing unique name value. For example, if three items have "automatedTest" as their uniqueName, their name values in Helix ALM are automatedTest, automatedTest.1, and automatedTest.2.

uniqueName is not part of the JUnit specification. To set a specific uniqueName in the JUnit report, you would need a custom test report generator that creates <testcase> elements with uniqueName attributes.
Associated test cases tags Used to associate test cases with automated test results. You can specify multiple tags for one automated test result in a comma-separated list. For example, if the XML file specifies <testcase ... tag="TC-1,TC-2"...>, Helix ALM associates test case 1 and test case 2 with the automated test result.

tags is not part of the JUnit specification.
Start date timestamp --
Duration time --
Properties Any other attributes in the <testcase> element

The following attributes in the XML file are displayed in the Automated Test Result Details dialog box in Helix ALM. Any other attributes are displayed as name/value pairs in the Properties area in the Automated Test Result Details dialog box.

  • Browser
  • Browser version
  • Device
  • Manufacturer
  • Model
  • Operating system
  • Operating system version
Status Inferred from data See Automated test result status

Automated test result status

The status of an automated test is inferred from the XML report file. Detailed status information is displayed in the Automated Test Result Details dialog box in Helix ALM. See Viewing automated test result details.

If the <testcase> element contains this child element: It indicates that: In Helix ALM:
None The test passed. The test result is Passed.
<error> An error occurred when running the test. The test result is Failed. If a message is specified in the <error> element in the XML file, it is set as the error message on the test result, which is displayed in the Automated Test Result Details dialog box. If an errorMessage is already specified in the <error> element, the error message is displayed in the Properties area in the Automated Test Result Details dialog box. If a type or value are specified in the <error> element, they are displayed as errorType and errorValue in the Properties area in the Automated Test Result Details dialog box.
<failure> The test failed. The test result is Failed. If a message is specified in the <failure> element in the XML file, it is set as the error message on the test result, which is displayed in the Automated Test Result Details dialog box. If an errorMessage is already specified in the <failure> element, the error message is displayed in the Properties area in the Automated Test Result Details dialog box. If a type or value are specified in the <failure> element, they are displayed as failureType and failureValue in the Properties area in the Automated Test Result Details dialog box.
<skipped> The test was skipped. The test result is Skipped. If a message is specified in the <skipped> element in the XML file, it is set as the error message on the test result, which is displayed in the Automated Test Result Details dialog box. If an errorMessage is already specified in the <skipped> element, the error message is displayed in the Properties area in the Automated Test Result Details dialog box. If a type or value are specified in the <skipped> element, they are displayed as failureType and failureValue in the Properties area in the Automated Test Result Details dialog box.