Integrate with your Continuous Integration process

Automation with Hiptest publisher – continuous integration

Once the tests have been successfully generated from an Hiptest project, it is possible to integrate them in a continuous integration process. We will not discuss here the details on how to integrate (as it will change based on your CI tool) but describe the steps necessary to integrate Hiptest in your CI tool.

In this documentation, we will consider the following pre-requisites:

  • you have structured the project for automation and you have been able to generate tests with hiptest-publisher
  • you want to have the results of the Hiptest tests execution back to Hiptest
  • you already have a CI server and task for your project

Create a test run dedicated to the CI

First, you will have to create a test run dedicated to the tests that will be run by the CI tool. Once created, note the ID of this test run (it can be found in the test run URL, just after the “testRuns” part. In the image below, the ID is 1464):


Now update your hiptest-publisher configuration file in order to specify the test run ID by adding the following line:

test_run_id = <your test run ID>

Regenerate the tests by using the following command:

hiptest-publisher -c <path to your configuration file> --only=tests

You should now see that all the scenarios names have been updated to include the UIDs of the test run scenarios.

Configure for pushing the results back to Hiptest

Once the tests are generated and working, you can tell the CI to push back the results to Hiptest. The first step is to configure the test framework to generate one of the four formats of report handled by Hiptest:

  • jUnit XML (which is handled by many test frameworks. It is also called xUnit sometimes)
  • NUnit XML (the default XML output for NUnit)
  • TAP (Test Anything Protocol)
  • Robot framework XML (the output.xml file generated by default when running the Robot framework tests)

Now edit the hiptest publisher configuration file to add the information about this report file:

push_format = <type of report: tap, unit, nunit or robot>

On the CI slave(s), you need to have Ruby and hiptest publisher installed (as it was done before on the developper machine). In the task, you have to add a post-execution script that will use hiptest-publisher to send the report file to Hiptest. The command line to execute is:

hiptest-publisher -c <path the hiptest publisher configuration file> -p <path to the execution report file>

Next time the CI task is ran, the results of the execution will be pushed to the Hiptest test run.

Updating tests before running them

If you need, you can update the tests before running them using the latest definition from Hiptest. Note that this need a stable set of action words (if new action words are added and used in test, they will fail until action words are implemented). To do so, you will have to add a pre-execution script to your CI task:

hiptest-publisher -c <path to the configuration file> --only=tests


I used CucumberJs that generates a JUnit format but it doesn’t push the test results to Hiptest. Why?

When CucumberJS exports JUnit reports, it does not place the test UIDs.
Here’s a quick workaround for this king of issue:

  •  in packages.json, use this line to select the cucumberjs-junit package:
"cucumberjs-junit": "git://"
  • when running the tests, use this command line:
node_modules/.bin/cucumber-js --format=json | node_modules/cucumber-junit/bin/cucumber-junit --features-as-suites > junit_output.xml

Now, you can push the junit_output.xml file to hiptest and it should work fine 🙂