Integrate with your Continuous Integration process

Automation with Hiptest publisher – continuous integration

Once the tests have been successfully generated from an Hiptest project, it is possible to integrate them in a continuous integration process. We will not discuss here the details on how to integrate (as it will change based on your CI tool) but describe the steps necessary to integrate Hiptest in your CI tool.

In this documentation, we will consider the following pre-requisites:

  • you have structured the project for automation and you have been able to generate tests with hiptest-publisher
  • you want to have the results of the Hiptest tests execution back to Hiptest
  • you already have a CI server and task for your project

Create a test run dedicated to the CI

First, create a test run dedicated to the CI. This test run should only contain the tests that have been fully automated.
This test run should not be used for manual testing, only for the results sent by the CI tool. If you need to complete your validation process with some manual testing, use other test runs to set the results.

Once the test run is created, click on the “Plus” menu and then on the “Automate” link:

Opening the automate link.png

You should now see the following menu:

Automate menu content.png

Note: each command line displayed in the menu can be copied to the clipboard by clicking on the clipboard icon on the right of the command line.

There are three command lines displayed in the menu. Let’s detail them a bit further.

Installing hiptest-publisher on the CI

In order to be able to synchronise the test run and the code and also to push the results back to hiptest, you will need to have hiptest-publisher installed on the machine that will run the tests.
This machine will need to have Ruby installed (as you did previously on the developer machine). Then, running the command

gem install hiptest-publisher

will install or update the tool. Note that this is a good practice to have this command line ran before executing the tests every time, in order to ensure that the latest version of hiptest-publisher is installed.

Updating the tests from the test run

The second command line should look something like that:

hiptest-publisher --config-file <path to your config file> --test-run-id 215 --without=actionwords

As the previous command, it should be executed prior to executing the tests, so copy the command in the pre-build part of your CI job configuration.

Its role is to ensure that you are executing the correct version of the tests (the one that is fixed in the test run). It will also add some annotations to the tests so Hiptest will be able to match the results and the tests.

One important thing to note here is the argument “–without=actionwords”. It ensures that only the tests will be regenerated, not the implementation you did for the action words.

Pushing the results back to Hiptest

The third command line should look like this:

hiptest-publisher --config-file <path to your config file> --push <path to your results file> --push-format junit

Its role is to push the results back to Hiptest, so it should be executed after the tests have been executed.
One good practice is to copy it in the post-build part of your CI job, not directly after the command running the tests, otherwise it might not be ran if some test fails (that depends on your CI tool configuration).

The command varies according to the type of report generated by your test framework, so ensure you selected the correct format before copying the command line.

Once every command has been added to the CI job, run it once manually to ensure everything is working fine and you should see the results appear in the test run you created previously.


I used CucumberJs that generates a JUnit format but it doesn’t push the test results to Hiptest. Why?

When CucumberJS exports JUnit reports, it does not place the test UIDs.
Here’s a quick workaround for this king of issue:

  •  in packages.json, use this line to select the cucumberjs-junit package:
"cucumberjs-junit": "git://"
  • when running the tests, use this command line:
node_modules/.bin/cucumber-js --format=json | node_modules/cucumber-junit/bin/cucumber-junit --features-as-suites > junit_output.xml

Now, you can push the junit_output.xml file to hiptest and it should work fine 🙂