[Documentation] [TitleIndex] [WordIndex] 

To catch regressions in our code base, we run thousands of tests on all the packages in the ROS distributions. The tests range from small unit tests, to large simulation based integration tests. Our Jenkins server manages and schedules many of the tests. Below you find more details on each class of test that we run.

Contents

  1. Development Tests
    1. What gets tested?
    2. When do tests run?
    3. Who gets a notification?
    4. How long does it take to run them?
  2. Pull Request Tests
  3. Prerelease Tests
    1. What gets tested?
    2. Which repositories can be tested?
    3. When do tests run?
    4. How long does it take to run them?
    5. Prerelease Instructions
      1. Setting up Your Computer for Prerelease
      2. Creating a Prerelease Command
      3. Running the Prerelease
    6. Examples
      1. A set of commands the website provides
      2. Customizing prerelease test commands
    7. What can I do when a prerelease fails?

search

Development Tests

What gets tested?

The development tests will build and test all packages from the development branch of each repository. The development branch is specified in the source sections of the distribution.yaml rosdistro file (One from ROS Indigo | ROS Kinetic). The devel branch is chosen by the maintainer of a repository, and is often trunk/default/master or <DISTRO>-devel. All necessary dependencies will be installed from Debian packages in the shadow-fixed repo before. Currently the development tests are only run for Ubuntu LTS releases for amd64 due to limited resources on the build farm.

When do tests run?

Every time you commit to a development branch the corresponding development tests will get triggered within an hour. It might take a while until the job is processed if the build farm is busy though. For example, you can find the development jobs for Indigo on Jenkins.

Who gets a notification?

The person that did the commit will receive an email if the build succeeds or fails. If the development job is marked as unstable (due to warnings or failing tests) all committers since the last stable build get notified.

How long does it take to run them?

One test typically takes less than 10 minutes. If the build farm is busy rebuilding Debian packages it might take longer until the job actually gets started.

Pull Request Tests

Pull request tests are identical to development tests but they are being triggered for the branches of pull requests. Currently only GitHub repositories are supported. Commonly pull requests need to be specifically enabled (see REP 143). You can find the pull request jobs for Indigo on Jenkins.

Prerelease Tests

Before pushing out a new release of your package(s), you should run the prerelease tests on your repository. This will help you to avoid breaking the ROS Debian builds for the entire ROS community, and having to push out a bugfix release five minutes later. Starting with ROS Jade, prereleases are run locally on your machine using docker. There is a webpage to help you configure the command line tool that runs the prerelease tests on your machine: http://prerelease.ros.org/.

What gets tested?

For the selected repositories (you can select one or multiple) all packages will be built and the tests will be run. All necessary dependencies will be installed from Debian packages in the shadow-fixed repo before. Additionally, it can download, build, and test packages which depend on your packages. The prerelease website can help you to figure out which settings will test the right amount of packages for you. You can control the number of packages that will be tested by adjusting level of recursive dependent packages. You can additionally specify arbitrary additional packages to test. If you want to avoid building a package, for example a package which is known to be broken for another reason, you can also explicitly exclude individual packages, though this might prevent the testing of other packages. The prerelease website can help you see how different settings affect the packages that will be tested.

Which repositories can be tested?

When do tests run?

The prerelease tests only run when you run them on your local machine.

How long does it take to run them?

Low level repositories can take more than an hour to test, while high level repositories can be finished in 5 minutes. The test duration mainly depends on how many repositories depend on the repositories being tested. You can use the prerelease website to help you adjust the number of packages that are tested and built to avoid extremely long builds due to a large number of dependent packages being tested.

Prerelease Instructions

Setting up Your Computer for Prerelease

First, you need to install Docker. We recommend following this guide at docker.io because you'll need a newer version of docker than the default version that comes with Ubuntu 14.04. Make sure to follow the directions to add your user to the docker group so you can invoke docker without sudo.

Install the ros_buildfarm package to get the scripts and tools you need to run the prerelease: 

$ sudo apt-get install python3-ros-buildfarm

If you haven't installed ROS on your machine before, you'll also need to setup apt-get sources to include packages.ros.org. Run the following commands. 

$ sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
$ sudo apt-key adv --keyserver hkp://pool.sks-keyservers.net --recv-key 421C365BD9FF1F717815A3895523BAEEB01FA116
$ sudo apt-get update

You can also install the Python 2 equivalent of the buildfarm, which is called python-ros-buildfarm. For the Python 2 package the template evaluation of the prerelease scripts still use Python 3 internally. Therefore you'll need to have EmPy for Python 3 installed. The Debian package is named python3-empy and is available from Ubuntu Trusty onward. For older versions of Ubuntu, or on other platforms, it can be installed with the following commands: 

  • $ sudo apt-get install python3 python3-pip
$ sudo python3 -m pip install -U EmPy

Creating a Prerelease Command

Once you've set up your computer, use the prerelease website to help you generate a command to run the prerelease: http://prerelease.ros.org/

First, the website will prompt you to select a ROS distribution. Select whichever distribution you want to target with your new package. Next, the site will guide you through a 3-step process to generate a terminal command.

  1. Select Repositories. In this step, you add packages to the prerelease. You're essentially making a mini build of ROS, and you don't want to build and test every package! So in this step, you'll add your new package(s) to the list. If your package is already on the build farm, you should be able to find it in the drop-down list. If your package is brand new or otherwise doesn't appear in the list, you should use the "Add a custom repository" button and fill in the requested information. When you're done, click Next.

  2. Additional Options. In this step, you can choose a custom build farm config file (for advanced users). You should be able to use the default one without issues. You can also choose an operating system to target. This can be useful to test your release on a different OS than the one you're using for local development. Once you've chosen an OS, click Next.

  3. Select Dependents to Test. This step may take a bit of time to load. If your package is brand new, you can ignore the rest of this step and go to Step 4. If your package is already on the build farm and has dependencies, this step will automatically add those dependencies to your prerelease for testing, to ensure that your package's changes didn't break those packages. You can add or remove dependencies for testing as desired (but you should probably test all of them).

  4. The wizard is complete. Click the Generate Command button and copy the commands into your terminal and run them.

Running the Prerelease

Wait, you still haven't done the prerelease yet! Look at the terminal output carefully. The custom command you ran just created prerelease.sh. To actually run the prerelease, run that file: 

$ ./prerelease.sh

Be patient, the command takes a while. The prerelease takes place in a Docker container, and should automatically build your packages, test them, and print the results using catkin_test_results. Make sure that there are no failed tests, and your prerelease is done!

After prerelease, you can run catkin_test_results to get the result of the prerelease tests again. This is an optional step. In order to run that command you need catkin installed and sourced. You'll have to have ROS and catkin installed. Install catkin with the command 

  • $ sudo apt-get install ros-kinetic-catkin

At some point, you also need to source the setup.bash file before you try to use catkin_test_results, by using the command 

  • $ source /opt/ros/kinetic/setup.bash

You can do this at any point, rest assured that your system and environment will not affect the prerelease since they are executed inside the docker container.

Examples

Below a few common prerelease examples are described. The ros_buildfarm documentation contains additional information and examples on prereleases: https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/jobs/prerelease_jobs.rst

A set of commands the website provides

The command set provided by the prerelease test website above would look like the following. Variables that you can configure on the website are surrounded with %%. 

mkdir -p /tmp/prerelease_job
cd /tmp/prerelease_job
generate_prerelease_script.py \
  https://raw.githubusercontent.com/ros-infrastructure/ros_buildfarm_config/production/index.yaml \
  %ROS_DISTRO% default ubuntu %UBUNTU_DISTRO% amd64 \
  %PACKAGE_TARGETED% \
  --level %DOWNSTREAM_DEPTH% \
  --output-dir ./

Customizing prerelease test commands

As you see, the configuration of prerelease test is done by a script generate_prerelease_script.py (that was installed on your computer with python3-ros-buildfarm package). See its option with the help to get some idea for customization. 

generate_prerelease_script.py --help
usage: generate_prerelease_script.py [-h] --output-dir OUTPUT_DIR
                                     [--custom-branch [REPO_NAME:BRANCH_OR_TAG_NAME [REPO_NAME:BRANCH_OR_TAG_NAME ...]]]
                                     [--custom-repo [REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME [REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME ...]]]
                                     [--pkg [PACKAGE_NAME [PACKAGE_NAME ...]]]
                                     [--exclude-pkg [PACKAGE_NAME [PACKAGE_NAME ...]]]
                                     [--level DEPENDENCY_LEVEL]
                                     config_url rosdistro_name
                                     source_build_name os_name os_code_name
                                     arch [REPO_NAME [REPO_NAME ...]]

Generate a 'prerelease' script

positional arguments:
  config_url            The url of the ROS buildfarm configuration index
  rosdistro_name        The name of the ROS distro from the index
  source_build_name     The name / key of the 'source-build' file from the
                        index
  os_name               An OS name from the build file
  os_code_name          An OS code name from the build file
  arch                  An architecture from the build file

optional arguments:
  -h, --help            show this help message and exit
  --output-dir OUTPUT_DIR
                        The output directory

Repositories in underlay workspace:
  The repositories in the underlay workspace will be built and installed as
  well as built and tested. Dependencies will be provided by binary
  packages.

  REPO_NAME             A name of a 'repository' from the distribution file
  --custom-branch [REPO_NAME:BRANCH_OR_TAG_NAME [REPO_NAME:BRANCH_OR_TAG_NAME ...]]
                        A name of a 'repository' from the distribution file
                        followed by a colon and a branch / tag name
  --custom-repo [REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME [REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME ...]]
                        The name, type, url and branch / tag name of a repository,
                            'e.g. "common_tutorials:git:https://github.com/ros/common_tutorials:pullrequest-1"')

Packages in overlay workspace:
  The packages in the overlay workspace will be built and tested. The
  workspace will be extended with packages from GBP repositories for
  dependencies between the overlay and underlay workspaces. All other
  dependencies will be provided by binary packages.

  --pkg [PACKAGE_NAME [PACKAGE_NAME ...]]
                        A name of a released 'package' from the distribution
                        file to be included in the overlay workspace
  --exclude-pkg [PACKAGE_NAME [PACKAGE_NAME ...]]
                        A name of a released 'package' from the distribution
                        file which should be excluded from the overlay
                        workspace
  --level DEPENDENCY_LEVEL
                        The depth of the depends-on tree to be included in the
                        overlay workspace (-1 implies unlimited, default: 2)

For example,

What can I do when a prerelease fails?

Once the generate prerelease command is finished, it will instruct you to run the ./prerelease.sh shell script. This in turn runs a shell script for each step of the prerelease. If a step fails, then you can edit the files which are local (not stored inside the docker container) and then each step of the prerelease process can be rerun by executing the corresponding shell script. These are the steps that you might want to rerun:

If you look in the folder where you ran the prerelease then you'll also notice some folders of interest when debugging:

You can modify code in these two repositories and rerun one or more of the above shell scripts to iterate on the prerelease test and try to resolve issues.

2022-05-28 12:55