Running automated tests

There are three sets of tests:

The test suite

The test suite is in the tests directory. It runs each test in its own temporary experiment, so it doesn't touch your experiments, and it doesn't need a running midas.

Requirements

You need these system packages:

• a C++ compiler, make, CMake 3.24 or newer and git
• Python 3 and pip
• Node.js 22 or newer and npm. Linux distributions often have an older version, so install it from NodeSource if needed
• lcov, only for coverage reports

For example, on Debian or Ubuntu:

sudo apt install build-essential cmake git python3 python3-pip lcov
curl -fsSL https://deb.nodesource.com/setup_current.x | sudo -E bash -
sudo apt install nodejs

On macOS with Homebrew:

xcode-select --install
brew install cmake git python node lcov

The tests also need about 32 MB of shared memory (/dev/shm on Linux). This matters in Docker containers, which have only 64 MB by default.

The first build of the tests downloads GoogleTest (unless it is already installed), so it needs network access once. A normal midas build doesn't build the tests, and doesn't need network access.

Running everything with make test

From the midas directory:

cd $MIDASSYS
make test

make test has three steps, which you can also run one at a time:

Building the tests by hand

The tests are built by the normal midas CMake build when MIDAS_BUILD_TESTS is on:

cd $MIDASSYS
mkdir build
cd build
cmake .. -DMIDAS_BUILD_TESTS=ON
make
make install

make install is needed for the tests that run midas programs (e.g. mhttpd) or use the midas library from Python, as they use the installed copies in the midas directory. Install the Python and JavaScript test packages first (e.g. with make test-deps), as their tests are only registered with CTest if pytest and tests/resources/node_modules exist when CMake runs. Tests whose prerequisites are missing are skipped rather than failing.

Running the tests with ctest

ctest runs all the tests. Each C++ test case is a separate CTest entry, while the Python and JavaScript tests are run in a few larger entries. Run it from the midas directory:

export MIDASSYS=$PWD
ctest --test-dir build --output-on-failure

Useful options:

Some tests fail on purpose: they check for the correct behaviour where midas still has a known bug. They are listed in tests/expected_failures.txt, and CTest counts them as passing while they fail. So ctest reports no failures on an unmodified midas, and anything that fails is a new problem. If a bug is fixed, its test starts passing and CTest reports it as failed, as a reminder to remove it from expected_failures.txt. The bugs themselves are described in tests/BUG_SUMMARY.md.

Running one suite

C++

Each C++ suite is built into one program, which you can run directly. This gives more detailed GoogleTest output, and --gtest_filter selects tests by name:

./build/tests/src/midas_unit_tests --gtest_filter='OdbXxValues.*'
./build/tests/src/midas_unit_tests --gtest_list_tests

The programs are in the same directories under build. Tests that use an ODB need MIDASSYS to be set, as they run $MIDASSYS/bin/odbinit to create their temporary experiment. To add a test, add a test_*.cxx file to the suite's directory: it is found automatically.

Python

The Python tests are in tests/python, and are run with pytest. They are divided into tiers with pytest markers. The tests that need no midas installation run with:

pip install -e python/
pytest tests/python -m "not needs_midas"

The tests that use the midas library need a built midas, and create their own temporary experiment:

export MIDASSYS=$PWD
export PYTHONPATH=$PWD/python
pytest tests/python -m "needs_midas and not external"

Tests marked external need hardware or a network service, and are left out unless you select them with -m external. In CTest, the two tiers are the python_pure and python_live entries. See tests/python/README.md for the markers and fixtures.

JavaScript

The JavaScript tests are in tests/resources. The unit tests load the JavaScript files of the web pages into a simulated browser (jsdom), and the end-to-end tests run a real mhttpd and drive the Chromium browser with Playwright:

cd tests/resources
npm install
npm run test:unit
npx playwright install chromium
npm run test:e2e

package.json also has scripts that run the unit tests of one file, e.g. npm run test:unit:mplot. In CTest, the two tiers are the js_unit and js_e2e entries. See tests/resources/README.md for details.

Coverage

make coverage builds midas with coverage instrumentation, runs all the tests, and writes a coverage report for each language:

make coverage-recapture measures the coverage again with the existing build, without configuring it again. It doesn't rebuild either, so run make and make install in build first. Some code is deliberately left out of the coverage figures, e.g. obsolete functions. See tests/coverage_exclusions.md for the list and the reasons.

Continuous integration

The GitLab CI job build_and_test (in .gitlab-ci.yml) runs the test suite on Ubuntu 24.04, with ROOT installed. It runs make test-deps and make test-build as root, then make test-run as an unprivileged user, because some tests check what happens when a file can't be read. The job doesn't run on every commit. It runs on a schedule, or when started by hand from the GitLab web interface with the variable JOB_NAME set to full-test. If it fails, the CTest log and the Playwright reports are kept as job artifacts for a week.

Python unit tests

The original unit tests of the Python package are in python/tests, and use the Python unittest framework. Many of them have since been moved into the test suite (tests/python), which runs them in a temporary experiment.

With the environment set up for a test experiment:

cd $MIDASSYS/python/tests
python3 -m unittest                            # all the tests
python3 -m unittest test_odb                   # the tests in test_odb.py
python3 -m unittest test_odb.TestOdb.testInt   # one test

Test programs

midas also builds some programs that test parts of midas by hand, such as test_odb, test_mvodb and torture_odb for the ODB, hwtest for the hardware and operating system, and the fetest frontend. They are installed in bin. Some of them (e.g. test_mvodb and torture_odb) don't report failures through their exit code, so check their output. See Test programs for the list.