Thursday, September 1, 2022

[SOLVED] ctest: Disable a set of labeled tests by default - but run them when explicitly targeted

Issue

I'd like to disable a set of tests by default, but be able to run them when explicitly targeted.

E.g. suppose I have a project with lots of fast-running unit tests that have been added via a add_test(NAME SomeNameHere COMMAND SomeCommandHere) command. And suppose further that there are also a bunch of long-running integration tests that should be excluded from the default ctest run, But it should be possible to run all integration tests with a single command when explicity targeted.

As a concrete example:

  • TestThatShouldAlwaysRun - I'd like this to run when a user runs $ ctest
  • FooIntegrationTest - I'd like this to run only when a user runs $ ctest --some-option-here integration
  • BarIntegrationTest - I'd like this to run only when a user runs $ ctest --some-option-here integration

Here's a my attempt to achieve this.

cmake_minimum_required (VERSION 3.1)
project (HOW_TO_RUN_DISABLED_TESTS NONE)
enable_testing()

# Add a fast running test
# This test should be run when a user runs: 'ctest'
add_test(NAME TestThatShouldAlwaysRun COMMAND echo 'I am always enabled')

# Add a long-running integration test
# This test should be run when a user runs: 'ctest --label-regex my_integration_tests'
add_test(NAME FooIntegrationTest COMMAND echo 'Foo integration test')
set_tests_properties(FooIntegrationTest PROPERTIES DISABLED True)
set_tests_properties(FooIntegrationTest PROPERTIES LABELS my_integration_tests)

# Add another long-running integration test
# This test should be run when a user runs: 'ctest --label-regex my_integration_tests'
add_test(NAME BarIntegrationTest COMMAND echo 'Bar integration test')
set_tests_properties(BarIntegrationTest PROPERTIES DISABLED True)
set_tests_properties(BarIntegrationTest PROPERTIES LABELS my_integration_tests)

Now, let's run this via

$ mkdir build
$ cd build/
$ cmake ..
$ ctest -V

As we can see in the output (posted below) ctest did exclude the integration tests (great!).

1: Test command: /bin/echo "'I" "am" "always" "enabled'"
1: Test timeout computed to be: 10000000
1: 'I am always enabled'
1/3 Test #1: TestThatShouldAlwaysRun ..........   Passed    0.00 sec
test 2
    Start 2: FooIntegrationTest
2/3 Test #2: FooIntegrationTest ...............***Not Run (Disabled)   0.00 sec
test 3
    Start 3: BarIntegrationTest
3/3 Test #3: BarIntegrationTest ...............***Not Run (Disabled)   0.00 sec

100% tests passed, 0 tests failed out of 1

Label Time Summary:
my_integration_tests    =   0.00 sec*proc (2 tests)

Total Test time (real) =   0.05 sec

The following tests did not run:
      2 - FooIntegrationTest (Disabled)
      3 - BarIntegrationTest (Disabled)

But, I can not figure out how I can now, through ctest, run tests labeled with my_integration_tests.

ctest --label-regex my_integration_tests
Test project /path/to/code/example_repo/build
    Start 2: FooIntegrationTest
1/2 Test #2: FooIntegrationTest ...............***Not Run (Disabled)   0.00 sec
    Start 3: BarIntegrationTest
2/2 Test #3: BarIntegrationTest ...............***Not Run (Disabled)   0.00 sec
No tests were found!!!

Is there a way to run disabled tests when explicitly targeted?

I have explored other ways like

  • Not disabeling the integration tests. Then, when a user wants to run the fast running tests they need to specify ctest --label-exclude my_integration_tests. Which isn't great. That puts a burdon on everyone to remember this. I'd like the options that's used the most (running only the fast-running tests) to be runnable via ctest.
  • Not disableing the integration tests but provide the argument CONFIGURATIONS my_integration_tests to the add_test call. But I think that way I am semantically misusing CONFIGURATIONS. Also this doesn't allow me to run only the integration test when targeted.
  • Add the integration tests via add_custom_target. Also here I think I am misusing the API. add_custom_target is meant for custom targets, not as a way to run test, right?

Solution

I believe a simpler way to do this is just via a cmake configuration option.

# cat CMakeLists.txt 
cmake_minimum_required (VERSION 3.1)
project (HOW_TO_RUN_DISABLED_TESTS NONE)
enable_testing()
add_test(NAME TestThatShouldAlwaysRun COMMAND echo 'I am always enabled')
if(BUILD_INTEGRATION_TESTING)
   add_test(NAME FooIntegrationTest COMMAND echo 'Foo integration test')
   add_test(NAME BarIntegrationTest COMMAND echo 'Bar integration test')
endif()

This is simple and also is more flexible, ie. enables addition of test dependent targets to be build (and setup (and ctest fixtures to be used)) for those tests. (I usually in projects see variables named like SOMETHING_BUILD_TESTING so it's similar to BUILD_TESTING cmake variable.)

Another idea I have is to use an environment variable and wrap the test in a script with SKIP_RETURN_CODE (or even without SKIP_RETURN_CODE and just return success if integration tests shouldn't be run):

add_test(NAME FooIntegrationTest 
        COMMAND sh -c "if [ \"\${RUN_INTEGRATION:-}\" ]; then echo 'Foo integration test'; else exit 127; fi"
        VERBATIM)
set_tests_properties(FooIntegrationTest PROPERTIES 
        SKIP_RETURN_CODE 127)

then:

$ ctest -V
....
2: Test command: /usr/bin/sh "-c" "if [ "${RUN_INTEGRATION:-}" ]; then echo 'Foo integration test'; else exit 127; fi" "VERBATIM"
2: Test timeout computed to be: 10000000
2/3 Test #2: FooIntegrationTest ...............***Skipped   0.01 sec
....
The following tests did not run:
      2 - FooIntegrationTest (Skipped)

but:

$ RUN_INTEGRATION=TRUE ctest -V
...
2: Test command: /usr/bin/sh "-c" "if [ "${RUN_INTEGRATION:-}" ]; then echo 'Foo integration test'; else exit 127; fi" "VERBATIM"
2: Test timeout computed to be: 10000000
2: Foo integration test
2/3 Test #2: FooIntegrationTest ...............   Passed    0.01 sec
...


Answered By - KamilCuk
Answer Checked By - David Goodson (WPSolving Volunteer)
at