a72244f2d0
Fix a problem of Ztest suite names not taken into account by Twister to identify a TestCase, so in some situations a Ztest test's status was not assigned to the proper TestCase and it remains 'None' whereas the actual status value lost, eventually the resulting total execution counters not correct. The issue was observed in these situations: * Ztest application with multiple test suites having same test names. * Ztest suite is 'skipped' entirely on execution with all its tests. The proposed solution extends Twister test case name for Ztest to include Ztest suite name, so the resulting identifier looks like: `<test_scenario_name>.<ztest_suite_name>.<ztest_name>` The above naming scheme now requires ztest_suite_name part to be provided for `--sub-test` command line option. Testcase identifiers in twister.json and testplan.json will also include ztest_suite_name component. The Twister Ztest(Test) Harness is improved to track all state changes known from the test application's log for Ztest suites and test cases, so now it parses log output from a Ztest application more scurpulously. Regular expressions to match log records are extended and optimized to compile them only once and, in some cases, fixed (suite summary). Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com> |
||
|---|---|---|
| .. | ||
| pytest_integration | ||
| test_data | ||
| conftest.py | ||
| README.md | ||
| test_cmakecache.py | ||
| test_config_parser.py | ||
| test_environment.py | ||
| test_errors.py | ||
| test_handlers.py | ||
| test_hardwaremap.py | ||
| test_harness.py | ||
| test_jobserver.py | ||
| test_log_helper.py | ||
| test_mixins.py | ||
| test_platform.py | ||
| test_quarantine.py | ||
| test_runner.py | ||
| test_scl.py | ||
| test_testinstance.py | ||
| test_testplan.py | ||
| test_testsuite.py | ||
| test_twister.py | ||
Twister Testing
Running the tests require the environment variable ZEPHYR_BASE to be set.
Twister Testsuite are located in $ZEPHYR_BASE/scripts/tests directory with all the data files in $ZEPHYR_BASE/scripts/test_data directory.
Dependencies
Install all the dependencies using
pip install -r $ZEPHYR_BASE/scripts/requirements-build-test.txt
Executing testsuite
The testcases can be executed from the root directory using
pytest $ZEPHYR_BASE/scripts/tests/twister
Twister Coverage
The coverage for all the tests can be run using the command below. This will collect all the tests available.
coverage run -m pytest $ZEPHYR_BASE/scripts/tests/twister/
Then we can generate the coverage report for just twister script using
coverage report -m $ZEPHYR_BASE/scripts/pylib/twister/
To generate the coverage report for twister script use below command
coverage report -m $ZEPHYR_BASE/scripts/twister
The html coverage report for twister can be generated using
coverage html twister
If needed,the full coverage html report can be generated in every run of "pytest" in the tests directory using configuration file (setup.cfg).
Organization of tests
- conftest.py: Contains common fixtures for use in testing the twister tool.
- test_twister.py : Contains basic testcases for environment variables, verifying testcase & platform schema's.
- test_testsuite_class.py : Contains testcases for Testsuite class (except reporting functionality) in twisterlib.py.
- test_testinstance.py : Contains testcases for Testinstance and Testcase class.
- test_reporting_testsuite.py : Contains testcases for reporting functionality of Testsuite class of twister.