When doing a split build and test with coverage enabled, (i.e.
`twister --coverage --build-only` followed by `twister --coverage
--test-only`), Twister attempts to run coverage reports immediately
after building, which fails because tests haven't actually run yet
and causes several error messages to print in to the log.
This change causes twister to skip the call to `run_coverage()` if
`--build-only` is set and print an info message. This suppresses the
error messages from the coverage report tool complaining about missing
coverage files, but still instruments the built code for coverage data
collection. When twister is invoked again, but with `--test-only`
instead, the coverage files will be emitted and the reports
successfully generated.
#### Testing
I ran...
```
$ zephyr/scripts/twister -T zephyr/tests/lib/cmsis_dsp/basicmath/
--coverage -p native_posix --build-only
$ zephyr/scripts/twister -T zephyr/tests/lib/cmsis_dsp/basicmath/
--coverage -p native_posix --test-only
```
... followed by a combined build and test ...
```
$ zephyr/scripts/twister -T zephyr/tests/lib/cmsis_dsp/basicmath/
--coverage -p native_posix
```
... as a control and diffed the HTML coverage reports, which were
identical other than a timestamp.
Signed-off-by: Tristan Honscheid <honscheid@google.com>
Do not modify testplan.json if it already exists. Testplan is only
created once and any work in the same worskpace should not change it.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Get options from passed environment class instead of using command line
options to the init function.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move all code related to instance filtering and loading into the class
instead of having it in the main script. Simplify option passing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Previously, twister would exit with a 0 status code if there were build
failures but no execution failures. This makes it easier to catch
build breakages when using twister in CI workflows.
Signed-off-by: Benjamin Gwin <bgwin@google.com>
Add a filter type to distinguish between testsuite filters that are
maintained in the testcase.yaml file and those provided on the command
line.
The issue was that when something is excluded deliberately on the command
line twister would report an error if that filter would filter out an
integration platform. We do not want that, because the filtering is
being done by the caller knowing that some tests would be not run, even
if those are integration platforms.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Colorama, by default, strips out the color escape characters when the
output is redirected, and this may have an unintended consequence of
disabling color outputs when using a utility such as `tee` and in the
CI runners that redirect the stdout and stderr console outputs.
This commit adds a new command line option called `--force-color` to
always force the ANSI color escape sequence output even when the output
is redirected.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
This commit adds a call to the Colorama initialisation function during
the module execution so that ANSI color sequences are properly
converted to the relevant Win32 API calls on the Windows.
Note that the init function needs to be called per module, hence it is
also called in the `twisterlib.py`.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
The flag --testcase-root has been renamed to --testsuite-root in
15bc98eb50 twister: rename TestCase -> TestSuite
Fix the --help text to reflect the change.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Filtered tests are now not being reported by default to allow generation
of reports that are easier to parse and work with in different tools.
The complete filtered set of tests is still available in the json output
for review and verification.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This was forgotten from an early overhaul, fix this as
connected_hardware attribute does not exist anymore.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Remove some legacy code and set status and failure/skip reason directly
without set_state and get_state.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Remove unused arguments that became obsolete through the refactoring.
- filter_status
- only_failed
- json_report
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
TestCase in the old terminology is the test application, but we have
just freed up TestSuite, so use TestSuite here to signify the test
application which consists of many test scenarios and testcases.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Change terminology and fix usage based on the current hierarchy. A
testsuite is each test application we have in the tree. The top class is
where we detect all the test suites and create the testplan to be run
later.
Content of testplan will have to split out, so it only deals with the
plan and not the overall execution of tests. This step will be done at a
later point.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
An obsolete feature that is no longer being used due to the amount of
data that is needed per release and the fact it becomes stale very fast.
Instead, point to files directly for comparison.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Drop CSV generation which was the standard for sharing information since
the early days and instead use json with more content to share
information across sessions and for report generation.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Specify why something was skipped (either filtered or through a ztest
case) and do not report testcases that are only being built as passed,
mark those tests as skipped with the reason specified as "Build only".
Keeping --no-skipped-report for now, although it has no impact.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Many option cancel out very already on,. so lets do this as part of the
option processing function first before we create workspace.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Cleanup main() and move logging code into own function.
This reduces clutter in main function and simplifies workflow.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Support new keyword in the testcase.yaml to allow filtering based on
enabled modules. This is useful when you only whitelist a limited set of
modules in your project and want to only run those samples that have
their dependencies satisfied and those modules available.
The option allows adding multiple modules, all of which need to be
enabled in the manifest.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
if --enable-valgrind is selected but no valgrind executable can be
found in PATH, exit with an error
Signed-off-by: Mikkel Jakobsen <mikkel.aunsbjerg@escolifesciences.com>
