Fixes a bug causing tests with error status get silently skipped
when --subset is used.
Fixes: #39619
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Fixes#38046
Document the changes in twister.rst
Add baud rate to debug message in twister
Add baud parameter to twister's add_device function
Set the twister baud rate from input parameters
Use 115200 as the default baud rate if not specified
Add baud to the hwmap-schema.yaml file
Add --device-serial-baud to twister arguments
Fix compliance issues
Fix mistake in parameter name from device-baud to device-serial-baud
Refactoring of the code in orded to simplify the logic
and clear multiple definitions of the default baud rate.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Signed-off-by: Dennis Ruffer <daruffer@gmail.com>
Do calculation based on what is actually going to be run and evaluated
at runtime, ignore the cases we already know going to be skipped.
This fixes an issue where some sets would get majority of skips and
basically run nothing beside filtering.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The info of rom/ram usage by an application was lost along the
way of data processing in twister. The commit add a line which
pass further these metrices as well.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Adds feature allowing to use yaml file with dictionaries defining
tests to be quarantined (extra arg "--quarantine-list FILENAME").
The dictionaries are validated according to the proper schema
and loaded.
A flat list is created containing quarantined configurations
(configuration = platform + scenario). Configurations under quarantine
are skipped and get "Quarantine" as a reason in the results reports.
A "comment" can be added to a quarantine entry in the quarantine yaml
with more details (e.g. issue #) and it will be also added to
the report.
The status of tests under quarantine can be verify if
`--quarantine-verify` is used in addition to
"--quarantine-list FILENAME". Using these args will make twister skip
all tests which are not on the quarantine list.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Fix twister output. "Test suite" used previously did not correspond
to what was counted (test scenarios). A test scenario is an entry from
testcase/sample.yaml under "tests:", which specify configuration used to
build a test. Changes in docs will follow later on. Also replace the
amount of preselectd platforms with the amount of platforms that
were not filtered out and actually tested.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Change the sorting of test instances if --device-testing is used
within subsets. Test instances are sored depending on the context.
For CI runs the execution order is: "platform1-testA, platform1-testB,
..., platform1-testZ, platform2-testA, ...". For hardware tests, were
multiple platforms can run the tests in parallel, it is more efficient
to run in the order: "platform1-testA, platform2-testA, ...,
platform1-testB, platform2-testB, ..."
This can significantly reduce the tests execution time
for setups with multiple different platforms connected.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Now we create plaform reports for each enabled platform, this is time
consuming, especially if you are not using those reports for anything.
Make this an option and enable only if the generated reports are going
to be used.
Individual reports can also be generated from the JSON file if needed.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
