Make it possible to cross reference documentation for various harnesses
by adding target locations.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
In order to make it easier for folks to discover available twister
options when using/searching the documentation, or when making external
search in their favorite search engine, inline the output of
`twister --help` into the docs so as to have an always up-to-date
and searchable snapshot of it.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Extend the testcase schema to provide a way to communicate that a given
test is expected to reboot during execution. The generic harness takes
that information and suppresses `already STARTED` warnings if the
restart is expected.
Signed-off-by: Jordan Yates <jordan@embeint.com>
Update the quarantine section in the twister.rst documentation
to include information about excluding specific platforms,
architectures, and simulators
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Allow user to add shell commands in testcase/sample yaml file
alongside the harness_config like in the console harness.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
The BLE acronym is not an official description of Bluetooth
LE, and the Bluetooth SIG only ever refers to it as Bluetooth
Low Energy or Bluetooth LE, so Zephyr should as well.
This commit does not change any board or vendor specific
documentation, and the term BLE may still be used in those.
It will be up to the vendors to update it if they want,
since many of them are using the term BLE in their
products.
Signed-off-by: Emil Gydesen <emil.gydesen@nordicsemi.no>
Add an example of using Twister for an 'out-of-tree' application's
code coverage report generation with `--coverage-basedir` option,
and a note about `gcovr` tool possible problems with symlinks.
Fixes: #83764
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend Twister Harness 'recording' feature to allow multiple
regular expression patterns to extract different types of records
from test output.
Add 'merge' recording mode to collect all extracted data fields
into a single record object of the test instance.
Export to CSV file now takes all field names occurred in the collected
records, sort it alphabetically, and then use it for columns instead of
using only the first record's fields. This is done to address possible
situation when records have different set of fields.
Adjust Twister documentation and test suite to the above changes.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend `--no-detailed-test-id` command line option: in addition to its
current behavior to exclude from a test Suite name its configuration path
prefix, also don't prefix each Ztest Case name with its Scenario name.
For example: 'kernel.common.timing' Scenario name, the same Suite name,
and 'sleep.usleep' test Case (where 'sleep' is its Ztest suite name
and 'usleep' is Ztest test name.
This way both TestSuite and TestCase names follow the same principle
having no parent object name prefix.
There is no information loss in Twister reports with this naming:
TestSuite is a container object for its TestCases, whereas TestSuite
has its configuration path as a property.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Add documentation for the get_filtered_output method in both the Python
implementation and the Integration with pytest chapter. The method is used
to filter command outputs by removing prompts and log messages, making it
easier to process shell command results.
Signed-off-by: Thomas Günther <thomas.guenther@limatica.com>
removed traling whitespace
add vendor_allow and vendor_exclude to the list of filters. The vendor
data is maintained in the board.yml files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This change introduces the ability in twister to select which
emulation/simulation tool to use on the command line.
If none is specified, it will select the first in the list.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Update Test Application diagram as well as Test Scenario and
Test Case naming conventions to make them more clear and aligned
to Ztest suite name included as a part of Test Case name.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extended hardware map to share a single board between variants.
To run tests for different variants on the same board
without re-configuring the hardware map file for each variant,
one can use a `platform` atribute as a list of names.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Currently, Twister Statuses are only useful for
the initiated, save for the very basics.
This is in opposition to the fact
that they are the main thing end users take into
account when checking their Twister run reports.
In order to make Statuses more useful for the end user,
a new documentation page has been created,
elucidating the full meaning of all Statuses.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
There is no way to know for sure the size of an image built for a test,
given that zephyr can build for so many different architectures,
platforms, software supports, etc. So it should be clear that the filter
for min_ram and min_flash on the twister testcases is an estimate, and
not a strictly well known value.
Signed-off-by: Declan Snyder <declan.snyder@nxp.com>
Fixes bad usage of single backticks in lieu of double backticks for
rendering inline literals, or simple '*' for italics.
When appropriate, a better construct than double backticks has been
selected (ex. :file:, :kconfig:option:, :c:func:, ...), or proper :ref:
have been used if the original intention was to have a link.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Expanding the ZTEST shell to include a command that
accepts additional parameters, such as ztest run_testcase
<suite:test> -p any_value.
Adding the new ZTEST_P macro, to create the
parameterized test.
Signed-off-by: Arkadiusz Cholewinski <arkadiuszx.cholewinski@intel.com>
A common question is how twister decides on which platform to run
a given test. Added section is meant to explain those rules.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Extend pytest-args from configuration yaml file with args
from command line instead of overwriting.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add a new fixture, `unlaunched_dut`, which is basically the `dut`
one, but without launching the device - thus also not building the
application. It is useful for tests who need a finer control of
the building of the application, before the dut can be launched
to run it.
It will be used on a future patch, which will use it to enable LLEXT EDK
tests.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Add a diagram to illustrate a test application's project
components with relationships between Ztest and Twister
objects.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
The current version of scipts do not consider OOT boards use cases and
the tests with robot now are strict to only one robot file, which is
not realistic for real environment. This address those issues and allow
multiple testsuits at command line and lists at tests entries. It add
another test parameter to allow configure robotframework options.
Fixes: #74563
Signed-off-by: Gerson Fernando Budke <gerson.budke@ossystems.com.br>
Remove legacy use of 'test case' and replace with `test scenario` where
applicable.
Update terminology and some other parts of twister docs.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Extend Twister Harness recording feature to allow selected data fields,
extracted from the log by a regular expression, to be parsed into JSON
objects and eventually reported in `twister.json` as `recording` list
property of the test suite.
With this extension, log records can convey layered data structures
passed from a test image as summary results, traces, statistics, etc.
This extension also allows flexible recording structure: a test image
can output different types of data records incapsulated into a fixed
set of fields, so `recording.csv` file columns are respected, whereas
some of the columns keep strings with json-encoded semi-structured data.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Allow twister fixtures to contain extra information, which can be used for
test suite configuration. The extra information can be appended to existing
fixtures separated by a colon (i.e. <fixture>:<configuration>).
This is especially useful for the pytest harness, where a fixture of a
given type may need to refer to an instance of a particular piece of host
hardware needed by the pytest suite (e.g. a network interface, a UART, or a
CAN interface connected to the device under test).
Signed-off-by: Henrik Brix Andersen <hebad@vestas.com>
Initial integration with renode-test was introduced in commit bdf02ff,
which added support for calling the `renode-test` command from both west
and twister.
This commit removes the custom run_renode_test target used for running
Robot tests with the `west build` command and makes twister call
`renode-test` directly instead.
Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
Removed Harness's repeat field.
Added to schemas in e722db14ad,
together with the rest of harness_config.
Added to Harness when it was created.
Now present only in one sample -
mec15xxevb_assy6853 power management sample.
Could not find definitive proof whether it has ever been used.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This test is intended to teach the conventions and best practices to a
potential contributor that wants to add a test, for say a new feature.
It is liberally commented on purpose, so new contributors don't have to
spend two weeks understanding the intricacies of the simulator, the bsim
test framework, the native builds, backchannels, etc..
Signed-off-by: Jonathan Rico <jonathan.rico@nordicsemi.no>
On the request of the maintainer.
Background:
We have recently had issues with tests timing out because of the
difference of speed between CI runners and a standard developer machine.
In that case, the logs are printed, and some tests (by yours truly) are
a bit on the chatty side. Github's UI chokes (for a good minute or more
lol) on more than a few hundred lines, so it's a bad experience for
contributors.
Signed-off-by: Jonathan Rico <jonathan.rico@nordicsemi.no>
