To make possible to build bsim tests by Twister, it is necessary to
copy executables to BabbleSim bin directory.
Signed-off-by: Piotr Golyzniak <metody159@gmail.com>
- Support for listing both testcases and testsuites
- Support for running single suites or single test cases
- Support shuffling tests and repeating execution based on command line
arguments.
For example, build with
west build -p -b qemu_cortex_m3 tests/kernel/sleep -t run -- \
-DCONFIG_ZTEST_SHUFFLE=y -DCONFIG_ZTEST_SHELL=y
Following commands are available:
uart:~$ ztest
ztest - Ztest commands
Subcommands:
run-all :Run all tests
shuffle :Shuffle tests
list-testsuites :List all test suites
list-testcases :List all test cases
run-testsuite :Run test suite
run-testcase :Run testcase
shuffle accepts two arguments --suite_iter and --case_iter which allows
repeated exercution of testcases or suites.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Twister now uses GCOVR by default as the more reliable code
coverage reporting tool instead of LCOV.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Add README.rst file with description of sample which shows how to use
pytest and Twister together.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Improve documentation about pytest integration with Twister. Add
examples of usage, improve description of available options and
introduce automatic doc generation of two plugin classes (DeviceAdapter
and Shell) basing on their docstrings from source code.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
The Console Harness is able to parse its log with patterns to compose
extracted fields into records in 'recording.csv' file in the test's build
directory. This feature allows to extract custom test results like
performance counters.
With this change the extracted records are also written into 'twister.json'
as a part of each test suite object. This makes easier to store
all the data collected by the test for its further processing.
Other improvements:
- compile parsing pattern only once instead of at each input line;
- quote fields in '.csv' to avoid unexpected field separators;
- make 'regex' a required schema field of 'harness_config';
- Twister documentation update.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
One can not even replace sucessfully pytest basic sample `pytest-args`
with command line "--pytest-args", as all it does is to append a single
string to current list of commands, making it impossible to send several
arguments.
This patch fixes that by allowing several instances of `--pytest-args`
to compose the whole list of arguments to be passed to pytest.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Extend Twister command line with --pytest-args. This parameter
is passed to pytest subprocess. It allows to select a specific
testcase from a test suite.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
In the docs replace references to native_posix with native_sim
Switch the default test platform to native_sim from native_posix
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Update this page to account for the fact that now
we have several bsim boards in tree.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
The native device type name is "native", "native posix" is just
one of the possible target boards (which is going to be deprecated
in the next release).
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Added pytest_dut_scope keyword under harness_config section.
New keyword is used to determine the scope of dut and shell
fixtures in pytest-twister-harness plugin.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
- Added language info to code-blocks when relevant (ex. for yaml
snippets)
- Formatted grammar for expression language as antlr and added
syntax highlighting
- Fixed various formatting issues with lists, definition lists...
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Document how to use pytest_root keyword. Update docs
with changes in pytest-twister-harness plugin.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
the Test Framework example present in documentation does not work
out of the box and has to be slightly modified. This is an attempt
to fix the example so it would just work directly from documentation.
Signed-off-by: Andrei Hutanu <andrei.hutanu.i@gmail.com>
We use ZTEST_DMEM and ZTEST_BMEM for running tests with memory shared
between userspace and kernelspace. This ought to be documented.
Document ZTEST_DMEM and ZTEST_BMEM.
Signed-off-by: Aaron Massey <aaronmassey@google.com>
Enable the possibility for boards to implement a custom `run` target in
its board.cmake to run any arbitrary commands. This is helpful for devs
who would like to add support for proprietary simulator to their boards
that can't be upstreamed.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Add DeviceAbstract class to default imports from pytest-twister-harness
package to simplify importing DUT package, when creating tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add support for calling the `renode-test` command from west and twister.
Enable running Robot Framework tests suites in Renode.
Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
Signed-off-by: Mateusz Hołenko <mholenko@antmicro.com>
Some 3P logic (Pigweed for example) is already heavily invested in
gTest as a testing framework. Adding the `gtest` harness enables
running the existing 3P tests on various Zephyr platforms and configs.
Signed-off-by: Yuval Peress <peress@google.com>
For users convenience, as it is a typical usecase to
want to see the actual coverage of a set of tests.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
About how do we use it in general, and with lots of links so
people can find more details.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Use explicit flash command execution timeout at DeviceHandler
instead of the hardcoded value of 60 sec.
When a HW platform executes test cases right after the flash command,
the test case remaining timeout is affected by how much time the flash
operation consumes. Some simulation platforms need to spend significant
amount of time on each 'flash' cycle, usually adding the same delay on
each test case.
This improvement adds two new command line options and device map fields:
--device-flash-timeout - for the flash operation timeout
('flash-timeout' device map field).
--device-flash-with-test - to indicate that the platform flash
command also runs a test case, so the overall timeout should be
calculated as a sum of the flash timeout and the current test case
timeout to receive all console output from the platform
('flash-with-test' device map field).
The device map field values override command line values for the
particular platform where configured.
Default behavior is backward compatible: flash operation fixed timeout
is 60 sec. not including the test case timeout.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
This fixes a typo in the testing framework docs where the
ZTEST_EXPECT_SKIP MACRO seems to be referring to the wrong function.
Signed-off-by: Ivan Herrera Olivares <ivan.herreraolivares@gmail.com>
Add supplementary header <zephyr/fff_extensions.h>. Add macro
to fff_extensions.h for simplifying definition of custom fake functions
needing call-unique information for producing desired output data.
When an array of custom fake context structures is defined and
the return field within the first structure instance is registered
with the standard SET_RETURN_SEQ() macro of FFF, the
RETURN_HANDLED_CONTEXT() macro provides the inverse logic to
recover the context structure for this called instance. The body of
the custom fake handler is provided to the RETURN_HANDLED_CONTEXT()
macro for appropriate execution and access to the custom fake
parameters.
A test suite is also provided to verify macro implementation and
illustrate usage. It is at:
zephyr/tests/subsys/testsuite/fff_fake_contexts/
This code was verified by:
1. (Pass) west build -p always \
-b unit_testing tests/subsys/testsuite/fff_fake_contexts/ && \
./build/testbinary
2. (Pass) west build -p always \
-b native_posix tests/subsys/testsuite/fff_fake_contexts/ && \
./build/zephyr/zephyr.exe
3. (Pass) ./scripts/twister -p unit_testing \
-T tests/subsys/testsuite/fff_fake_contexts/
4. (Pass) ./scripts/twister -p native_posix \
-T tests/subsys/testsuite/fff_fake_contexts/
5. (Pass) cd doc && build html-fast
Fix#55246
Signed-off-by: Gregory Shue <gregory.shue@legrand.com>
This page is:
- using literal blocks [1] where it should be using code
blocks [2], which won't produce the desired results [3]
- missing required whitespace between code block directives and their
pygments syntax arguments [4]
In other words, this:
.. code-block::foo
should be:
.. code-block:: foo
- using an incorrect syntax (option:`CONFIG_FOO` or
:kconfig:option:`CONFIG_FOO=y1) to refer to Kconfig
options (should be :kconfig:option:`CONFIG_FOO`, etc)
Fix these issues so we get syntax-highlighted code blocks in the
correct pygments highlight syntax and valid references to Kconfig
options.
Adjust directive content indentation to match the usual .rst style and
make some grammar tweaks as well.
[1] https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#literal-blocks
[2] https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-code-block
[3] Use of literal blocks can produce the same results if there is a
consistent choice of highlight language specified for the file, but
this file is intermixing code blocks that should be highlighted as
either C or shell commands, so there is no single choice of
highlight syntax that works.
[4] https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#directives
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Twister allows filtering based on kconfigs and dts, however the
filtration is a part of the cmake stage, i.e. the stage has to pass
first and then twister checks if required properties are available.
This causes problems, when the full cmake stage is unable to pass.
If so, other filtration methods had to be used, e.g. platform_allow.
The commit modifies the twister workflow:
if a test defines filters based on kconfig/dts first a cmake package
helper script is used to extract dt and/or kconfigs and if the
conditions are fulfilled it proceeds to a regular cmake stage.
If not, test is skipped.
Co-authored-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Add the zexpect API, inspired by GoogleTest's EXPECT API. This API reports
test failures while allowing test execution to continue. This enables test
reports to show more than a singule failing property on a failing test.
Signed-off-by: Aaron Massey <aaronmassey@google.com>
This commit documents the new SCA tool infrastructure.
The existing documentation for sparse are relocated into a dedicated
documentation folder for SCA tools.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
