Replaces the check-then-create pattern for symlinks with
opportunistic creation. Instead of checking for existence
before creating the symlink, the code now attempts to
create it directly and gracefully handles the case
where it already exists.
Signed-off-by: Tim Pambor <tim.pambor@codewrights.de>
Use --test-pattern to filter scenarios using regular expressions, for
example:
./scripts/twister --test-pattern '^kernel.semaphore.*' -v
will run all those test using this identifier pattern, and nothing else.
Multiple patterns are possible.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Make this option default and reduce amount of details displayed by
twister.
This options has been enabled in CI for a while now with the intention
of making it default.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Change "Renaming output directory to" message to "Renaming previous
output directory to" to make it clearer that twister is renaming the
existing directory, not the current output directory.
This helps avoid confusion for new twister users who might think the
current output is going to the renamed directory.
Signed-off-by: Afonso Oliveira <afonsoo@synopsys.com>
Use --platform-pattern <regex> to match platforms, something like
twister -T samples/hello_world/ --platform-pattern ".*/stm32f4.*"
will build/run all those platform targets with stm32f4 in them.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This fixes a typo in the formatting code that caused an exception
if the symlink creation failed.
Signed-off-by: Tim Pambor <tim.pambor@codewrights.de>
Split test configuration code into own class and do parsing in two steps
to get the right context of platforms and scenarios.
Fixeszephyrproject-rtos/zephyr#89774
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fix amp to milliamp conversion where a 10^4 factor was applied instead
of 10^3.
Adjust existing test cases accordingly (including a small change to one
of the expected RMS values in pm.power_states that either changed
since the test was introduced or was set too aggressively back then).
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
If test configuration is not quarantined and `--quarantine-verify`
mode is enabled, these should be filtered and not attached
to reports.
Proposed change applies this approach.
Signed-off-by: Katarzyna Giądła <katarzyna.giadla@nordicsemi.no>
If test plan is used, memory usage metrics are overridden.
Result of this operation is missing memory footprint.
This change removes redundand metrics update and fixes
described inconvenience.
Signed-off-by: Katarzyna Giądła <katarzyna.giadla@nordicsemi.no>
With junitparser 4.0.2, the JunitXml.fromfile() now returns a JUnitXml
instance. In order to access the junit.TestSuite instance, we must use
TestSuite.fromelem() to convert it back.
Signed-off-by: Daniel DeGrasse <ddegrasse@tenstorrent.com>
When running pytest tests with coverage enabled, the test process starts
to write the coverage data when it receives the SIGTERM signal. On slow
machines, this can take longer than 0.5s. Increase the timeout and only
attempt to kill the process if it is still running after the timeout.
Signed-off-by: Marco Widmer <marco.widmer@bytesatwork.ch>
Enable `flash-before` mode for devices connected via serial-pty
the same way, as for physical serial.
This mode allows Twister to start reading from the DUT's serial
connection only after the flashing stage has been completed,
thus excluding an early part of the log stream not directly
related to the current test.
It can help in situations, when the device is connected via some
serial-over-network harness which is still transferring device
logs or keeping in its buffer a tail with different RunID etc.
from the previous test image run.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Now that Python 3.12 is the minimum required version for Zephyr, ruff
should target that instead of 3.10.
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
When we have skips in ctest, the overall result should not be marked as
a SKIP unless all tests have been skipped. If we have passing tests,
mark the test as PASS. Any failures/errors obviously will fail the
tests.
Also pass the skip reason from ctest to twister to give a hint about
what is going on in case of skips.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Statuses of quarantined test cases were not updated properly.
Quarantined tests shouldn't cause errors in integration mode.
Removed obsolate filter type.
Signed-off-by: Maciej Perkowski <maciej.perkowski@nordicsemi.no>
Some Build failures are actually cmake issues, so in that case, if
nothing is found as build failure, try to parse for cmake issues.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Skipped status fits quarantined items better than filtered.
Filtered tests are by default removed from reports, which
shouldn't be the case for quarantined tests.
Adjust tests unit and blackbox tests accordingly.
Signed-off-by: Maciej Perkowski <maciej.perkowski@nordicsemi.no>
Verbosity >1 should be enough to see filtered tests. Coupling
it with debug logging level makes the output messy.
Signed-off-by: Maciej Perkowski <maciej.perkowski@nordicsemi.no>
Just move a code with workaroud to convert platform names
to the full name with variants.
It fixes an issue, that not every platforms are printed
before running tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Some simulators - like simics - may end up adding extraneous suffixes to
logged lines. This may cause some regex that match too much fail. This
patch fixes two such cases:
- regex to find RunID changed to only match valid hexadecimal
characters;
- regex to match start of testsuite changed to only match valid word
characters (0-9A-Za-z_).
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
use public queue methods to clear queue on failure
-s Felix Behnke felixbeiderarbeit@gmail.com
Signed-off-by: Felix Behnke <felixbeiderarbeit@gmail.com>
In twister coverage runs we create the gcov sym link to llvm-cov
executable. This code runs multiple times which causes errors with the
current implementation since creating the symlink will fail; then
copying will also fail because we're using the symlink (self copy the
file).
Add a helper function to `try_making_symlink` which will handle the
following cases:
1. If the symlink already exists and is correct, do nothing.
2. If the symlink exists but points to the wrong file, replace it.
3. If the symlink doesn't exist create it.
4. If creating a symlink fails, just copy the file.
Signed-off-by: Yuval Peress <peress@google.com>
This commit improves the alignment of columns in the verbose Twister
output by reclaiming extra space in the testsuite name column when the
platform name oveflows its allocated width.
Signed-off-by: Luca Burelli <l.burelli@arduino.cc>
While cmake/emu/simics.cmake uses SIMICS_PROJECT env var, twister checks
if a build is "runnable" by checking if `simics` is executable, but it
doesn't look at SIMICS_PROJECT env var.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Extend the testcase schema to provide a way to communicate that a given
test is expected to reboot during execution. The generic harness takes
that information and suppresses `already STARTED` warnings if the
restart is expected.
Signed-off-by: Jordan Yates <jordan@embeint.com>
Since BOOT-SERIAL isn't a recognized STMicroelectronics product name,
when Twister saves the new map.yaml, it will also check if the previous
map.yaml had the 'SERIAL-BOOT' product name and attach the corresponding
serial device.It won't generate a new one with an unknown platform name.
Signed-off-by: Fabrice DJIATSA <fabrice.djiatsa-ext@st.com>
Fix missing match on Ztest test case summary log entries, so even
if the test case 'END' log entry is missed or corrupted, its status
will be updated from the Ztest summary log entries, if present.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Kconfig settings of string type is passed to CMake with quotes to ensure
they are properly handled, for example when specified in testcase.yml as
> extra_args:
> - CONFIG_FOO="bar"
Support sysbuild Kconfig settings `SB_CONFIG` by performing the same
quoting for settings starting with `SB_CONFIG_`.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
need add protection when check with sysbuild filter out
application, the domains is NULL
fixes: #87163
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Quarantine filter was not correctly identifying the simulator name.
Now it uses the simulator name from the Platform object,
ensuring that quarantined tests are properly excluded or verified
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
The section lists that govern whether the ELF
section is recognised or not are out of date.
A programming bug has rendered the unrecognised
section check unrunnable for years.
Thus, the practically dead code of the
unrecognised section check is removed
in this commit.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Removed logger.setLevel(...) from all twister modules, becasue
there should be only one place where twister loggers are configured,
and the log level should be set in that place.
This should be done in twister_main.py in setup_logging function.
Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
Add option to force integration mode on a defined list of tests, for
example tests for sample that are identified with 'sample.'.
A sample per definition shall be tested on documented and supported
platforms listed in the sample documentation. Samples shall not be used
as tests to verify functionality of a feature on all available plaforms
in Zephyr.
To still allow testing on platforms not listed in the doc, and when such
platforms are covered by the provided filter, we should still be able to
build/run the tests on those platforms (not directly listed in
integration_platforms).
We detect a sample by its test identifier, i.e., it starts with 'sample.'.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If BB tests failed, crashed, etc., log handles
could linger and crash test cleanup.
Adding log closing to a `finally` section
should prevent those issues.
Loggers should not be set up as globals,
as it makes testing much harder.
Running multiple Twisters may cause for the
logfiles to be still "in use".
When exiting main, close all logfiles
and remove their handlers from all loggers.
Do that for conftest as well.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Rename the driver from uart_native_posix to uart_native_pty.
Including renaming the DTS compatible, and kconfig options, deprecating
the old ones.
And refactor the driver, generalizing it, so we can have any number of
instances.
Note that, unfortunately generalizing to N instances cannot be done
without a degree of backwards compatibility breakage: This driver was
born with all its configuration and selection of the instances based on
kconfig.
When the driver was made to use DT, it was done in a way that required
both DT and kconfig needing to manually coherently enable the 2nd UART.
This has now been fixed, which it means only DT is used to decide how
many instances are avaliable, and UART_NATIVE_POSIX_PORT_1_ENABLE is
just ignored.
Including:
* Deprecate UART_NATIVE_WAIT_PTS_READY_ENABLE: the options is always on
now as it has no practical drawbacks.
* Deprecate UART_NATIVE_POSIX_PORT_1_ENABLE: DTS intanciation defines it
being available now.
* Rename a few functions and in general shorten pseudo-tty/pseudo-
terminal to PTY instead of PTTY.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
MD5 and SHA1 are not supposed to be used nowadays on security context.
Some ancillary scripts in tree do use them, but for verification only -
or where externally mandated, such the SPDX tool.
This patch marks those usages as `usedforsecurity=False`, which helps
clarify intent.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Rename the driver files, binding and kconfig options, while deprecating
the old binding and kconfig options.
Uses in tree are replaced.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Incremental refactoring.
Fix PEP8 issues to make ruff green for config_parser.py module.
Add __init__.py to twister directory to make it a proper python package,
and make modules importable.
Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
Setup logging per process to fix issue on both mac and windows where
handlers are not available to the processes.
Fixes#86237
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
