This test fails on older python versions (3.10) and only on CI.
Disabling it while we investigate. The test itself verifies inline logs
options, so the functionality test is not impacted.
Tracked in #87769
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Removed logger.setLevel(...) from all twister modules, becasue
there should be only one place where twister loggers are configured,
and the log level should be set in that place.
This should be done in twister_main.py in setup_logging function.
Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
If BB tests failed, crashed, etc., log handles
could linger and crash test cleanup.
Adding log closing to a `finally` section
should prevent those issues.
Loggers should not be set up as globals,
as it makes testing much harder.
Running multiple Twisters may cause for the
logfiles to be still "in use".
When exiting main, close all logfiles
and remove their handlers from all loggers.
Do that for conftest as well.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Rename the driver from uart_native_posix to uart_native_pty.
Including renaming the DTS compatible, and kconfig options, deprecating
the old ones.
And refactor the driver, generalizing it, so we can have any number of
instances.
Note that, unfortunately generalizing to N instances cannot be done
without a degree of backwards compatibility breakage: This driver was
born with all its configuration and selection of the instances based on
kconfig.
When the driver was made to use DT, it was done in a way that required
both DT and kconfig needing to manually coherently enable the 2nd UART.
This has now been fixed, which it means only DT is used to decide how
many instances are avaliable, and UART_NATIVE_POSIX_PORT_1_ENABLE is
just ignored.
Including:
* Deprecate UART_NATIVE_WAIT_PTS_READY_ENABLE: the options is always on
now as it has no practical drawbacks.
* Deprecate UART_NATIVE_POSIX_PORT_1_ENABLE: DTS intanciation defines it
being available now.
* Rename a few functions and in general shorten pseudo-tty/pseudo-
terminal to PTY instead of PTTY.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Rename the driver files, binding and kconfig options, while deprecating
the old binding and kconfig options.
Uses in tree are replaced.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
These files can be useful to troubleshoot a test that's failing,
and they're really small, too.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Updated test_report_summary to match new string with detailed
reason of build failure.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
With this change, the coverage data (GCOV dump) is extracted from the
test log files after each of the test instance parallel execution,
instead of post processing all the logs at the end of the Twister run.
The new `--coverage-per-instance` mode extends Twister coverage operations
to report coverage statistics on each test instance execution individually
in addition to the default reporting mode which aggregates data to one
report with all the test instances in the current scope of the Twister run.
The split mode allows to identify precisely what amount of code coverage
each test suite provides and to analyze its contribution to the overall
test plan's coverage. Each test configuration's output directory will have
its own coverage report and data files, so the overall disk space and
the total execution time increase.
Another new `--disable-coverage-aggregation` option allows to execute
only the `--coverage-per-instance` mode when the aggregate coverage
report for the whole Twister run scope is not needed.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Added integration_toolchains to allow building/testing with multiple
toolchains available in the environment.
This changes the output structure and adds another level in the path
under twister_out signifying the toolchain used. The toolchain used
(variant) is also part of the json output now.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Set missing TestCase statuses when a test under the BinaryHandler failed
(crashed), so remaining 'STARTED' and 'NONE' are now 'BLOCK' instead.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend `--no-detailed-test-id` command line option: in addition to its
current behavior to exclude from a test Suite name its configuration path
prefix, also don't prefix each Ztest Case name with its Scenario name.
For example: 'kernel.common.timing' Scenario name, the same Suite name,
and 'sleep.usleep' test Case (where 'sleep' is its Ztest suite name
and 'usleep' is Ztest test name.
This way both TestSuite and TestCase names follow the same principle
having no parent object name prefix.
There is no information loss in Twister reports with this naming:
TestSuite is a container object for its TestCases, whereas TestSuite
has its configuration path as a property.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix Ztest test function name extraction from ELF symbols
for C++ compiled binaries where symbol names need additional
'demangling' to match with corresponding test names.
The `c++filt` utility (part of binutils) is called for
demangling when it is needed.
Twister test suite extension and adjustment.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix a problem of Ztest suite names not taken into account by Twister
to identify a TestCase, so in some situations a Ztest test's status
was not assigned to the proper TestCase and it remains 'None'
whereas the actual status value lost, eventually the resulting total
execution counters not correct.
The issue was observed in these situations:
* Ztest application with multiple test suites having same test names.
* Ztest suite is 'skipped' entirely on execution with all its tests.
The proposed solution extends Twister test case name for Ztest to
include Ztest suite name, so the resulting identifier looks like:
`<test_scenario_name>.<ztest_suite_name>.<ztest_name>`
The above naming scheme now requires ztest_suite_name part to be
provided for `--sub-test` command line option.
Testcase identifiers in twister.json and testplan.json will also
include ztest_suite_name component.
The Twister Ztest(Test) Harness is improved to track all state changes
known from the test application's log for Ztest suites and test cases,
so now it parses log output from a Ztest application more scurpulously.
Regular expressions to match log records are extended and optimized
to compile them only once and, in some cases, fixed (suite summary).
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix wrong reporting about where testcases were executed.
We have:
INFO - 1130 of 1130 executed test cases passed (100.00%) on 0 out of
total 860 platforms (0.00%).
which is obviously wrong.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fix Twister TestCase statuses left not assigned ('NONE') in these 'NOTRUN'
situations:
* a test suite has `integration platform` which is not available to run.
* `--cmake-only` execution.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
When using the --build-only into --test-only
Twister setup, NOTRUN statuses were not properly rerun.
Now they are properly run again if runnable.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
ExecutionCounter has been expanded and now hold i.a.
more information on the statuses of TestCases.
This information is now incorporated in relevant summaries
- runner.py and reports.py.
Layout of those was changed to present that
and previous information in a clear and concise way.
TestInstance execution counter now is more intuitive.
Instances filtered out before running are no longer included there.
Retries now properly reset the counter.
TestCases with None and other incorrect final statuses
are logged as errors, but do not
exit Twister with a nonzero exit code.
This is because None statuses, although incorrect,
are currently common.
Inconsistent spacing in ERROR and FAILED fixed.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
scripts: Dmitri fix
Fix of a problem noticed by Dmitri
Removed unnecessary additional spaces when
printing FAILED and ERROR status.
Now TwisterStatus.get_color is used more.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
As per its creators, CMake is written with a capital "M".
The initial reason for this change is that I want Twister to print
"ERROR : CMake build failure" instead of "ERROR : Cmake build failure".
Signed-off-by: Reto Schneider <reto.schneider@husqvarnagroup.com>
Adds a new NOTRUN status, which indicates
that a test was successfully built, but
not run on account of being not
runnable in given test instance.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
native_posix is now deprecated.
Building this sample in both native_sim and native_posix does not
improve coverage for the sample or subsystem but doubles CI time.
As anyhow native_posix will be removed all together in
2 releases, let's remove it already for this sample.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Currently, debug logging in the console and verbosity
are tightly coupled - verbosity of level 2 and higher
enables logging at the debug level.
This change introduces a separate Twister flag
responsible for controlling the debug logging,
while leaving the rest of verbosity unchanged.
This allows for controlling the verbosity on
both logging levels, according to one's needs.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Some dict.get() calls did not use a TwisterStatus
as a default value, thus using a NoneType where
TwisterStatus should appear.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Various different Statuses were joined into a single class,
TwisterStatus. This change anticipates further streamlining
of the Twister's approach to Status.
Code guarding Twister's Properties was shortened to a
value check only.
QEMUOutputStatus was left separate, as doubts were cast
whether it should remain a status. Leaving it separate makes
its removal easier.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Now statuses are not just a str that can be easily mistyped
or assigned wrong. Now they are an Enum.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Do not depend on platforms that need a HAL. This should speed things up
and should resolve issues where runner did not have enough space to deal
with all HALs.
t
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
`checkpatch.pl` requires that dts sources are indented with tabs,
fix all the spaces that slipped in while checkpatch wasn't watching.
Signed-off-by: Jordan Yates <jordan@embeint.com>
we use reason for a failure to indicate state and then set the status
later and reason for the failure, in case of the failure is taken from
the handler status. Clean this up by setting status and reason coming
from the handler very early, so we do not have to go through replacing
meaning later.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Filtered testcases are removed by default from Twister tests.
Older functionality is preserved via a new Twister flag:
--report-filtered.
Old tests were adjusted and a new test for that flag added.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Sped up Twister via name prefix loading.
If the only thing influencing the platform selection
is the platform filter (-p), then we only load *.yaml
files that start the same way as those selected platforms.
We split the platform name to the first '/' or '@'.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Improve Twister command line arguments for memory footprint:
* group and order footprint arguments meaningfully,
* clearer help descriptions,
* resolve logical inconsistences for combinations of arguments,
in particular:
`--last-metrics` now forces `--enable-size-report`,
`--show-footprint` now controls only detailed log output of
footprint deltas in comparison modes.
* align twister tests accordingly.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Blackbox tests related to platform choice
now moved to test_platform.py:
* test_emulation_only
* test_exclude_platform
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Add test related to disable-suite-name-check flag
This flag disabling extended test suite name verification at
the beginning of Ztest test. This option could be useful for tests
or platforms, which from some reasons cannot print early logs.
Add test related disable-warnings-as-errors
Do not treat warning conditions as errors.
Signed-off-by: Artur Wilczak <arturx.wilczak@intel.com>
Adds tests related to the addon flags:
* --enable-ubsan
* --enable-lsan
* --enable-asan
* --enable-valgrind
* --allow-installed-plugin
* --pytest-args
* -x, --extra-args
* extra test args for the binary
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
