We have been overriding OVERLAY_CONFIG coming from tests. Concat all
files and create one variable that we pass to kconfig as list.
Fixes#13320
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Device testing has various problems:
- the main thread doesn't halt the serial monitoring thread when an
error occurs while flashing, meaning flash errors can look like
timeouts
- serial_line is unbound if the ser.readline() call raises an
exception, so the following "if serial_line" would raise NameError
in that case
- the serial monitoring thread is not a daemon, so exceptions or
errors in the handle() thread that cause thread exit will leave the
process sitting around until the serial monitoring thread has a
chance to exit
Fix these issues:
- Rewrite the serial monitoring thread as a select loop over the
serial port file descriptor and a pipe shared by the main thread and
serial monitoring thread. If flashing fails, the main thread uses
the pipe to signal to the serial monitoring thread that it should
exit.
- Make sure serial_line is always bound when read.
- Mark the serial thread daemonic.
Signed-off-by: Marti Bolivar <marti@foundries.io>
If the user has requested verbose output, then use check_call() when
running the flash target so that the build system output goes to the
controlling terminal's stdout instead of being swallowed.
This makes it easier to debug flashing failures.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Parse CMakeCache.txt and filter based on variables defined in CMake.
The CMakeCache.txt parsing is borrowed from west.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some cases sanitycheck handles a build error that started with 'make
run' as a handler crash when it actually failed during building. Right
now we try to attach the handler.log to the XML output even if it did
not exist. Check for the file and if it was not found, go back to the
build.log
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When calling genhtml to generate a coverage report directly from
sanitycheck (-C option), call it with the "--ignore-errors source"
option, so it does not exit when it meets a missing file, but
instead it warns the user and continues.
Fixes#13014
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
There are no longer per-partition initialization functions.
Instead, we iterate over all of them at boot to set up the
derived k_mem_partitions properly.
Some ARC-specific hacks that should never have been applied
have been removed from the userspace test.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Make sure we capture data from gcov and do not timeout before all the
data has been captured. Also report on incomplete data capture.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When building with older versions of the toolchain we need to use gcov
from the same tool set, otherwise data formar might not be compatible.
Added an option to point to the gcov tool from the same toolchain used
to build the code. By default the host gcov is used.
Relates to #12571
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Running sanitycheck from Windows is not supported yet. Unfortunately,
the error message is obscure when a user is unaware of this and runs
it from Windows anyway.
This patch gives an easy to understand error message explaining this
issue.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
While trying to catch crashes after test completes, do not keep
extending the timeout which might lead to and endless loop and not
getting out based on set timeout.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We want to run coverage on select platforms even if a few do support
that now. Specify which platforms should be enabled using the new option
--coverage-platform
By default we have the posix platform configured. Any additional
platforms can be specified using the option above.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The change adds an optional capability to kill process by PID in the
BinaryHandler. At the moment proc.terminate only closed the make
process, also closing the stdin/stdout, but this is impossible to detect
in .NET.
Signed-off-by: Dawid Wojciechowski <dwojciechowski@internships.antmicro.com>
Signed-off-by: Jan Kowalewski <jkowalewski@antmicro.com>
Signed-off-by: Piotr Zierhoffer <pzierhoffer@antmicro.com>
Extend the "slow" flag on tests to inhibit building too. The original
assumption was that building would be fast but running slow, but now
We have tests using a component (OpenThread) that wants to pull and
build software from github for every app.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The segment size computation has value, but it's actually a very slow
process that parallizes poorly. It seems to be bound by the Python
GIL doing the parsing, so never sees more than about 150% of CPU in
use even on wildly parallel systems. And it takes about 75-80
seconds, which is 15-20% of the entire runtime of the test on that
box!
And the only "failure" case this can detect (unexpected sections in
the output file) is now a duplicate of the orphan section warning
we've since enalbled at the linker level.
This defaults to enabling the test to preserve behavior (as I don't
know where all the existing users of the size report are to change
them), but long term we might consider making "disabled" the default
and switching this to an --enable flag.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
When running valgrind in sanitycheck, use the suppression file
and dump the log in its own file.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
If the test process returns an error (return code != 0),
it should be considered a failure, even if the handler
considers the test passed.
But this is not done when we are killing (SIGTERM) the processes,
as then the return code is not necessarily sensible.
Without this, for example, when running with valgrind, all valgrind
errors are missed.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
In addition to Kconfig values from generated .config, also parse and
filter based on configs generated by device-tree.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
There's no current need for this and it makes work items
declared with K_WORK_DEFINE() inaccessible to user mode.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Fix false passed on localized error message in make invocation.
Fixes#8348
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We should leave ZEPHYR_BASE alone and create the test output wherever
we are running.
This will support running sanitycheck on test roots other than zephyr's
main git repo.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
right now we assume we are running the tests inside the Zephyr tree and
things do not work well when someone tries to use sanitycheck on tests
outside the tree with a different test root. Cleanup the name handling
and support running outside of Zephyr.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Set COVERAGE option using an overlay instead of piping it through CMAKE
which breaks dependecies, i.e. if someone wants to disable that option
on the application level.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Support specifying a test fixture attached to a board to filter in/out
tests that require this fixture to be able to run a test.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Allow the user to disable the "unrecognized section" test. I can see
multiple use-cases for disabling the test.
If orphan sections exist and are dynamically or unpredictably named
the unrecognized section test will fail.
If out-of-tree sections exist, one might want to temporarily disable
the "unrecognized section" test until one has made it recognized.
The test is disabled through a CLI flag.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
We now place the linker directives for the SW ISR table
in the common linker scripts, instead of repeating it
everywhere.
The table will be placed in RAM if dynamic interrupts are
enabled.
A dedicated section is used, as this data must not move
in between build phases.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
New Qemu is emitting binary charachters which is disturbing generation
of XML, so make sure we only log printable charachters.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This puts the priviledged stack at the end of RAM.
This combines PR #10507 and #10542.
Fixes#10473Fixes#10474Fixes#10515
Signed-off-by: Adithya Baglody <adithya.nagaraj.baglody@intel.com>
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Some boards depend on environment variables so we want to make sure we
do not attempt to build boards requiring additional setup.
Add the section below into the board YAML file, sanitycheck will check
the environment and will only run tests on that board if the variables
are defined.
env:
- VAR1
- VAR2
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
To be able to collect all coverage reports from the "native"
applications in the first CI instance, let's also place the
nrf52_bsim testcases first. The order now is:
unit_testing
native_posix
*_bsim
anything else
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
run tests based on sub-testcases, for example:
sanitycheck --sub-test net.app.app_tcp4_client_hostname_fail
will run all the tests in tests/net/app/net.app.
Useful for re-running tests that are reported in testrail and in reports
generated by sanitycheck.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not check for board/toolchain combination and if a certain toolchain
is configured for a board, run with whatever toolchain we have.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
CMake will translate from the toolchain variant 'gccarmemb' to
'gnuarmemb', but sanitycheck does not. This causes inconsistent and
therefore confusing behaviour between CMake and sanitycheck.
Until gccarmemb is dropped support for, do the same translation with
sanitycheck.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
Add a new option as fixture in harness configurations for utilizing
sanitycheck to identify test cases that require external hardware
such as sensor, ble, networking for validation. The config will be
added to yaml files with unique fixture name to identify each hardware
and allow automation to trigger test execution on setup having the
specific fixture enabled. Also, remove the default required for type and
regex configs that is not essential in case of ztest based test cases.
Signed-off-by: Praful Swarnakar <praful.swarnakar@intel.com>
New shell support features like:
- multi-instance
- command tree
- static and dynamic commands
- multiline
- help print function
- smart tab (autocompletion)
- meta-keys
- history, wildcards etc.
- generic transport (initially, uart present)
Signed-off-by: Jakub Rzeszutko <jakub.rzeszutko@nordicsemi.no>
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Signed-off-by: Piotr Zięcik <piotr.ziecik@nordicsemi.no>
Do not close console after PASS is reported, wait a bit for any
remaining messages from the tests, sometimes we have faults that need to
be parsed.
This now works for Qemu handler, support for other handlers to follow.
Fixes#9646
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
multiprocessing.cpu_count() already returns the number of threads
(not cores), no need to multiply it by 2.
Signed-off-by: Oleg Zhurakivskyy <oleg.zhurakivskyy@intel.com>
