Execute the test binary from the output directory instead of directory
where sanitycheck was started.
This will ensure that any artifact created with a relative path by the
test binary will be placed in the output directory instead of creating
the artifact in the directory where sanitycheck was executed and prevent
any possible conflicts.
Signed-off-by: Jan Van Winkel <jan.van_winkel@dxplore.eu>
When Zephyr crashes immediately QEMU reports an error immediately. This
is immediately reported by "make run". Then sanitycheck points the user
at the output of "make run". However the error message(s) are in QEMU's
output which is in a different .log file.
To address this situation point the error message at handler.log
instead of run.log if and only if handler.log is not empty.
To reproduce here's an artificial but very simple crash:
sanitycheck --extra-args=CONFIG_TEST_USERSPACE=n \
-p qemu_x86 -T tests/kernel/mem_protect/stackprot/
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Fix issue where sanitycheck wrongly assumed tests inside ZEPHYR_BASE
to be outside ZEPHYR_BASE and dropped the prefix in their name. This
happened when:
- ZEPHYR_BASE contains symbolic link(s), and
- relative --testcase-root argument(s) are passed
To generate unique names, TestCase.get_unique(testcase_root) first
checks whether "testcase_root" starts with ZEPHYR_BASE. Either may or
may not include symbolic links so both must be canonicalized before
comparison. While fixing this method, replace explicit forward slash
"/" and string replace with os.path.relpath() and make a couple other
simplifications and minor pydoc fixes.
Add new canonical_zephyr_base = os.path.realpath(ZEPHYR_BASE) constant
and corresponding comments and guidelines.
The most visible effect of this mismatch was sanitycheck dropping the
--testcase-root prefix from the unique name of tests inside
ZEPHYR_BASE. This means some test names could be not unique anymore
and silently overwrite each other's results, example:
bash# cd zephyr_dir_with_symlink; export ZEPHYR_BASE=$(pwd)
./scripts/sanitycheck -T samples/portability/cmsis_rtos_v1 \
-T samples/portability/cmsis_rtos_v2
The more systematic and practical consequence (and how I actually
found this) was test outputs landing in unexpected locations.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Notably fix the wrong comment I added in commit 6f011c95c4: when
testing with QEmu sanitycheck does _not_ spawn QEmu; it relies on "make
run" instead.
Searching the code for "Spawning" now cycles directly to all the places
starting processes and threads.
- Sample -v -v verbose output (lines wrapped for commit message check)
Spawning QEMUHandler Thread for \
qemu_x86/samples/hello_world/sample.helloworld 'make run'
- native_posix example:
Spawning process /home/.../sanity-out/native_posix/\
/samples/hello_world/sample.helloworld/zephyr/zephyr.exe
Spawning BinaryHandler Thread for native_posix/\
samples/hello_world/sample.helloworld
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
This reintroduces support for static service in the form of a new API,
BT_GATT_SERVICE_DEFINE, and changes the internal services (GAP/GATT)
to be defined as const as they are never register/unregistered.
Internal service needed to be renamed in order to keep the same order
as before since the section elements are sorted by name.
The result is the following (make ram_report):
before:
gatt.c 572 0.66%
cf_cfg 32 0.04%
db 8 0.01%
db_hash 16 0.02%
db_hash_work 32 0.04%
gap_attrs 180 0.21%
gap_svc 12 0.01%
gatt_attrs 160 0.18%
gatt_sc 80 0.09%
gatt_svc 12 0.01%
sc_ccc_cfg 32 0.04%
subscriptions 8 0.01%
after:
gatt.c 210 0.24%
cf_cfg 32 0.04%
db 8 0.01%
db_hash 16 0.02%
db_hash_work 32 0.04%
gatt_sc 80 0.09%
last_static_handle 2 0.00%
sc_ccc_cfg 32 0.04%
subscriptions 8 0.01%
Signed-off-by: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>
The function error expects only one parameter. The excpetion handler in
scan_path was calling this function with multiple parameters instead of
formatting the string.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Using the new option --timestamps, any output from sanitycheck will have
a timestamp to help identify bottle necks and monitor execution time.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
One of the first things needed when comparing builds of tests across
different environments/systems is to make sure the same (sub)tests were
selected and run in the first place. For that purpose sort the output of
--testcase-report and --discard-report as they were in random order.
Actually make the entire class TestInstance sortable by adding a
standard __lt__() method comparing unique instance names; it could be
useful again.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
After the testcase configs are built, there is a step to
filter all the test case information to determine the set
of tests to run.
As this step takes a nontrivial amount of time, add an
informational message about it.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Update the files which contain no license information with the
'Apache-2.0' SPDX license identifier. Many source files in the tree are
missing licensing information, which makes it harder for compliance
tools to determine the correct license.
By default all files without license information are under the default
license of Zephyr, which is Apache version 2.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some logging text and colors were not escaped correctly, make sure we
generate well formed XML reports.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have not been counting samples in reports. This change lists tests
associated with sample code which in many cases is just verifying output
from the sample and counts as 1 test.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fix --help message. Also rename run_report() to save_tests() as it's
used only once by --save-tests and nowhere else. Maybe the code was
shared with some --other-report feature in the past but not any more.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Also add a "generated by sanitycheck" header to indicate origin and a
warning about the dependency required to actually run the test.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
- Fix wrong --test help message
- Provide more examples to clarify naming hierarchy
- Document that --sub-test runs its entire --test
- Point out that save/load options use their own format
- Document that --list-tests is flattened
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
They're effectively mutually exclusive already because
options.sub_test in main() immediately discards any --test
argument(s). This commit preempts user confusion thanks to this new
message:
sanitycheck: error: argument --sub-test: not allowed with
argument -s/--test
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
$ sanitycheck -h # is re-ordered like this:
< ... all other options ... >
-C, --coverage Generate coverage reports. Implies
--enable_coverage
--coverage-platform COVERAGE_PLATFORM
Plarforms to run coverage reports on. This
option may be used multiple times.
Test case selection:
-f, --only-failed Run only those tests that failed the previous
sanity check invocation.
-s TEST, --test TEST Run only the specified test cases. These are
named by <path to test project relative to
--testcase-root>/<testcase.yaml section name>
--sub-test SUB_TEST Run only the specified sub-test cases and its
parent. These are named by test case name
appended by test function, i.e.
kernel.mutex.mutex_lock_unlock.
--list-tests list all tests.
-F FILENAME, --load-tests FILENAME
Load list of tests to be run from file.
-E FILENAME, --save-tests FILENAME
Save list of tests to be run to file.
-T TESTCASE_ROOT, --testcase-root TESTCASE_ROOT
Base directory to recursively search for test
cases. All testcase.yaml files under here will
be processed. May be called multiple times.
Defaults to the 'samples' and 'tests' directories
in the Zephyr tree.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Commit 45a7e5d076 removed scripts/sysgen
and _k_task_list/_k_event_list sections are no longer being used.
So remove them.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Commit 73cb9586ce renamed linker
section from _k_memory_pool to _k_mem_pool, but the references
in linker scripts are still there. So remove them now.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When re-running failed tests, do not go through any filtering, just load
the failed tests directly and execute them. The filtering was done based
on default platforms and any tests that were failing on non-default
platforms were ignored.
Fixes#13956
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have been overriding OVERLAY_CONFIG coming from tests. Concat all
files and create one variable that we pass to kconfig as list.
Fixes#13320
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Device testing has various problems:
- the main thread doesn't halt the serial monitoring thread when an
error occurs while flashing, meaning flash errors can look like
timeouts
- serial_line is unbound if the ser.readline() call raises an
exception, so the following "if serial_line" would raise NameError
in that case
- the serial monitoring thread is not a daemon, so exceptions or
errors in the handle() thread that cause thread exit will leave the
process sitting around until the serial monitoring thread has a
chance to exit
Fix these issues:
- Rewrite the serial monitoring thread as a select loop over the
serial port file descriptor and a pipe shared by the main thread and
serial monitoring thread. If flashing fails, the main thread uses
the pipe to signal to the serial monitoring thread that it should
exit.
- Make sure serial_line is always bound when read.
- Mark the serial thread daemonic.
Signed-off-by: Marti Bolivar <marti@foundries.io>
If the user has requested verbose output, then use check_call() when
running the flash target so that the build system output goes to the
controlling terminal's stdout instead of being swallowed.
This makes it easier to debug flashing failures.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Parse CMakeCache.txt and filter based on variables defined in CMake.
The CMakeCache.txt parsing is borrowed from west.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some cases sanitycheck handles a build error that started with 'make
run' as a handler crash when it actually failed during building. Right
now we try to attach the handler.log to the XML output even if it did
not exist. Check for the file and if it was not found, go back to the
build.log
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When calling genhtml to generate a coverage report directly from
sanitycheck (-C option), call it with the "--ignore-errors source"
option, so it does not exit when it meets a missing file, but
instead it warns the user and continues.
Fixes#13014
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
There are no longer per-partition initialization functions.
Instead, we iterate over all of them at boot to set up the
derived k_mem_partitions properly.
Some ARC-specific hacks that should never have been applied
have been removed from the userspace test.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Make sure we capture data from gcov and do not timeout before all the
data has been captured. Also report on incomplete data capture.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When building with older versions of the toolchain we need to use gcov
from the same tool set, otherwise data formar might not be compatible.
Added an option to point to the gcov tool from the same toolchain used
to build the code. By default the host gcov is used.
Relates to #12571
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Running sanitycheck from Windows is not supported yet. Unfortunately,
the error message is obscure when a user is unaware of this and runs
it from Windows anyway.
This patch gives an easy to understand error message explaining this
issue.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
While trying to catch crashes after test completes, do not keep
extending the timeout which might lead to and endless loop and not
getting out based on set timeout.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We want to run coverage on select platforms even if a few do support
that now. Specify which platforms should be enabled using the new option
--coverage-platform
By default we have the posix platform configured. Any additional
platforms can be specified using the option above.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The change adds an optional capability to kill process by PID in the
BinaryHandler. At the moment proc.terminate only closed the make
process, also closing the stdin/stdout, but this is impossible to detect
in .NET.
Signed-off-by: Dawid Wojciechowski <dwojciechowski@internships.antmicro.com>
Signed-off-by: Jan Kowalewski <jkowalewski@antmicro.com>
Signed-off-by: Piotr Zierhoffer <pzierhoffer@antmicro.com>
Extend the "slow" flag on tests to inhibit building too. The original
assumption was that building would be fast but running slow, but now
We have tests using a component (OpenThread) that wants to pull and
build software from github for every app.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The segment size computation has value, but it's actually a very slow
process that parallizes poorly. It seems to be bound by the Python
GIL doing the parsing, so never sees more than about 150% of CPU in
use even on wildly parallel systems. And it takes about 75-80
seconds, which is 15-20% of the entire runtime of the test on that
box!
And the only "failure" case this can detect (unexpected sections in
the output file) is now a duplicate of the orphan section warning
we've since enalbled at the linker level.
This defaults to enabling the test to preserve behavior (as I don't
know where all the existing users of the size report are to change
them), but long term we might consider making "disabled" the default
and switching this to an --enable flag.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
When running valgrind in sanitycheck, use the suppression file
and dump the log in its own file.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
If the test process returns an error (return code != 0),
it should be considered a failure, even if the handler
considers the test passed.
But this is not done when we are killing (SIGTERM) the processes,
as then the return code is not necessarily sensible.
Without this, for example, when running with valgrind, all valgrind
errors are missed.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
In addition to Kconfig values from generated .config, also parse and
filter based on configs generated by device-tree.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
