This file uses strtok_r which is an extension to the the
std C library. Let's explicity select one of the extensions
which includes it instead of relaying on somebody having
set it for this file somewhere else.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Export the minimal set of ztest_test_* symbols to llexts so that the
zassert_* macros can be used by the extension code to affect test
execution. Calling these also requires vprintk() to be exported.
Signed-off-by: Luca Burelli <l.burelli@arduino.cc>
- Support for listing both testcases and testsuites
- Support for running single suites or single test cases
- Support shuffling tests and repeating execution based on command line
arguments.
For example, build with
west build -p -b qemu_cortex_m3 tests/kernel/sleep -t run -- \
-DCONFIG_ZTEST_SHUFFLE=y -DCONFIG_ZTEST_SHELL=y
Following commands are available:
uart:~$ ztest
ztest - Ztest commands
Subcommands:
run-all :Run all tests
shuffle :Shuffle tests
list-testsuites :List all test suites
list-testcases :List all test cases
run-testsuite :Run test suite
run-testcase :Run testcase
shuffle accepts two arguments --suite_iter and --case_iter which allows
repeated exercution of testcases or suites.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Compiler can't tell that k_thread_abort() won't return and issues a
warning unless we tell it that control never gets this far.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
rand32.h does not make much sense, since the random subsystem
provides more APIs than just getting a random 32 bits value.
Rename it to random.h and get consistently with other
subsystems.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Rather than yielding to idle thread, keep the part awake.
So debugger can still access it,
since some SOCs cannot be debugged in low power states.
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Flaky tests give innacurate test summary indicating a misleading
passing test suite. Add new status for flaky tests.
Signed-off-by: Al Semjonovs <asemjonovs@google.com>
Flaky tests give innacurate test summary indicating a misleading
passing test suite. Add new status for flaky tests.
Signed-off-by: Al Semjonovs <asemjonovs@google.com>
When building with the native simulator instead of attempting to
call directly to the host libC, use the trampolines provided
by the runner.
In this way we can build this code even if we are building
Zephyr with an embedded C library.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
When writing a test suite, it's more common to want the 'before'
hook to run in privilaged mode, even when the test is run in userspace.
Reconfigure ztest to first run the test thread callback in privilaged
mode and only enter userspace after the test rule and suite's 'before'
functions ran.
Signed-off-by: Yuval Peress <peress@google.com>
Fix ztress_execute() out-of-bounds in case it is called with timer
and for maximum number of threads allowed (CONFIG_ZTRESS_MAX_THREADS).
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Corrected printed message on failing test with expected or received
NULL pointer in ztest_check_expected_data.
Signed-off-by: Stine Åkredalen <stine.akredalen@nordicsemi.no>
As both C and C++ standards require applications running under an OS to
return 'int', adapt that for Zephyr to align with those standard. This also
eliminates errors when building with clang when not using -ffreestanding,
and reduces the need for compiler flags to silence warnings for both clang
and gcc.
Most of these changes were automated using coccinelle with the following
script:
@@
@@
- void
+ int
main(...) {
...
- return;
+ return 0;
...
}
Approximately 40 files had to be edited by hand as coccinelle was unable to
fix them.
Signed-off-by: Keith Packard <keithp@keithp.com>
Fix checkpatch issue: UNNECESSARY_INT: Prefer 'unsigned long' over
'unsigned long int' as the int is unnecessary
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Many areas of Zephyr divide and round up without using the DIV_ROUND_UP
macro. Make use of it, so that we make use of a tested system macro and
at the same time we make code more readable.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
The init infrastructure, found in `init.h`, is currently used by:
- `SYS_INIT`: to call functions before `main`
- `DEVICE_*`: to initialize devices
They are all sorted according to an initialization level + a priority.
`SYS_INIT` calls are really orthogonal to devices, however, the required
function signature requires a `const struct device *dev` as a first
argument. The only reason for that is because the same init machinery is
used by devices, so we have something like:
```c
struct init_entry {
int (*init)(const struct device *dev);
/* only set by DEVICE_*, otherwise NULL */
const struct device *dev;
}
```
As a result, we end up with such weird/ugly pattern:
```c
static int my_init(const struct device *dev)
{
/* always NULL! add ARG_UNUSED to avoid compiler warning */
ARG_UNUSED(dev);
...
}
```
This is really a result of poor internals isolation. This patch proposes
a to make init entries more flexible so that they can accept sytem
initialization calls like this:
```c
static int my_init(void)
{
...
}
```
This is achieved using a union:
```c
union init_function {
/* for SYS_INIT, used when init_entry.dev == NULL */
int (*sys)(void);
/* for DEVICE*, used when init_entry.dev != NULL */
int (*dev)(const struct device *dev);
};
struct init_entry {
/* stores init function (either for SYS_INIT or DEVICE*)
union init_function init_fn;
/* stores device pointer for DEVICE*, NULL for SYS_INIT. Allows
* to know which union entry to call.
*/
const struct device *dev;
}
```
This solution **does not increase ROM usage**, and allows to offer clean
public APIs for both SYS_INIT and DEVICE*. Note that however, init
machinery keeps a coupling with devices.
**NOTE**: This is a breaking change! All `SYS_INIT` functions will need
to be converted to the new signature. See the script offered in the
following commit.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
init: convert SYS_INIT functions to the new signature
Conversion scripted using scripts/utils/migrate_sys_init.py.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
manifest: update projects for SYS_INIT changes
Update modules with updated SYS_INIT calls:
- hal_ti
- lvgl
- sof
- TraceRecorderSource
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
tests: devicetree: devices: adjust test
Adjust test according to the recently introduced SYS_INIT
infrastructure.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
tests: kernel: threads: adjust SYS_INIT call
Adjust to the new signature: int (*init_fn)(void);
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Add the zexpect API, inspired by GoogleTest's EXPECT API. This API reports
test failures while allowing test execution to continue. This enables test
reports to show more than a singule failing property on a failing test.
Signed-off-by: Aaron Massey <aaronmassey@google.com>
Test summary can add a lot of noise to the logs when debugging
a specific test using `-test` argument.
Add control to turn this off.
Signed-off-by: Al Semjonovs <asemjonovs@google.com>
We get compiler warnings on testcases that set CONFIG_MP_NUM_CPUS=1
and platforms that have CONFIG_SMP=y. Qualify the code so its only
built if CONFIG_SMP && (CONFIG_MP_NUM_CPUS > 1).
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Move the function in the `subsys/testsuite/ztest/src/ztest_mock.c` files.
This is motivated by the fact that there is others re-implementation of
`*printk` functions using libc counterparts in the `ztest_mock.c` file.
Signed-off-by: Théo Battrel <theo.battrel@nordicsemi.no>
For test_1cpu_start/test_1cpu_stop make the code only build if
CONFIG_SMP and move to using arch_num_cpus() for runtime loops
and CONFIG_MP_MAX_NUM_CPUS for array decleration.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
3000 milliseconds may not always be enough time for all 1cpu type
tests to finish on all platforms. Making the CPU hold time
configurable allows for additional flexibility.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Change for loops of the form:
for (i = 0; i < CONFIG_MP_NUM_CPUS; i++)
...
to
unsigned int num_cpus = arch_num_cpus();
for (i = 0; i < num_cpus; i++)
...
We do the call outside of the for loop so that it only happens once,
rather than on every iteration.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Adding the new Kconfig (enabled by default) to make a failed assumption
mark the final result as failed. This change has the following benefits
which have been asked for by the Zephyr community:
1. A failed assumption does not go silent. In this example, the failed
assumption will still mark the test as skipped, but the final result
will be to mark the full test run as failed. This would allow
blocking the CI when an assume fails.
2. Normal test skipping via the ztest_test_skip() is unaffected by this
change. Those tests will be marked as skipped, but the binary will
still pass.
Signed-off-by: Yuval Peress <peress@google.com>
Some testsuites dump lots of output very fast where some systems are not
able to capture the complete output from the tests. Add a slight delay
between each test in the suite.
New kconfig CONFIG_ZTEST_TEST_DELAY_MS is added to ztest.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When CONFIG_ZTEST_SHUFFLE is enabled and ASSERTS are disabled
`start_pos` becomes an unused variable leading to a compile error.
Cleaned-up shuffling algorithm to not need a `start_pos` check.
Signed-off-by: Al Semjonovs <asemjonovs@google.com>
Added a workaround to call random generator once during the
initialization. It is done to handle XOSHIRO generator limitation
which performs initialization in the first sys_rand32_get call.
And for some entropy generators it cannot be done from an interrupt
context and it happens if k_timer context is used which expires first.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
As of today <zephyr/zephyr.h> is 100% equivalent to <zephyr/kernel.h>.
This patch proposes to then include <zephyr/kernel.h> instead of
<zephyr/zephyr.h> since it is more clear that you are including the
Kernel APIs and (probably) nothing else. <zephyr/zephyr.h> sounds like a
catch-all header that may be confusing. Most applications need to
include a bunch of other things to compile, e.g. driver headers or
subsystem headers like BT, logging, etc.
The idea of a catch-all header in Zephyr is probably not feasible
anyway. Reason is that Zephyr is not a library, like it could be for
example `libpython`. Zephyr provides many utilities nowadays: a kernel,
drivers, subsystems, etc and things will likely grow. A catch-all header
would be massive, difficult to keep up-to-date. It is also likely that
an application will only build a small subset. Note that subsystem-level
headers may use a catch-all approach to make things easier, though.
NOTE: This patch is **NOT** removing the header, just removing its usage
in-tree. I'd advocate for its deprecation (add a #warning on it), but I
understand many people will have concerns.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Current ztress design supports both timer context and thread
context. The timer context always uses the first element of
several global array variables like exec_cnt[] and backoff[].
But progress_timeout() is using the last element of exec_cnt[]
to calculate the progress of timer context. It is not right.
Signed-off-by: Ming Shao <ming.shao@intel.com>
Tests with KERNEL enabled perform their cleanup logic after the suite's
after and test rules are executed. Unittests should do the same.
Signed-off-by: Yuval Peress <peress@google.com>
Updates the implementation when KERNEL is available to safely bail on
the test when the test calls fail, skip, or pass during invalid test
phases. Print a detailed message, and skip all other tests. The test
run will be marked as failed.
Signed-off-by: Yuval Peress <peress@google.com>
Move the function used for printing the phase name up so its available
for both unittest and KERNEL mode of tests.
Signed-off-by: Yuval Peress <peress@google.com>
When not running in setup, before, or in the test. Calling fail, skip,
or pass is invalid and should be considered an error. Properly handle
these cases by printing a more detailed error message and bailing the
process.
Signed-off-by: Yuval Peress <peress@google.com>
When KERNEL was not defined (unittest), the call to init_testing was
used to set a longjump target using 'stack_fail'. When triggered,
this was actually causing a segfault, because longjmp is only valid
if going directly up the stack. Since init_testing returned, it was
no longer on the stack. Instead, that logic MUST be inlined.
Signed-off-by: Yuval Peress <peress@google.com>
Add duration stats at both suite level and unit test
level into test summary. The duration is at second
level. Since the new ztest fx can execute a test suite
for multiple times, the worst/longest test duration is
collected.
Note that even a skipped test can have a duration greater
than 0 because the skip operation itself is not free.
Signed-off-by: Ming Shao <ming.shao@intel.com>
Previously, unit test duration is collected within the
TC_START and Z_TC_END_RESULT macros. With existing tests,
the TC_START macro can be invoked by both the ztest fx
and the tests themselves. And the TC_START macro definition
went lengths to avoid the interference when it is invoked
within a unit test. This commit decouple the time collection
and the TC_STRAT/Z_TC_END_RESULT macros to fix this issue.
Now only the (old) ztest framework is responsible for the
test duration measure. The test duration stats of new ztest
fx is different from this btw.
Signed-off-by: Ming Shao <ming.shao@intel.com>
Add test summary after all test suites finish running.
The summary can be one-line or verbose, which is configured
with CONFIG_ZTEST_VERBOSE_SUMMARY. The one-line summary covers
overall suite stats. The verbose summary covers each test
function within the suite besides the one-line summary.
The new ztest output ultimately go through the printk. If
printk go through the logging subsystem, there may be log
messages dropped. And if log_panic is invoked, log messages
can be flushed in a mess. So several explicit log flush
are used when printing summary to ensure no content is lost
and content is in good shape.
Some macros are shared between old and new ztests. Such as
TC_START_PRINT and TC_END_PRINT. The are defined accordingly.
Signed-off-by: Ming Shao <ming.shao@intel.com>
Updates the ztest_test_fail() function to allow failures in setup.
When executed, a failed assert will fail every test in the suite owning
the setup function. This was verified by adding a suite which asserts
in the setup function and has a test that should pass. During
exeuction, ztest marks the test as failing.
In order to verify exection I also added 2 new APIs:
- ZTEST_EXPECT_FAIL(suite_name, test_name)
- ZTEST_EXPECT_SKIP(suite_name, test_name)
Signed-off-by: Yuval Peress <peress@google.com>
