Add an assume API which works like JUnit's. Assumptions can be made
at any point before your test returns (setup, before, and during the
test). If an assumption fails, the test will be marked as skipped.
This helps avoid a cascading affect of failed tests when a base
feature breaks. The feature is expected to have tests and the tests
which depend on it should be skipped (not failed) if that feature
is broken.
Issue #42472
Signed-off-by: Yuval Peress <peress@google.com>
Fix the deprecated calls to loading ZephyrUnittest by replacing them
with `Zephyr COMPONENTS unittest`.
Signed-off-by: Yuval Peress <peress@google.com>
Since output can be cumstomized, so slightly change the output format,
so that twister can parse the output and collect the passrate correctly.
Signed-off-by: Guo Lixin <lixinx.guo@intel.com>
Adds few missing zephyr/ prefixes to leftover #include statements that
either got added recently or were using double quote format.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Update the new API to use K_USER as the flags for both
CONFIG_USERSPACE and CONFIG_TEST_USERSPACE. Also, fix the linker
script to properly include the suites, tests, and rules.
Fixes#44108
Signed-off-by: Yuval Peress <peress@google.com>
1.Remove filter to allow platforms(e.g. qemu_x86) supporting
CONFIG_ARCH_HAS_USERSPACE run testcases when userspace is disabled.
2.Define all testcases in the yaml for consistency and issue parsing
them during setup.
Signed-off-by: Guo Lixin <lixinx.guo@intel.com>
Instead of relying on runtime filter to limit scope to emulation
platforms, use the type attribute for each platform and do the filtering
very early on. This will speed things up for tests where we only run on
emulation platforms.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Enable ZTEST_DO_THE_SHUFFLE to shuffle the order tests are ran.
Additional configs ZTEST_DO_THE_SHUFFLE_SUITE_REPEAT_COUNT
ZTEST_DO_THE_SHUFFLE_TEST_REPEAT_COUNT specify the number of times
the test or suite is executed.
Signed-off-by: Al Semjonovs <asemjonovs@google.com>
In order to bring consistency in-tree, migrate all tests to the new
prefix <zephyr/...>. Note that the conversion has been scripted, refer
to #45388 for more details.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Some names of the test cases are duplicated within the project.
This commit contains the proposed names of the test scenarios.
Signed-off-by: Katarzyna Giadla <katarzyna.giadla@nordicsemi.no>
The filter in the testcase.yaml made it impossible for these to run.
Also, the Kconfig names for the options didn't really make much sense
so they were renamed to avoid conflict.
This fixes an issue found by #36433.
Signed-off-by: Yuval Peress <peress@google.com>
Test were executed on single CPU only and with qemu_cortex_a9
excluded. Removing those limitations after fixes.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Seen failures on some platforms. No harm to relax the
check for test timeout.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Change the CPU_CORTEX_R kconfig option to CPU_AARCH32_CORTEX_R to
distinguish the armv7 version from the armv8 version of Cortex-R.
Signed-off-by: Bradley Bolen <bbolen@lexmark.com>
1. Test suites in prior ztest serve no purpose other than logical
ordering of tests into a named-group. Move the construct of setup and
teardown into the test suite and away from individual tests.
Additionally, add the constructs of before/after to the test suites.
This model more closely resembels other testing frameworks such as gTest
and Junit.
2. Test can be added to a suite by using ZTEST() or ZTEST_F() where _F
stands for fixture. In the case where _F is used, the argument `this`
will be provided with the type `struct suite_name##_fixture*`. Again,
this models other modern testing frameworks and allows the test to
directly access the already set up data related to the test suite.
3. Add the concept of test rules (from Junit). Rules are similar to the
before/after functions of the test suites but are global and run on all
suites. An example of a test rule can be to check that nothing was
logged to ERROR. The rule can cause the test to fail if anything was
logged to ERROR during an integration test. Another example would be a
rule that verifies that tests ran within some defined timeout.
Signed-off-by: Yuval Peress <peress@google.com>
Because RISC-V arch (M extension) doesn't trigger exception for
division-by-zero, this test can't support RISC-V and is disabled in
RISC-V.
Signed-off-by: Jim Shu <cwshu@andestech.com>
Introduce a weak implementation of test_main() which calls:
* ztest_run_registered_test_suites(NULL);
* ztest_verify_all_registered_test_suites_ran();
This will attempt to run all registered test suites and verify that
they each ran.
Signed-off-by: Yuval Peress <peress@chromium.org>
Add new functionality to ztest to improve test modularity. The two
primary new entry points are:
* ztest_register_test_suite
* ztest_run_registered_test_suites
When registering a new test suite, users provide the name as well as
an optional predicate used to filter the tests for each run. Using NULL
as the predicate ensures that the test is run exactly once (after which
it is automatically filtered from future runs).
Calls to ztest_run_registered_test_suites take a state pointer as an
argument. This allows the the pragma functions to decide whether the
test should be run.
The biggest benefit of this system (other than the ability to filter
tests and maintain a larger test state) is the ability to better
modularize the test source code. Instead of all the various tests
having to coordinate and the main function having to know which tests
to run, each source file manages registering its own test
suite and handling the conditions for running the suite.
Signed-off-by: Yuval Peress <peress@chromium.org>
Issue #38643
Introduce a more powerful and well established mocking framework
into Zephyr. It also allows running the actual FFF tests using the
zephyr SDK and ztest framework to ensure compatibility.
As per TSC meeting, the fff.h header was directly added to
subsys/testsuite/include/. As per the guidelines, the file is exactly
the same as it is in FFF's library, but re-styled with clang-format.
The west.yml entry was added using the "ci" group and filtered by
default. (note that the tests will break until the CI actually
specifies that the group is needed).
Signed-off-by: Yuval Peress <peress@google.com>
Fix the trigger_fault_access. The fix is similar to cortex m.
Fix the trigger_fault_divide_zero. The fix is similar to qemu, i.e.
skip this testcase, because FVP would not trigger the exception for
dividing zero either.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Move to CMake 3.20.0.
At the Toolchain WG it was decided to move to CMake 3.20.0.
The main reason for increasing CMake version is better toolchain
support.
Better toolchain support is added in the following CMake versions:
- armclang, CMake 3.15
- Intel oneAPI, CMake 3.20
- IAR, CMake 3.15 and 3.20
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The Cortex-R platforms may no use XIP which means userspace code with
have read and execute access to some of the RAM so that it can execute
code. However, userspace will not have access the the protected
read/write parts of RAM so it is safe to use the Cortex-M version of
trigger_fault_access for this test.
Signed-off-by: Bradley Bolen <bbolen@lexmark.com>
On the Cortex M0+, M0, M23 the div-by-zero is not mapped
as an hardware exception but triggers a HardFault.
The test case is skipped for any ARMV6_M_ARMV8_M_BASELINE.
Signed-off-by: Francois Ramu <francois.ramu@st.com>
This compile test should be checking if a symbol has been defined,
otherwise it is using the kconfig value directly. This fixes a warning
../src/main.c:115:37: warning: "CONFIG_SOC_QEMU_ARC" is not defined,
evaluates to 0 [-Wundef]
when using the -Wundef flag.
Signed-off-by: Bradley Bolen <bbolen@lexmark.com>
qemu arc has limited support for DIV/REM, no DIVZERO exception
throw, so we need to skip trigger_fault_divide_zero test case.
Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
Put the testcase test_catch_assert_in_isr() to execute last, to prevent
it affects other test cases. Because when we caught an assert failure
in the ISR handler, it cannot be guaranteed that all the current
program status would be recovered.
Fixes#34844.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
Make the divide instruction not being optimized of the ztest example
test case. Do this because it will be replaced by invalid opcode
exception instruction generated by GCC.
Fixes#33403.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
em_starterkit has ICCM at 0x0 address, access to 0x0
address doesn't generate any exception, so we access
to 0xFFFFFFFF address instead to trigger exception.
Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
When running the test on Cortex-M platforms, use access to
_current to trigger a memory access fault, as this address
is guaranteed to be in kernel ram and will not trigger
SecureFault exceptions.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
This functions is being called across the tree, no reason why it should
not be a public API.
The current usage violates a few MISRA rules.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
