Commit Graph

4089 Commits

Author SHA1 Message Date
Marc Herbert
4975c94845 west.cmake: make MIN_WEST_VERSION catch up with requirements-base.txt
Also add a comment in each file reminding to keep them the same.

Fixes 251f269e23 ("west: v0.14.0 is required now (and soon, v1.1")

Confusing error message before this commit:

```
-- Found west (found suitable version 0.13.1, minimum required is 0.7.1)
CMake Error at SOF/zephyr/cmake/modules/zephyr_module.cmake:77 (message):
  Traceback (most recent call last):

    File "SOF/zephyr/scripts/zephyr_module.py", line 733, in <module>
      main()
    File "SOF/zephyr/scripts/zephyr_module.py", line 678, in main
      west_projs = west_projects()
                   ^^^^^^^^^^^^^^^
    File "SOF/zephyr/scripts/zephyr_module.py", line 536, in west_projects
      from west.configuration import MalformedConfig

  ImportError: cannot import name 'MalformedConfig'
                      from 'west.configuration'
  (west/src/west/configuration.py)
```

Clearer error message after this commit:

```
CMake Error at SOF/zephyr/cmake/modules/west.cmake:68 (message):
  The detected west version, 0.13.1, is unsupported.

    The minimum supported version is 0.14.0.
    Please upgrade with:
        /usr/bin/python3.11 -m pip install --upgrade west
```

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2023-05-22 10:17:11 +02:00
Evgeniy Paltsev
15b46039b8 ARC: used 64bit MDB binary by default.
This significantly improve user experience as 32 bit mdb binary
require to install multiple libraries before it can be used on
modern linux distros.

Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
2023-05-22 10:16:28 +02:00
Martí Bolívar
251f269e23 west: v0.14.0 is required now (and soon, v1.1)
Commit ce2a7d9a1a
("scripts: zephyr_module: handle expected west errors")
introduced some better error handling that requires
west version v0.14.0 or later to work.

Bump the west version in requirements-base.txt accordingly.

Due to the way zephyr_module.py is handling imports, this API change
resulted in zephyr_module.py running on older versions of west
reacting as if west was not installed, instead of erroring out.
Fix that so users who are on older west will get a hard error.

(We're about to force everyone to move to west v1.1 as soon as I can
get that release done, but this hotfix should still be helpful in the
interim as well.)

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2023-05-19 13:17:05 -07:00
Filip Kokosinski
442a452903 scripts/ci/check_compliance: add GitDiffCheck compliance check
This commit adds a new `GitDiffCheck` compliance check that checks the
newly added commits with `git diff --check` and reports them back if an
error is found.

This check is needed for some files (e.g. Kconfig) as they are not checked
by `Kconfig` and `KconfigBasic` checks on every commit in a pull request.

Signed-off-by: Filip Kokosinski <fkokosinski@antmicro.com>
2023-05-19 14:52:49 +00:00
Martí Bolívar
ce2a7d9a1a scripts: zephyr_module: handle expected west errors
Errors can occur if the user's workspace is not set up properly.
Output a better error message instead of dumping stack in these cases.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2023-05-19 10:56:02 +02:00
Marc Herbert
6697c5aa0b west: sign: rimage: drop default -c option when the user provides one
sign.py has an internal and indirect way to compute a default -c signing
schema option and pass it to rimage. It is built by appending
`$platform.toml` to whatever `rimage/config/` location was
found. Defaults are very convenient but in this case this computed -c
option can conflict with an explicit -c option spelled out by the user.

Scan for any explicit -c coming directly from the user. If any found,
ignore the default we computed. This is what is already being done for -k.

The precedence across rimage parameters coming from different places is
too complicated. Not passing multiple -c options simplifies the logic a
little bit.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2023-05-18 12:40:30 -07:00
Anas Nashif
729a795aef twister: fix testcase parsing from binary
We have been clearning the list of testcases in every iteration over the
symbols ending up with only one testcase in the instance.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-05-18 07:05:15 -04:00
Torsten Rasmussen
49389b546a west: sign: Add Kconfig with application version
Adds a Kconfig option which controls the version of the application
to use when the image is signed using imgtool.

When an application VERSION file is present, the default value will be
identical to the application version, else it will be 0.0.0+0, but a
project may still decide another value, if it so prefers.

Signed-off-by: Jamie McCrae <jamie.mccrae@nordicsemi.no>
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
2023-05-17 13:56:58 +02:00
Martí Bolívar
0c29e07e30 devicetree: better DT_PROP_BY_IDX()/DT_FOREACH_PROP_ELEM() support
Support use of these macros with properties of type phandle and
string by allowing iterating over:

- a phandle as if it were a phandles of length 1, for convenience and
  consistency with our ability to take its length (and getting 1)

- the non-null characters in a string: we exclude the null for
  consistency with the return value of DT_PROP_LEN() on string
  properties, which, like strlen(), does not include the null

With this and a previous patch expanding the usage of DT_PROP_LEN(),
there is now a relationship between being able to take a property's
logical length with DT_PROP_LEN() and being able to iterate over its
logical elements with DT_FOREACH_PROP_ELEM(). Explain this in the
documentation.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2023-05-16 18:14:26 +02:00
Martí Bolívar
8aa83f6ae8 devicetree: support DT_PROP_LEN() on phandle and string
It will be convenient to treat these respectively as degenerate cases
of 'phandles' and 'string-array'. Add support for this and regression
tests. (There's nothing to do in the case of 'phandle' beyond
documenting the guarantee.)

For the record, the other DT_PROP_LEN() tests for each type are in:

  type            test case              property
  ------------    --------------------   ------------
  array           test_arrays            a
  string-array    test_path_props        compatible
  uint8-array     test_arrays            b
  phandles        test_phandles          phs
  phandle-array   test_phandles          pha-gpios
  phandle         test_phandles          ph

Update docstrings and fix some issues in them.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2023-05-16 18:14:26 +02:00
Martí Bolívar
52043691e9 scripts: gen_defines: add some missing comments
We generally try to have comments in this file that show the form of
each generated macro. This is particularly important in the
write_vanilla_props() function, since that is called on every node in
the tree and handles generic macros that are widely applicable.

Various generated macros have been added over time that don't have
the corresponding comments; add these now.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2023-05-16 18:14:26 +02:00
Gerard Marull-Paretas
21b9a1c823 twister: allow loading external configuration
Add a new option to Twister that allows to load alternative
configuration files, `--alt-config-root`. This new option takes an
arbitrary number of root folders where alternative configuration files
can be stored. Twister will check if a test configuration file exists in
any of the alternative test configuration folders. For example, given
`$test_root/tests/foo/testcase.yaml`, Twister will use
`$alt_config_root/tests/foo/testcase.yaml` if it exists.

This feature can be useful if an out-of-tree project needs to run
upstream tests in different configurations, potentially not available
upstream (e.g. an out-of-tree board, or Kconfig setting).

Note that overlaying has been discarded because we can't easily delete
fields from the original configuration file, something that in certain
cases could be required. Current approach may lead to some dupplication,
but guarantees full control of the test configuration.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-16 07:25:52 -04:00
Maciej Perkowski
164ad1065f twister: Raise error when duplicates found on default
Make twister to raise error by default whenever duplicated test ids
are found. Remove redundant option to list duplicates.

Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
2023-05-16 06:21:25 -04:00
Jordan Yates
59167e1888 scripts: dts: gen_defines: add ENUM_VAL_<val>_EXISTS define
Add a define of the form
`DT_N_<node-id>_P_<prop-id>_ENUM_VAL_<val>_EXISTS` for enumerated
devicetree properties. This enables the devicetree API to check whether
an enum is a given value directly, without resorting to error-prone
checks against the enum index.

Example generated defines (int and string):
	`#define DT_N_S_test_S_enum_4_P_val_ENUM_VAL_5_EXISTS 1`
	`#define DT_N_S_test_S_enum_6_P_val_ENUM_VAL_zero_EXISTS 1`

Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
2023-05-15 09:03:37 -07:00
Torsten Rasmussen
c03e1900ae scripts: adjust WEST_PYTHON to posix path
Adjust WEST_PYTHON to posix path to be consistent with other Python
scripts which passes paths to the Zephyr CMake build system.

Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
2023-05-15 08:13:40 -07:00
Flavio Ceolin
02ac343c18 security: hardening: Update general recommendations
Update security recommendations. That is not a
thorough list though.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2023-05-13 02:28:29 +09:00
Flavio Ceolin
8276658ac7 security: hardening: Update debug susbsys options
Update information for debug susbys options.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2023-05-13 02:28:29 +09:00
Flavio Ceolin
d74e77147b security: hardening: Update experimental features
Update all experimental features on Zephyr. Remove outadated and
added new ones.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2023-05-13 02:28:29 +09:00
Flavio Ceolin
a06083c3c4 security: hardening: Organize hardened file
Separate debug and experimental options in hardened.csv
file for easier maintenance.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2023-05-13 02:28:29 +09:00
Dmitrii Golovanov
7617046c67 twister: Add expected and detected test case names to debug log
Log expected and detected test case names for the test suite name
check debug.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2023-05-12 10:03:52 +02:00
Dmitrii Golovanov
8ee23dcdb8 twister: Don't override test name check with RunID check
Don't override a test case name check failed result by the following
RunID check failed result to report clearly about test case mismatch
situation as well as more specifically when the same test case appears
with unexpected RunID.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2023-05-12 10:03:52 +02:00
Dmitrii Golovanov
14ba39c50e twister: Fix suite-name-check bypass on flash error
Fix suite-name-check bypassed at DeviceHandler on flash errors.

If the flash timeout is smaller than needed, but test images are
actually flashed successfully and appropriate tests executed,
then results from the previous test case were taken because
the flash error status makes the suite-name-check bypassed.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2023-05-12 10:03:52 +02:00
Gerard Marull-Paretas
76b91c700d twister: fix handling of mixed str/list configurations
We can now have situations like this when it comes to lists in Twister
config:

```yaml
common:
  platform_exclude: foo
tests:
  my.test:
    platform_exclude: bar
```

```yaml
common:
  platform_exclude: foo
tests:
  my.test:
    platform_exclude:
      - bar
      - baz
```

```yaml
common:
  platform_exclude:
    - foo
    - bar
tests:
  my.test:
    platform_exclude: baz
```

```yaml
common:
  platform_exclude:
    - foo
    - bar
tests:
  my.test:
    platform_exclude:
      - baz
      - bazz
```

This patch fixes handling of cases (2), (3) and (4).

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-11 09:17:44 -04:00
Jamie McCrae
2c2d1c72db requirements: Update pyOCD requirement to 0.35.0
Updates the pyOCD requirement to 0.35.0 to support a new flasher
option of flashing and preventing reset of the device.

Signed-off-by: Jamie McCrae <jamie.mccrae@nordicsemi.no>
2023-05-10 10:15:52 +02:00
Gerard Marull-Paretas
a91620f5bb twister: deprecate space-separated lists
Deprecate in favor of native YAML lists.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-10 09:52:37 +02:00
Gerard Marull-Paretas
47c2feca20 scripts: utils: add twister_to_list tool
Add a new tool that convers string-based lists on Twister config files
to native YAML lists.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-10 09:52:37 +02:00
Gerard Marull-Paretas
11218baffc twister: set 'any' type for set/list fields
This allows to write both, lists or strings in the YAML file.
Unfortunately pykwalify schema language does not support multi-type
fields. JSON schema or Marshmallow could be used instead.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-10 09:52:37 +02:00
Gerard Marull-Paretas
0db74702c3 twister: handle mixed common/tests types
This patch handles mixed types between common/tests sections in the
configuration, e.g. str+list, list+str, str+str.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-10 09:52:37 +02:00
Gerard Marull-Paretas
d3b8b8d9db twister: improve diagnostics for 'any' set/list types
Since pykwalify does not support multi-type fields, we are forced to use
the 'any' type. This is used for fields that allow sets/lists written as
either YAML lists or space-separated strings. This patch checks for
str/list Python type, and raises a configuration error if not true so
that user obtains better error messages in case the configuration file
contains invalid format.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-10 09:52:37 +02:00
Gerard Marull-Paretas
560a72171e twister: allow list input for set config fields
Twister configuration files have a few fields defined with the "set"
type. The set needs to be specified as a space-separated list of
entries, for example:

```yaml
plaform_allow: boardx boardy
```

This patch allows users to write these fields as YAML lists, for
example:

```yaml
platform_allow:
  - boardx
  - boardy
```

The list gets converted to a Python set, therfore the same exact
behavior is kept in Twister.

In some cases like in the example above, using lists allows to have more
readable configuration files, specially if the list is long. It also
helps on diffs, provided we have one item per line.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-10 09:52:37 +02:00
Maciej Perkowski
3bf7f83f21 twister: Bring back scope selection rule using platform_allow
A change to the scope selection rules was introduced by #52715
but it doesn't comply with the description in the code. What's more,
the change introduces major issues i.e. non deterministic scope
selection, especially in a CI environment. More context at #57595

Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
2023-05-08 11:18:44 +02:00
Kumar Gala
bae0a5b8b6 logging: Use TYPE_SECTION macros for log const
Clean up log_const to utilize macros for handling sections.

Update database_gen.py to match naming convention change.

Signed-off-by: Kumar Gala <kumar.gala@intel.com>
2023-05-03 10:43:31 +02:00
Gerard Marull-Paretas
baeee77ba6 twister: automatically add module board roots
Twister does not handle out-of-tree boards automatically. Modules like
e.g. example-application that define their own board root folder, are
required to inform twister via `--board-root`. In contrast, west is able
to to this automatically. This patch makes twister a bit smarter by
automatically filling the board root list with module paths as well.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-02 13:25:12 -04:00
Piotr Kosycarz
7892f40a52 twister: add options to shuffle tests across subsets
Add option to shuffle tests (randomly) before they are split into subsets.
This allow to fairly distribute tests across subsets,
so that long tests are not crowded in a single group.

This also allows to detect unwanted dependencies in test execution order.

Also, added option to provide custom seed to random generator
used to shuffle tests.
This will allow to reproduce certain test order if needed.
Used seed is printed at console.

Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
2023-05-01 06:12:59 -04:00
Rihards Skuja
9a6f7efd00 scripts: fix 'compatible' property of generated thermistor tables
The generated 'compatible' property doesn't match the associated devicetree
binding.

Signed-off-by: Rihards Skuja <rihards.s@origin-robotics.com>
2023-04-27 17:38:31 +09:00
Carles Cufi
0ebc37a972 scripts: runners: nrf: Fix the formatting of the old --nrf-family
The now legacy --nrf-family parameter takes the "raw" family name as
nrfjprog accepts it. But now we use an intermediate representation with
{FAMILY}_FAMILY as the format, so covert to it as required.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2023-04-27 09:50:48 +02:00
Aurelien Jarno
bdb3fd0ae9 scripts: gen_isr_tables: fix pylint issues
pylint reports the following issues on the
scripts/build/gen_isr_tables.py:

 C0325:Unnecessary parens after '=' keyword (superfluous-parens)
File:scripts/build/gen_isr_tables.py
Line:16
Column:0
 C0325:Unnecessary parens after '=' keyword (superfluous-parens)
File:scripts/build/gen_isr_tables.py
Line:316
Column:0

Fix that so that unrelated PR touching the same file have a chance to
pass the compliance checks.

Signed-off-by: Aurelien Jarno <aurelien@aurel32.net>
2023-04-26 12:58:13 +02:00
Gerard Marull-Paretas
a01d40f604 scripts: utils: migrate_sys_init: handle empty line after ARG_UNUSED
The script left empty lines below ARG_UNUSED. After this patch,

```c
static int f(const struct device *dev)
{
	ARG_UNUSED(dev);

	/* code */
	...
}
```

will become:

```c
static int f(void)
{
	/* code */
	...
}
```

instead of:

```c
static int f(void)
{

	/* code */
	...
}
```

Fixes #56954

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-04-25 19:57:00 +02:00
Gerard Marull-Paretas
2dbbfaec6b scripts: utils: migrate_sys_init: remove dead ZEPHYR_BASE
ZEPHYR_BASE is not used, so delete it.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-04-25 19:57:00 +02:00
Roman Dobrodii
2e0820f08c scripts: west: add silabs_commander runner
- Runs Silabs Commanders which is useful for SiLabs boards that
can not use J-Link for various reasons.
- Flash support only, no debug.

Signed-off-by: Roman Dobrodii <rdobrodii@antmicro.com>
2023-04-25 10:34:16 -07:00
Stephanos Ioannidis
428f4a6466 scripts: get_maintainer: Fix glob pattern directory match error logic
The `get_maintainer` script currently generates an error when a non-
slash-suffixed glob pattern matches any directories -- this is
incorrect because the glob pattern may be used to match files.

This commit updates the script to only generate an error when a
non-slash suffixed _only_ matches directories.

Signed-off-by: Stephanos Ioannidis <stephanos.ioannidis@nordicsemi.no>
2023-04-26 00:29:25 +09:00
Marc Herbert
16880cc381 scripts: gen_handles: silence "should it be static?" warnings
Some compilers and static analyzers warn when non-static (globals) are
not declared in some .h file. Tweak the generated code and add an
`extern` declaration before each device handle definition to make it
look it was declared in a (non-existent) .h file.

This makes the code a bit longer but it is already generated and very
repetitive anyway.

In the typical SOF configuration this silences about 150 sparse
warnings; as many as device handles.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2023-04-25 12:19:33 +02:00
Georgij Cernysiov
d641be7fc8 runners: stm32cubeprogrammer: fix program files path for win x64
The '%ProgramFiles%' is not guaranteed to be 'C:\Program Files' on
Windows x64.

The value will be 'C:\Program Files (x86)' if the west is executed by
32-bit Python.

To correct the issue '%ProgramW6432%' must be used.

Signed-off-by: Georgij Cernysiov <geo.cgv@gmail.com>
2023-04-24 13:35:26 +02:00
Jamie McCrae
96a9b7e1ff scripts: ci: check_compliance: Ignore mcuboot Kconfigs
Adds mcuboot Kconfig options to the ignore list as these are
defined in mcuboot but used in documentation and samples which
utilise sysbuild to build mcuboot.

Signed-off-by: Jamie McCrae <jamie.mccrae@nordicsemi.no>
2023-04-24 13:27:53 +02:00
Jamie McCrae
71dfaff583 scripts: utils: migrate_mcumgr_kconfigs: Remove IMG dummy header
Removes converting the dummpy IMG MGMT group Kconfig as this has
been removed from the tree and users would not have used it
anyway.

Signed-off-by: Jamie McCrae <jamie.mccrae@nordicsemi.no>
2023-04-21 16:33:30 +02:00
Carlo Caione
de90dfccbc syscall: Introduce __syscall_always_inline
Sometimes we want to force the inlining of a __syscall. Introduce a new
__syscall_always_inline symbol to do that.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2023-04-20 14:56:09 -04:00
Kumar Gala
47cf00058e fb: cfb: Use TYPE_SECTION macros for cfb_fonts
Clean up cfb_fonts to utilize TYPE_SECTION macros for handling sections.

Signed-off-by: Kumar Gala <kumar.gala@intel.com>
2023-04-19 17:14:54 +02:00
Kumar Gala
5b5fbc4c11 shell: remove dead linker section for shell
The initshell sections in the linker scripts where associated with theo
old shell code.  The old shell code has been removed for some time so
remove references to initshell in the linker scripts and size_calc.py

Signed-off-by: Kumar Gala <kumar.gala@intel.com>
2023-04-19 17:14:37 +02:00
Andrei Emeltchenko
29dc4b68ec twister: Cleanup redundant if / else
Convert following construction:

elif self.options.device_serial or self.options.device_serial_pty:
 if self.options.device_serial:
   ...
 else
   ...

to the construction:

elif self.options.device_serial:
  ...
else self.options.device_serial_pty:
  ...

Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
2023-04-19 03:27:08 -04:00
Andrei Emeltchenko
186d706c73 twister: Use flash options also for device_serial_pty
Use flash_timeout and flash_with_test options not only for
serial console but also for device_serial_pty.

Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
2023-04-19 03:27:08 -04:00