Commit Graph

3833 Commits

Author SHA1 Message Date
Carles Cufi
1951071f14 requirements: Add lxml as a requirement
junitparser has a bug that prevents it from working correctly when lxml
is not installed.

See this GH issue for more information:
https://github.com/weiwei/junitparser/issues/99

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-11-13 06:25:59 -05:00
Carles Cufi
1ee0d60990 scripts: compliance: Parse the pylint output
Parse the output generated by pylint and store it as FmtdFailure
instances, so that they can be annotated later.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-11-13 06:25:59 -05:00
Carles Cufi
edd2c83dd4 scripts: compliance: Modernize pylint invocation
The current recommended API call for subprocess instantiation is run(),
so replace the low-level Popen and communicate() with it.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-11-13 06:25:59 -05:00
Carles Cufi
8275d21a6c scripts: compliance: Add GitHub-compatible annotation support
Add support for GitHub workflow annotations, so that errors are
displayed inline with the Pull Request diff.

More info here:
https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#about-workflow-commands

Enable the new option in the compliance GitHub action.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-11-13 06:25:59 -05:00
Carles Cufi
89c3885c28 scripts: compliance: Add a new formatted failure method and class
Format the JUnit failure according to the standard:
https://www.ibm.com/docs/en/developer-for-zos/14.2?topic=formats-junit-xml-format

This also includes a new FmtdFailure class, which will be used in
subsequent commits to implement GitHub-compatible annotations. A list of
FmtdFailure instances is kept in the ComplianceTest instance because
the TestCase instance cannot restore classes derived from Result that
the junitparser module doesn't know about.

Use the new function to format checkpatch errors.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-11-13 06:25:59 -05:00
Carles Cufi
e9415cf929 scripts: compliance: Rename add_failure to failure
For consistency with error() and skip(), rename add_failure() to
failure().

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-11-13 06:25:59 -05:00
Emil Gydesen
08c5a826e9 footprint: Update conf files for nrf5340_cpunet rpmsg
Update the configuration files for the footprint
tracking for the nrf5340dk_nrf5340_cpunet iso-broadcast
and iso-receive, after the files were recently renamed.

Signed-off-by: Emil Gydesen <emil.gydesen@nordicsemi.no>
2022-11-11 08:46:46 +00:00
Martí Bolívar
b907f74326 runners: trace32: mypy fixes
Fixes for the following issues that a new version of mypy started
reporting recently:

runners/trace32.py:34: error: Incompatible default for argument
"startup_args" (default has type "None", argument has type
"List[str]") [assignment]

runners/trace32.py:137: error: Incompatible default for argument
"cfg" (default has type "None", argument has type "Path") [assignment]

runners/trace32.py:34: note: PEP 484 prohibits implicit Optional.
Accordingly, mypy has changed its default to no_implicit_optional=True

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2022-11-09 11:10:09 -05:00
Mariusz Skamra
8bbd2e1488 footprint: Add unicast_audio_{server, client} samples to tracking
Add a Bluetooth unicast audio samples to track their footprint.

Signed-off-by: Mariusz Skamra <mariusz.skamra@codecoup.pl>
2022-11-09 09:40:01 +00:00
Carles Cufi
2e66fac6d3 scripts: compliance: Fix inheritance of KconfigBasicCheck
KconfigBasic should not be inheriting from both KconfigCheck and
ComplianceTest. This was a workaround to find all inheritors of
ComplianceTest, so fix the inheritors search instead.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-11-08 07:20:21 -05:00
Carles Cufi
aed9622a1e scripts: compliance: Allow for custom message and type
Instead of arbitrarily storing the string provided by the caller in
either the message attribute or the element text, let the reporter
choose instead.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-11-08 07:20:21 -05:00
Carles Cufi
72fde1b9b3 scripts: compliance: Generate one Result per issue encountered
Instead of accumulating failure information in a single Result instance,
located at index 0, generate one Result sub-class instance per
failure/error/skip as it is intended in the original library.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-11-08 07:20:21 -05:00
Carles Cufi
5ae70b9b77 scripts: compliance: Use standard TestCase class
No need to define a custom class anymore.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-11-08 07:20:21 -05:00
Carles Cufi
f090efa793 scripts: compliance: Remove unused info message handling
No need to keep this in the tree if it's not used at all.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-11-08 07:20:21 -05:00
Carles Cufi
c3933561b0 scripts: compliance: Use text setter/getters
The junitparser Python module now comes with a setter and getter for
Result.text, use them instead of peeking into the underlying XML.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-11-08 07:20:21 -05:00
Tom Burdick
e3d877f811 rtio: Userspace support
Add support for userspace with RTIO by making rtio and rtio_iodev
k_objects. As well as adding three syscalls for copying in submissions,
copying out completions, and starting tasks with submit.

For the small devices Zephyr typically runs on one of the most important
attributes tends to be low memory usage. To maintain the low footprint of
RTIO and its current executor implementations the rings are not shared with
userspace. Sharing the rings it turns out would require copying submissions
before working with them to avoid TOCTOU issues.

The API could still support shared rings in the future so that a
kernel thread could directly poll, copy, verify, and start the submitted
work. This would require a third executor implementation that maintains its
own copy of submissions similiar to how io_uring in Linux works.

Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2022-11-08 10:44:03 +01:00
Anas Nashif
e2cd634185 ci: also upload results of twiser a week full builds
Upload results of full runs we do twice a week to opensearch.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-11-07 14:20:02 -06:00
Martí Bolívar
a97295b6a2 edtlib: re-appease the linter
At some point in the past, we had to suppress a couple of false
positive pylint warnings to pass CI. But now the linter seems to have
figured out its original mistake and is complaining about a useless
supression. Sigh. Play along.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2022-11-07 19:00:31 +01:00
Martí Bolívar
15e3e317f7 dtlib: implement copy.deepcopy() for DT
The standard library copy module allows you to implement shallow and
deep copies of objects. See its documentation for more details on
these terms.

Implementing copy.deepcopy() support for DT objects will allow us to
"clone" devicetree objects in other classes. This in turn will enable
new features, such as native system devicetree support, within the
python-devicetree.

It is also a pure feature extension which can't harm anything and is
therefore safe to merge now, even if system devicetree is never
adopted in Zephyr.

Note that we are making use of the move from OrderedDict to regular
dict to make this implementation more convenient.

See https://github.com/devicetree-org/lopper/ for more information on
system devicetree. We want to add system devicetree support to dtlib
because it seems to be a useful way to model modern, heterogeneous
SoCs than traditional devicetree, which can really only model a single
CPU "cluster" within such an SoC.

In order to create 'regular' devicetrees from a system devicetree, we
will want a programming interface that does the following:

   1. parse the system devicetree
   2. receive the desired transformations on it
   3. perform the desired transformations to make
      a 'regular' devicetree

Step 3 can be done as a destructive modification on an object-oriented
representation of a system devicetree, and that's the approach we will
take in python-devicetree. It will therefore be convenient to have an
efficient deepcopy implementation to be able to preserve the original
system devicetree and the derived regular devicetree in memory in the
same python process.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2022-11-07 19:00:31 +01:00
Martí Bolívar
3c976cc3b6 devicetree: stop using OrderedDict
Regular dicts are insertion-ordered since CPython 3.6 and Python 3.7.
Zephyr now requires Python 3.8, so it should be OK to replace
OrderedDict with regular dict now. This results in less typing and
more readable object representations.

A nitpicker could argue that this is a functional change, since if a
user is doing 'assert isinstance(node.props, OrderedDict)', that will
fail now, but:

 1. nobody is doing something like that in the zephyr tree
 2. that would be a silly thing to do
 3. we don't currently make any API stability guarantees
    for this module right now anyway

so it should be fine.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2022-11-07 19:00:31 +01:00
Martí Bolívar
c25dde3511 dtlib: fix Property.labels docstring
The value of the 'labels' attribute is a list, not a set.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2022-11-07 19:00:31 +01:00
Martí Bolívar
b64d5d6d19 dtlib: move some internal code around
Refactor the file parsing methods for readability by moving the
_parse_header() and _parse_memreserves() calls from _parse_dt() to
_parse_file(). The header and memreserves are not part of the 'tree'
part of the devicetree; now that we have a dedicated _parse_file()
helper, it makes more sense to me to have them there.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2022-11-07 19:00:31 +01:00
Martí Bolívar
c875cb0f94 dtlib: fix Property.type type annotation
Holy overloaded technical terms, Batman.

Here, 'property' and 'type' each mean two different things, which
we can distinguish like this:

- Property (capital P): dtlib.Property class, represents
  a property in a devicetree node
- @property: a Python property
- type(): an "@property" in the Property class, that returns
  a dtlib.Type value
- Type (capital T): dtlib.Type class, represents the devicetree
  type of a Property value (dtlib.Type.BYTES, etc.)

The type() @property in the Property class currently has an 'int' as
its Python return type annotation. It really returns a dtlib.Type,
which is an int (since it's an IntEnum), but that's not the same thing
as an int.

Change this to Type to be clear that not just any int can be returned
by this @property.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2022-11-07 19:00:31 +01:00
Martí Bolívar
cd18ffee22 dtlib: clean up Property initialization
Make attribute initialization order match the order that attributes
appear within the class docstring. Move the 'type' property definition
up by the constructor to make it more obvious that this 'attribute' is
a (Python) property. This is for readability.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2022-11-07 19:00:31 +01:00
Martí Bolívar
47789aa1fa dtlib: clean up Node initialization
Reorder attribute initialization to match the order that attributes
appear in the class level docstring. This is for readability.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2022-11-07 19:00:31 +01:00
Martí Bolívar
cb78a5c322 dtlib: clean up DT initialization
Initialize all the public API interface related attributes within the
constructor instead of scattering them throughout the implementation
of the class, and make sure they all have type annotations.

Move all the parsing code away from the init routines and public API
down to the main parsing block.

This is for readability and paves the way for later changes that
affect the way initialization happens.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2022-11-07 19:00:31 +01:00
Anas Nashif
8f5d0a77b0 twister: set both commit and run date in json report
Version is not enough, we need the run date but also the commit date, to
ease reporting and use in dashboards.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-11-06 22:27:07 -05:00
Anas Nashif
148a6e2ee1 twister: do not recorde a null version
In some scenarios we might end up with a null version value in the json
file, if the version can't be determined, set it to Unkown instead.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-11-06 22:27:07 -05:00
Ederson de Souza
b1a3ce4016 scripts/build: Add program header support for gen_relocate_app.py
Some linker scripts also associate memory regions with program headers
(PHDRS [1]). This patch adds support on gen_relocate_app.py to also
place a phdr alongside the memory region.

To keep things simple (and more natural), the format is basically the
same used in linker scripts - an space followed by ':<phdr_name>', like:

  SRAM2\ :phdr0:COPY:/home/xyz/zephyr/samples/hello_world/src/main.c

(Note the escape char before the space.)

[1] https://sourceware.org/binutils/docs/ld/PHDRS.html

Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
2022-11-03 10:25:07 +01:00
Anas Nashif
662aed81d2 actions: Upload test results to opensearch service
On push, upload test results to opensearch for analysis and reporting.
Goal is to use this data to understand test coverage better and use this
services for all test reporting, also for results coming from testing on
hardware.
Opensearch is currenly being used for evaluation, we are considering
the switch to elasticsearch later.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-11-02 10:26:43 -04:00
Anas Nashif
b89a3d955a twister: count retries after failures or errors
When twister is set to retry any failures, count the number of retries
and record the number in the json file. This will help us identify
unstable tests or tests requiring attention.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-10-31 12:33:28 -04:00
Filip Zajdel
4508681020 kconfig: wrap description in guiconfig
Modify description frame to wrap the text so the window
doesn't have to be scrolled horizontally.

Signed-off-by: Filip Zajdel <filip.zajdel@nordicsemi.no>
2022-10-31 23:23:46 +09:00
Carles Cufi
192d01eea1 checkpatch: Document both endianness macros
Both CONFIG_BIG_ENDIAN and CONFIG_LITTLE_ENDIAN can be used in the code,
so reflect that in the error message in checkpatch.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-10-28 22:06:17 +09:00
Carles Cufi
43ae4a7025 checkpatch: Disallow using __BYTE_ORDER__
The canonical way of determining endianness is now CONFIG_BIG_ENDIAN
instead in Zephyr.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-10-28 19:23:46 +09:00
Adrian Bonislawski
30e4a4b71e west: sign: remove -i 3 parameter
in some builds there is need to pass different value for -i parameter
and current code will override it to 3
Also in rimage i parameter is by default set to 3

Signed-off-by: Adrian Bonislawski <adrian.bonislawski@intel.com>
2022-10-27 11:02:35 +02:00
Daniel DeGrasse
b579a44e9c scripts: twister: update twister to support QEMU platforms with sysbuild
Update twister to support running QEMU platforms with sysbuild, by parsing
domains.yaml and executing the "run" target of the default application.
This will allow twister to test QEMU targets with sysbuild. It is assumed
that QEMU targets will add any external images they need in the build
phase

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
2022-10-26 08:48:37 -04:00
Daniel DeGrasse
6a5b88e523 scripts: west: move domains.py to a generic library folder
Since twister can make use of domains.py as well, refactor west's use
this file so that domains.py can be moved to a generic library folder.

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
2022-10-26 08:48:37 -04:00
Jeremy Bettis
a84c271e58 twister: Fix OUTPUT: blank lines after test end
When running twister -vv many blank lines would be logged.

Fix handing of empty lines from the reader thread. Real blank likes will
be b"\n" not b"".

Change open call to use with, and logger to use %s to fix pylint
warnings.

Signed-off-by: Jeremy Bettis <jbettis@google.com>
2022-10-25 09:28:15 -04:00
Markus Swarowsky
1fb23b6509 west: nrfjprog: Allow flashing UICR without erase
Currently, it is not possible to use a normal flash command to flash two
hex files that contain data in the UICR region, even though these
data don't conflict with each other.
However, that is not a technical limitation of the device, but of the
nrfjprog runner as it fails as soon as the hex file contained UICR data
and there was no --force/--erase/--recover used.
That approach was used in the past as program operation was used
without --verify, but as this option was added a while ago it is an
unnecessary limitation.

If the UICR wasn't erased before programming, the verify of the program
operation will now fail, so giving the user a warning in this case that
the hex files contain data in the UICR will still give the user a hint
of what might have caused the failing verify.
Additionally, it enables partial programming of the UICR without the
use of the --force option.

Signed-off-by: Markus Swarowsky <markus.swarowsky@nordicsemi.no>
2022-10-24 21:35:24 +02:00
Piotr Węgliński
08eb361846 ci: twisterlib: Modify the overlay regular expression
Modify the regex used to merge overlays specified for
test samples in twister tests.
Overlays without any prefixes will be merged and passed to cmake.
Overlays with prefixes won't be merged but passed to cmake as they are.

Signed-off-by: Piotr Węgliński <piotr.weglinski@nordicsemi.no>
2022-10-24 08:26:17 -04:00
Stephanos Ioannidis
faf4208295 scripts: set_assignee: Make review requests more deterministic
When there are more than 15 reviewers, the `set_assignee` script was
adding the reviewers more or less on a random basis because the input
set was arbitrarily ordered (thanks to how Python "set" works), and the
attempt to add any reviewers beyond the count of 15 results in the
previoulsy added reviewer being removed.

This commit updates the `set_assignee` script such that:

1. The collaborator list (input for generating the reviewer list) is
   ordered by the area match, such that the collaborators of the most
   relevant area come first.

2. The reviewers of the relevant areas are added first, until the total
   reviewer count is 15. The script does not attempt to add more than
   15 reviewers because that can result in the previously added
   reviewers being removed from the list.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2022-10-20 23:52:49 +09:00
Ming Shao
75914111d9 twister: fix the mis-placed drain logic.
The logic to drain the leftover logs of previous test should be
put before the monitor_serial(). It was misplaced when re-adding
this logic back.

Signed-off-by: Ming Shao <ming.shao@intel.com>
2022-10-20 15:42:25 +02:00
Torsten Rasmussen
36ea41fad3 samples: introduce 'with_mcuboot' sample to show sysbuild feature
This sample is intended to show users how they can create a sample which
has dedicated configuration files for extra images included in a build.

MCUboot is used as example on how its default can be adjusted and
MCUboot itself automatically be included in the build when using
sysbuild.

Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
2022-10-18 14:13:27 +02:00
Manuel Arguelles
31b3993d58 scripts: west: introduce Lauterbach TRACE32 runner
This patch introduces a West runner for flashing and debugging with
Lauterbach TRACE32 debuggers. The runner consists of a wrapper around
TRACE32 software, and allows a Zephyr board to execute a custom start-up
script (Practice Script) for the different commands supported, including
the ability to pass extra arguments from CMake. Is up to the board using
this runner to define the actions performed on each command.

The `debug` command launches TRACE32 GUI to allow debug the Zephyr
application, while the `flash` command hides the GUI and executes the
start-up script in a background process.

Signed-off-by: Manuel Arguelles <manuel.arguelles@nxp.com>
2022-10-14 09:51:14 +02:00
Anas Nashif
d8197f7535 ci: testplan: error out if we find mis-configured tests
If we find errors during test plan creation, stop execution and report
out the issues.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-10-13 06:23:46 -04:00
Anas Nashif
cc7d337ea7 twister: do not error on intentionally skipped tests
if we mark a test to be skipped, do not error in integration mode about
filter conflicts.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-10-13 06:23:46 -04:00
Anas Nashif
6a5af72d66 twister: count errors identified during evaluation of tests
Some tests report errors based on configuration, those need to be
captured.

Fixes #51202.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-10-13 06:23:46 -04:00
Chris Duf
e067be5f0c Devicetree: edtlib: fix possible TypeError in Binding.__repr__()
Calling Binding.__repr__() when the attribute Binding.path is None
would raise TypeError: expected str, bytes or os.PathLike object,
not  NoneType.

Known bindings that may not have a path (Binding.path is None)
include bindings for properties such as 'compatible', 'reg', 'status'.

Signed-off-by: Chris Duf <chris@openmarl.org>

Co-authored-by: Gerard Marull-Paretas <gerard@teslabs.com>
2022-10-13 12:02:31 +02:00
Chris Duf
230c80e000 Devicetree: edtlib: fix possible AttributeError in Property.description
Attempting to access the property Property.description
when Property.spec.description is None would raise
AttributeError: 'NoneType' object has no attribute 'strip'.

Known properties that may not have a description
(Property.spec.description is None):
- 'compatible' for nodes such as / /soc /soc/timer@e000e010 /leds /pwmleds
- 'reg'        for nodes such as /soc/timer@e000e010
- 'status'     for nodes such as /soc/timer@e000e010
- 'gpios'      for nodes such as /leds/led_0 /buttons/button_0
- 'pwms'       for nodes such as /pwmleds/pwm_led_0

This patch checks the PropertySpec.description attribute before calling
strip(): will return None, and not raise AttributeError.

Signed-off-by: Chris Duf <chris@openmarl.org>

Co-authored-by: Gerard Marull-Paretas <gerard@teslabs.com>
2022-10-13 16:03:49 +09:00
Christopher Friedt
ef142c117e scripts: net: add enumerate_http_status.py
Add a script to extract HTTP status values and format them
in a way that is both human readable and machine parseable.

Each line of output is of the form:

```
HTTP_{key}_{upper_val} = {key}, /**< val */
```

Signed-off-by: Christopher Friedt <cfriedt@meta.com>
2022-10-12 09:02:21 -04:00
Chris Duf
5abce9b8ec Devicetree: edtlib: fix possible KeyError in Binding.description
Attempting to access the Binding.description property
when the description is unavailable would raise KeyError: 'description'.

Known bindings that won't define a 'description' key in the
Binding.raw dictionary include the 'compatible' property's binding
of nodes such as /, /soc, /leds or /pwmleds.
Note that this may also occur when a proper YAML
binding file is available (e.g. pwmleds.yaml).

This patch simply substitutes the Binding.raw dictionary indexing
with the get() function: will return None and not raise KeyError.

Signed-off-by: Chris Duf <chris@openmarl.org>
2022-10-12 18:45:32 +09:00
Gerard Marull-Paretas
882162357c scripts: build: gen_handles: use Z_DECL_ALIGN
Instead of hardcoding alignment size for pass 2 device handles, use
Z_DECL_ALIGN. This makes sure gen_handles.py is always in sync with the
type defined in device.h. The build assert in device.h can be removed as
a result, since we do not hardcode handles size anywhere else.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2022-10-11 18:11:47 +02:00
Evgeniy Paltsev
d9a64b7b23 twister: fix UnicodeDecodeError on input handling in BinaryHandler
In BinaryHandler we process input from simulator with decode('utf-8')
to convert it to string. decode has strict error handling by default -
so it raises UnicodeDecodeError exception if it can't decode input
binary sequence.

So if test start to print some junk to uart console we get
UnicodeDecodeError exception which cause the whole twister crash.

To fix that switch decode to less strict error handling when it
just replace undecoded binary sequence with unicode replacement
character.

Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
2022-10-11 08:25:39 -04:00
Anas Nashif
b2a95831eb Revert "twister: temporarily disable mec15xx boards"
This reverts commit b3ead37efa.

This is no longer needed, pinmux was removed and deprecated.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-10-11 07:46:39 -04:00
Martí Bolívar
98bcc2d88e west: runners: blackmagicprobe: fix on windows
We need to give gdb slash-separated paths to ELF files on Windows.
Just make sure we always have slash-separated paths. We use this
technique in various places around the tree.

Fixes: #50789
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2022-10-10 09:54:14 +02:00
Erwan Gouriou
7b31b0e23c scripts: west_commands: openocd: Use 'reset init' by default
Instead of 'reset halt', default the command to reset and halt the target
to 'reset init'.

OpenOCD strictly requires 'reset init' to halt and prepare the device
before flash programming.
See https://openocd.org/doc/html/Flash-Commands.html chapter
"12.2 Preparing a Target before Flash Programming."

For targets implementing the init hook, it will allows to perform some
target specific init improvements (such as speed flash improvements).
Targets not implementing this hook will end up executing the usual and
previous 'halt'.

Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
2022-10-07 12:19:46 -07:00
Maureen Helm
e73c363346 scripts: edtlib: gen_defines: Add model name macros based on compat
Follow up to 5b5aa6ebba adding model name
and existence macros for all compatibles of a node that match an entry
in vendor prefixes.

Signed-off-by: Maureen Helm <maureen.helm@intel.com>
2022-10-07 11:48:02 -07:00
Emil Gydesen
4c7c36cae8 footprint: Remove bt_hap_ha for nrf52840dk
Remove the tracking of that sample as it does not build due
to missing BT ISO support.

Signed-off-by: Emil Gydesen <emil.gydesen@nordicsemi.no>
2022-10-07 10:09:25 +00:00
Carles Cufi
2da82552f5 west: completion: bash: Clean up, fix and add missing options
Although 8644b333cd added support for some
missing commands an options, the bash completion file was still in bad
need for some cleanup, fixes and additions.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-10-07 10:01:23 +02:00
Cong Nguyen Huu
4d41f54d08 twister: support generate hardware map on windows
On windows, some board manufacturers have name that is "Microsoft"
instead of real its name. Need add it to manufacturer list.
Product information is missing. we can assign it "unknown" and
then add missing informations  by hand.

Signed-off-by: Cong Nguyen Huu <cong.nguyenhuu@nxp.com>
2022-10-06 07:23:17 -04:00
Cong Nguyen Huu
5b2b5dfec0 twister: run tests on device with twister on windows
Currently, support for Twister on Windows is only build.
The only thing that needs to be ported is that select()
cannot wait on file descriptors on Windows. Therefore
the serial monitor function needs to be reworked to
support both OSes.

Signed-off-by: Cong Nguyen Huu <cong.nguyenhuu@nxp.com>
2022-10-06 07:23:17 -04:00
Daniel DeGrasse
602b8b485d scripts: twister: add support for sysbuild
Add support for building with sysbuild using twister, via the "sysbuild"
yaml property in testsuites. This will currently disable Kconfig and
devicetree filtering.

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
2022-10-04 14:06:58 -04:00
Fabio Baltieri
279ab43bdd scripts: set_assignees: move misplaced log entry
The log refers to Platform taking precedence but the loop handles
Documentation as well. Move it under the proper section, drop "Drivers"
as that is not handle in any special way.

Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
2022-10-04 15:14:27 +00:00
Fabio Baltieri
d06450b055 scripts: set_assignees: handle the documentation + unmaintained case
The script right now assumes that all the areas have a maintainer and
that the second area in particular has one in case of Documentation,
that results in a ValueError when it's not the case. Handle that by
checking the lists before using them.

Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
2022-10-04 15:14:27 +00:00
Gerard Marull-Paretas
fdea3c9a44 devicetree: add DT(_INST)_FOREACH_PROP_ELEM_SEP(_VARGS)
Add a new set of helpers for expanding property entries with a
separator. These macros complement DT(_INST)FOREACH_PROP_ELEM(_VARGS) by
adding the capability to expand with a custom separator between property
entries. This allows, in some cases, to re-use existing macros (e.g.
DT_PROP_BY_IDX) without creating an auxiliary macro that just appends a
separator. Example:

```dts
n: node {
	...
	my-gpios = <&gpioa 0 GPIO_ACTIVE_HIGH>,
		   <&gpiob 1 GPIO_ACTIVE_HIGH>;
};
```

Before:

```c
 #define GPIO_DT_SPEC_BY_IDX_AND_COMMA(node_id, prop, idx) \
	GPIO_DT_SPEC_BY_IDX(node_id, prop, idx),

struct gpio_dt_spec specs[] = {
	DT_FOREACH_PROP_ELEM(DT_NODELABEL(n), my_gpios,
			     GPIO_DT_SPEC_BY_IDX_AND_COMMA)
};
```

After:

```c
struct gpio_dt_spec specs[] = {
	DT_FOREACH_PROP_ELEM_SEP(DT_NODELABEL(n), my_gpios,
				 GPIO_DT_SPEC_BY_IDX, (,))
};
```

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2022-10-03 10:11:18 +02:00
Ming Shao
8b46675e2b twister: add comment about stats logic
Add comments and docstring to class  ExecutionCounter to explain
the twsiter stats logic.

Signed-off-by: Ming Shao <ming.shao@intel.com>
2022-10-01 06:42:54 -04:00
Ming Shao
94f2d2437f twister: refine the twister test plan generation
Due to the diverse coding styles and lack of preprocessing when
scanning for test cases, there were many unintended combinations
of yaml test scenarios and C test functions with the regex-based
test case discovery, which caused an inaccurate test plan and test
result stats.

As the new ztest fx is used, the test cases of a test instance can
be accurately determined via zephyr.symbols file.

Because the zephyr.symbols file is not available until after build,
test cases determination is placed right after the build stage of
the pipeline and before the runtime test collection.

For those test instances that don't go through the build stage,
such as those with "Skip filtered" (statically filtered) reason
before entering the pipeline, they are not affected.

This patch also adjust the stats logic because the zephyr.symbols
file is generated after build. So ExecutionCounter update is split
and some must be postponed until test execution pipeline is completed.

Some concepts:

A test instance = a yaml scenario + a platform
"Test instance" and "test config" are synonyms in twister, unfortunately
excessive IMHO...

A test instance can be filtered or skipped in 3 ways.
Let's define "runtime" as "after entering the execution pipeline".

1) statically filtered (before runtime)
   Such test instance is filtered by analyzing the yaml and never
   enters the execution pipeline.
2) cmake filtered (runtime)
   Such test instance enters pipeline and is filtered at cmake stage.
3) build skipped (also runtime)
   Such test instance enters pipeline and is skipped at build stage.

All the test instances that enter the execution pipeline will go
through the report stage, where ExecutionCounter is further updated.

The meaning of the fields of ExecutionCounter are:

 .skipped_configs = static filtered + cmake filtered + build skipped
 .skipped_runtime = cmake filtered + build skipped
 .skipped_filter = static filtered
 .done = instances that enter the execution pipeline
 .passed = instances that actually executed and passed

Definition of the overall stats:
total_complete = .done + .skipped_filter
total = yaml test scenario * applicable platforms
complete percentage = total_complete / total
pass rate = passed / (total - skipped_configs)

Signed-off-by: Ming Shao <ming.shao@intel.com>
2022-10-01 06:42:54 -04:00
Stephanos Ioannidis
b8643cb429 twister: size_calc: Remove stale k_mem_pool_area reference
This commit removes the stale reference to `k_mem_pool_area` for the
memory pool API, which was removed in the v2.5.0 release.

Signed-off-by: Stephanos Ioannidis <stephanos.ioannidis@nordicsemi.no>
2022-09-28 14:24:34 +00:00
Fabio Baltieri
d43e4e9ca4 scripts: dump_bugs_pickle: strip bug titles
Strip trailing and leading whitespaces from but titles so that they can
be used directly in the release notes.

Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
2022-09-27 16:51:45 -05:00
Fabio Baltieri
47ae66d828 scripts: release: strip title names
Make sure listed titles have no trailing/leading space (they often do
and right now they have to be cleaned manually before going into the
release notes file).

Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
2022-09-27 16:51:45 -05:00
Fabio Baltieri
daff79b957 scripts: release: set per_page=100 for issue requests
GitHub issue API allows a max of 100 entries per page (up to a default
of 30). Use it for list_issues, makes the script runs a lot faster.

Api documentation:
  https://docs.github.com/en/rest/issues/issues
  https://docs.github.com/en/rest/pulls/pulls

Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
2022-09-27 16:51:45 -05:00
Stephanos Ioannidis
f503847d14 doc: Do not use Sphinx 5.2.0.post0
Sphinx 5.2.0.post0 release has a known compatibility issue with the
`sphinx_rtd_theme` (see the issue readthedocs/sphinx_rtd_theme#1343).

Revert this commit once this compatibility issue is resolved.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2022-09-25 10:37:32 +02:00
Ming Shao
2cea5850e2 twisterlib: re-add the logic to drain the serial leftover
This logic was added before but seems to be missed during the
twister refactoring.

Some tests can cause serial leftover logs buffered somewhere.
Such leftover can interfere with the next test case because the
serial log monitoring thread is started before a board is flashed.
And the monitoring thread can be fooled by such leftover logs and
make incorrect judgement of the test result.

A simple ser.flush() is not enough to eliminate such leftovers.
So add explicit readline() to drain such logs which ensures a clean
serial context for the case that follows.

An example from reel board captured with this patch:

leftover log of previous test: b'.287 seconds\r\n'
leftover log of previous test: b' - SKIP - [...test_coredump_backend]...
leftover log of previous test: b'\r\n'
leftover log of previous test: b'------ TESTSUITE SUMMARY END ------\r\n'
leftover log of previous test: b'\r\n'
leftover log of previous test: b'=====================================...
leftover log of previous test: b'RunID: 4e93757ad...53dcab9f0f5c6\r\n'
leftover log of previous test: b'PROJECT EXECUTION SUCCESSFUL\r\n

Signed-off-by: Ming Shao <ming.shao@intel.com>
2022-09-22 15:15:55 -05:00
Fabio Baltieri
96fc137654 twister: treat imgtool image exceeded as flash overflow
Twister detects FLASH overflow and skips tests that trigger that
condition by default, but sometimes images are just on the flash size
limit and then overflows at a later stage when imgtool adds a trailer,
which gets detected as a build fail and fails the run.

This detects the imgtool flash overflow together with the normal build
ones, causing the build to be skipped on imgtool flash overflow as well.

Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
2022-09-22 12:41:25 +00:00
Anas Nashif
b3ead37efa twister: temporarily disable mec15xx boards
A workaround to exclude boards from twister. Other solutions do not work,
setting twister fields to false in the board yaml file is not enough,
given that this board has many associated samples and tests, so it can't
be disabled on its own.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-09-21 18:05:51 +00:00
Martí Bolívar
22012894c0 dtlib: error out on duplicate node names
Attempts to define two nodes with the same name within a single set of
curly brackets should fail.

For example, this is invalid DTS according to dtc:

    / { foo {}; foo {}; };

By contrast, this is valid since the node named 'foo' appears twice in
two different sets of curly brackets:

    / { foo {}; };
    / { foo {}; };

Zephyr's dtlib currently does not error out on the invalid condition.

Now that Zephyr itself has been updated to not include such nodes (to
the best of my ability), we can fix this divergence from current dtc
behavior and add a regression test in dtlib.

Fixes: #49590
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2022-09-21 07:55:40 -07:00
Yves Vandervennet
9a4272853e west: jlink: adding optional flash loader flag required for some boards
With revision 7.70d, Segger changed the command line interface of the
jlink client, requiring some flash specific information, such as
the base address and the type of memory, e.g. QSPI.

This commit adds an optional argument to the West jlink runner,
'--loader' which passes the information to the jlink client.
This change is backward compatible with versions of jlink older
than v7.70d which introduces the CLI change.

Fixes: #50327

Signed-off-by: Yves Vandervennet <yves.vandervennet@nxp.com>
2022-09-21 09:33:42 -05:00
Daniel DeGrasse
6064da8690 scripts: kconfig: update size functions to convert bits to bytes
Update size functions within kconfig helpers to support converting
values stored in kilobits, mebibits, or gibibits to kilobytes,
mebibytes, and gibibytes via use of different size arguments.

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
2022-09-21 15:22:36 +02:00
Torsten Rasmussen
65f45507a5 west: fix warning on user args in run_common
When using west runners with multi domains we want to warn users when
they user runner args on multi domain builds.

Therefore fix the `>1` to the correct `>0` which ensures the warning
is printed in all expected cases.

Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
2022-09-20 16:58:09 +02:00
Daniel Leung
418c915f1b dts: add primitive support for multi-bus in bindings
For a single bus that supports multiple protocols, e.g. I3C and I2C,
the single value "bus:" setting is no longer sufficient, as a I3C bus
cannot be matched to a device having "on-bus: I2C". This commit
extends the "bus:" setting so that it can accept a list of values.
This change allows corresponding devicetree macros to be generated
so that DT_ON_BUS() can work properly in this scenario.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2022-09-09 17:42:33 -04:00
Stephanos Ioannidis
3e706376e6 scripts: ci: test_plan: Specify ZEPHYR_BASE for list_boards
The commit c3620c8a8e changed the
`list_boards` script such that it no longer searches the boards from
the `ZEPHYR_BASE` path by default.

This commit updates the `test_plan` script to invoke the `list_boards`
script with the `ZEPHYR_BASE` as a search path.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2022-09-08 18:17:10 +09:00
Martí Bolívar
7ef9c4b20e edtlib: remove python 3.5 workaround
Remove a yaml monkeypatch. It is no longer needed since we support 3.6
or later on Zephyr v2.7 LTS and 3.8 or later on what will become v3.2.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2022-09-07 15:38:26 +02:00
Gerard Marull-Paretas
e6fca0ef89 scripts: ci: check_compliance: ignore missing APP_LOG_LEVEL
Application log level is not detected correctly as the option is defined
using a template, so it can't be grepped.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2022-09-07 11:18:27 +00:00
Carles Cufi
13ea65b6a9 west: Add completion support for zsh
As part of the overall work on improving west integration with Zephyr,
add support for zsh completion in the `west completion` command.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-09-06 08:43:15 -07:00
Gerard Marull-Paretas
b0d4aea769 scripts: migrate_includes: migrate zephyr.h to kernel.h
The script will now automatically migrate all <zephyr.h> or
<zephyr/zephyr.h> (for already migrated projects) to <zephyr/kernel.h>.

Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
2022-09-05 16:31:47 +02:00
Gerard Marull-Paretas
79e6b0e0f6 includes: prefer <zephyr/kernel.h> over <zephyr/zephyr.h>
As of today <zephyr/zephyr.h> is 100% equivalent to <zephyr/kernel.h>.
This patch proposes to then include <zephyr/kernel.h> instead of
<zephyr/zephyr.h> since it is more clear that you are including the
Kernel APIs and (probably) nothing else. <zephyr/zephyr.h> sounds like a
catch-all header that may be confusing. Most applications need to
include a bunch of other things to compile, e.g. driver headers or
subsystem headers like BT, logging, etc.

The idea of a catch-all header in Zephyr is probably not feasible
anyway. Reason is that Zephyr is not a library, like it could be for
example `libpython`. Zephyr provides many utilities nowadays: a kernel,
drivers, subsystems, etc and things will likely grow. A catch-all header
would be massive, difficult to keep up-to-date. It is also likely that
an application will only build a small subset. Note that subsystem-level
headers may use a catch-all approach to make things easier, though.

NOTE: This patch is **NOT** removing the header, just removing its usage
in-tree. I'd advocate for its deprecation (add a #warning on it), but I
understand many people will have concerns.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2022-09-05 16:31:47 +02:00
Emil Gydesen
174c74206e footprint: Add bt_hci_rpmsg with ISO for the nRF5340 netcore
Add footprint tracking of the samples/bluetooth/hci_rpmsg
sample for the nRF5340 with ISO broadcast and ISO receive respectively.

Signed-off-by: Emil Gydesen <emil.gydesen@nordicsemi.no>
2022-09-02 11:08:31 +00:00
Maureen Helm
5b5aa6ebba scripts: gen_defines: Add vendor name macros based on vendor prefixes
Adds vendor name and existence macros for all compatibles of a node that
match an entry in the vendor prefixes file.

Signed-off-by: Maureen Helm <maureen.helm@intel.com>
2022-09-01 14:53:55 -07:00
Maureen Helm
a1b0c39c37 scripts: edtlib: Add compat2vendor LUT constructed from vendor prefixes
Adds a new compat2vendor lookup table that maps compatibles to vendor
names, constructed from the vendor prefixes file. This approach is a
more scalable alternative to adding a vendor name property to devicetree
bindings, as was previously proposed.

Signed-off-by: Maureen Helm <maureen.helm@intel.com>
2022-09-01 14:53:55 -07:00
Torsten Rasmussen
8d2998d4f9 cmake: rework of host tools and generic toolchain handling
Follow-up: #41301

This commit is a rework and cleanup of the tools handling in Zephyr
CMake build system.

Instead of directly loading code a CMake modules for tool lookup, the
host tools now follows the CMake `find_package()` pattern for finding
programs / tools in module mode.

This makes it more clear which modules are responsible for finding tools
and which modules provides build integration / features.

The following tools can now be found using `find_package()`:
- Zephyr-sdk        : find_package(Zephyr-sdk <version>)
- Generic host tools: find_package(HostTools)

This further allows us to decouple the `verify-toolchain` CMake script
part required by `twister` into a tool lookup module and a dedicated
CMake script which utilizes the lookup module.

Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
2022-09-01 15:59:43 +02:00
Tristan Honscheid
52db2d8279 twister: Remove import exception handler for psutil
`handlers.py` treats the `psutil` package as optional and quietly prints
a message to stdout if it is not present, but it is actually a hard
requirement for the base `Handler` class. This try/except block is bad
as it hides the requirement until a timeout occurs and the handler
attempts to use `psutil` in the terminate method, which will crash if
the module happens to be missing.

This PR removes the try/except guard so missing `psutil` will cause
twister to immediately fail. `psutil` is already specified in the
`requirements.txt` file, so this should only affect users who
unwittingly have an outdated/incorrect environment.

Also update the Github Action for twister tests to install from
requirements files.

Signed-off-by: Tristan Honscheid <honscheid@google.com>
2022-09-01 06:22:04 -04:00
Erwan Gouriou
7cd727b003 scripts: requirements-extras: Request imgtool 1.9 for write-block-size > 8
On some targets (such as STM32U5) using write-block-size >8, imgtool
should be called using option '--align 16/32' which is only available
starting version 1.9.

Update requirement for imgtool to make it available to such platforms.

Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
2022-09-01 10:28:33 +02:00
Andrzej Kaczmarek
868acfd505 scripts: Add ezFlashCLI runner
This adds support for ezFlashCLI tool to allow flashing DA1469x MCU
family from Renesas.

The tool is available at https://github.com/ezflash/ezFlashCLI or can
be installed via pip.

Signed-off-by: Andrzej Kaczmarek <andrzej.kaczmarek@codecoup.pl>
Signed-off-by: Ben Lauret <ben.lauret.wm@renesas.com>
2022-08-31 20:32:03 +02:00
Emil Gydesen
6cc8b57bc5 footprint: Add bt_hci_rpmsg for the nRF5340 netcore
Add footprint tracking of the samples/bluetooth/hci_rpmsg
sample for the nRF5340. This is a BT controller-only build
for the nRF5340 with minimum features.

Signed-off-by: Emil Gydesen <emil.gydesen@nordicsemi.no>
2022-08-31 10:41:32 +02:00
Carles Cufi
8644b333cd west: Update bash autocomplete file
Update the existing bash autocomplete with the latest commands and
command-line switches.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-08-31 10:38:01 +02:00
Gerard Marull-Paretas
fff9ecbc7f devicetree: add DT_(INST_)FOREACH_CHILD(_STATUS_OKAY)_SEP(_VARGS)
It is frequent to see in Devicetree code constructs like:

```c
 #define NAME_AND_COMMA(node_id) DT_NODE_FULL_NAME(node_id),

const char *child_names[] = {
	DT_FOREACH_CHILD(DT_NODELABEL(n), NAME_AND_COMMA)
};
```

That is, an auxiliary macro to append a separator character in
DT_FOREACH* macros. Non-DT API, e.g. FOR_EACH(), takes a separator
argument to avoid such intermediate macros.

This patch adds DT_FOREACH_CHILD_SEP (and instance/status okay/vargs
versions of it). They all take an extra argument: a separator. With this
change, the example above can be simplified to:

```c
const char *child_labels[] = {
	DT_FOREACH_CHILD(DT_NODELABEL(n), DT_NODE_FULL_NAME, (,))
};
```

Notes:
- Other DT_FOREACH* macros could/should be extended as well

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2022-08-30 16:19:57 +02:00
Gerard Marull-Paretas
d77f4e61de scripts: dts: gen_defines: add DT_DEBRACKET_INTERNAL helper
Add a utility macro used to remove brackets from around a single
argument. While __DEBRACKET exists in util_internal.h, this change makes
DT independent (otherwise we should include util_internal.h, which is
another option).

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2022-08-30 16:19:57 +02:00
Thomas Stranger
f123094482 scripts: west: runners: stm32cubeprogrammer: enable exec from PATH
Allows to use the stm32cubeprogrammer runner on Linux to be executed
from any installation directory as long as the executable is found
in PATH.

Until now on Linux the programmer could only be used if it either has
been installed in the default location or if the executable path was
passed via the cli parameter.

Signed-off-by: Thomas Stranger <thomas.stranger@outlook.com>
2022-08-29 17:31:56 +02:00
Carles Cufi
035dffcfcf scripts: modules: Mark the build as tainted if blobs present
Whenever binary blobs are present in the filesystem (i.e. fetched,
regardless of whether they are up-to-date or not) we consider the Zephyr
build as tainted. In order to ensure that it is marked as so, add a
Kconfig `select` statement whenever this is the case.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-08-29 15:57:59 +02:00
Carles Cufi
b662bc9093 scripts: modules: Move blob processing to zephyr_module.py
In preparation for the handling of taint flags in zephyr_module.py, move
blob-processing code from the west command to it.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-08-29 15:57:59 +02:00