This change enhances the devicetree library by adding support for tracking
the source filename and line number for nodes and properties.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Previously, dtlib would fail to parse the following:
/delete-node/ &{/};
This is accepted by dtc, so dtlib should be aligned.
The expected behavior is that the contents of the "deleted" root node
are emptied, but the node itself remains in the tree. This means that
it's possible to put that statement at the end of a DTS file and still
get a valid output. A small test case for this scenario is included.
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
We need to have an _include_path attribute to pretty-print
this object from within pdb, for some reason. Add it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
There's no need for _parse_node() to return the Node instance that is
its sole argument. The only user of the return value is a dead store.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This helper lets you place a node (really the entire subtree rooted at
that node) elsewhere in the devicetree. This will be useful when
adding system devicetree support, when we'll want to be able to, for
example, move the CPU cluster node selected by the current execution
domain to /cpus.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The standard library copy module allows you to implement shallow and
deep copies of objects. See its documentation for more details on
these terms.
Implementing copy.deepcopy() support for DT objects will allow us to
"clone" devicetree objects in other classes. This in turn will enable
new features, such as native system devicetree support, within the
python-devicetree.
It is also a pure feature extension which can't harm anything and is
therefore safe to merge now, even if system devicetree is never
adopted in Zephyr.
Note that we are making use of the move from OrderedDict to regular
dict to make this implementation more convenient.
See https://github.com/devicetree-org/lopper/ for more information on
system devicetree. We want to add system devicetree support to dtlib
because it seems to be a useful way to model modern, heterogeneous
SoCs than traditional devicetree, which can really only model a single
CPU "cluster" within such an SoC.
In order to create 'regular' devicetrees from a system devicetree, we
will want a programming interface that does the following:
1. parse the system devicetree
2. receive the desired transformations on it
3. perform the desired transformations to make
a 'regular' devicetree
Step 3 can be done as a destructive modification on an object-oriented
representation of a system devicetree, and that's the approach we will
take in python-devicetree. It will therefore be convenient to have an
efficient deepcopy implementation to be able to preserve the original
system devicetree and the derived regular devicetree in memory in the
same python process.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Regular dicts are insertion-ordered since CPython 3.6 and Python 3.7.
Zephyr now requires Python 3.8, so it should be OK to replace
OrderedDict with regular dict now. This results in less typing and
more readable object representations.
A nitpicker could argue that this is a functional change, since if a
user is doing 'assert isinstance(node.props, OrderedDict)', that will
fail now, but:
1. nobody is doing something like that in the zephyr tree
2. that would be a silly thing to do
3. we don't currently make any API stability guarantees
for this module right now anyway
so it should be fine.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Refactor the file parsing methods for readability by moving the
_parse_header() and _parse_memreserves() calls from _parse_dt() to
_parse_file(). The header and memreserves are not part of the 'tree'
part of the devicetree; now that we have a dedicated _parse_file()
helper, it makes more sense to me to have them there.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Holy overloaded technical terms, Batman.
Here, 'property' and 'type' each mean two different things, which
we can distinguish like this:
- Property (capital P): dtlib.Property class, represents
a property in a devicetree node
- @property: a Python property
- type(): an "@property" in the Property class, that returns
a dtlib.Type value
- Type (capital T): dtlib.Type class, represents the devicetree
type of a Property value (dtlib.Type.BYTES, etc.)
The type() @property in the Property class currently has an 'int' as
its Python return type annotation. It really returns a dtlib.Type,
which is an int (since it's an IntEnum), but that's not the same thing
as an int.
Change this to Type to be clear that not just any int can be returned
by this @property.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Make attribute initialization order match the order that attributes
appear within the class docstring. Move the 'type' property definition
up by the constructor to make it more obvious that this 'attribute' is
a (Python) property. This is for readability.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Reorder attribute initialization to match the order that attributes
appear in the class level docstring. This is for readability.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Initialize all the public API interface related attributes within the
constructor instead of scattering them throughout the implementation
of the class, and make sure they all have type annotations.
Move all the parsing code away from the init routines and public API
down to the main parsing block.
This is for readability and paves the way for later changes that
affect the way initialization happens.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Attempts to define two nodes with the same name within a single set of
curly brackets should fail.
For example, this is invalid DTS according to dtc:
/ { foo {}; foo {}; };
By contrast, this is valid since the node named 'foo' appears twice in
two different sets of curly brackets:
/ { foo {}; };
/ { foo {}; };
Zephyr's dtlib currently does not error out on the invalid condition.
Now that Zephyr itself has been updated to not include such nodes (to
the best of my ability), we can fix this divergence from current dtc
behavior and add a regression test in dtlib.
Fixes: #49590
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
A GCC linemarker of the form:
# 1 "filename" 2 3 4
or so is not currently being handled, because the current regular
expression assumes the "flags" values (the numbers after "filename")
are limited to a single value. Tweak the regular expression to allow
for up to 4 flags, which is what GCC documents it may emit:
https://gcc.gnu.org/onlinedocs/gcc-10.2.0/cpp/Preprocessor-Output.html
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Node names are subject to the rules in table 2.1 of the devicetree
specification v0.3, while properties are subject to rules in table
2.2. These rules mean that some property names are invalid node names.
However, the same regular expression is being used to validate the
names of nodes and properties in dtlib. This leads to invalid node
names being allowed to pass. Fix this issue by moving the node name
handling code to the Node constructor and checking against the
characters in table 2.1.
The test cases claim that the existing behavior matches dtc. I can't
reproduce that. I get errors when I use invalid characters (like "?")
in a node name. For example:
foo.dts:3.8-11: ERROR (node_name_chars): /node?: Bad character '?' in
node name
Try to make the dtlib error message reminiscent of that.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is unused since the very beginning of the module's introduction.
It looks like it was abandoned in favor of the approach where each
token can have only one capturing group.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This became useless when _init_tokens() was refactored not to use
global variables (in "dtlib: use IntEnum for token IDs"), and the
linter is complaining about it now.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The library was originally developed before Python 3.6 was the minimum
supported version. Use f-strings now that we can do that, as they tend
to be easier to read.
There are a few places where str.format() makes sense to preserve,
specifically where the same argument is used multiple times; leave
those alone.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
As a first step towards being more forgiving on invalid inputs, allow
string-valued aliases properties that do not point to valid nodes when
the user requests permissiveness.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Modeled after dtc's --force option, the idea is this will try harder
and harder over time to produce an object despite malformed input.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Now that all the other code it depends on is annotated, we can finish
up the type annotation of this module in the main DT class.
It's not worth it to try to annotate the private methods (the ones
that begin with '_'). Most of these are low level lexing helpers that
aren't particularly amenable to static type checking, because the type
of a token's value is often dependent on the token ID in ways that
static type annotations are not well equipped to capture.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We'd like users of this API to know that DT.root is always a Node,
and not an Optional[Node].
However, although DT.__init__ throws an exception if the resulting DT
object would have no root node, static analysis can't tell that since
the root instance attribute starts out as None during initialization,
so checkers like mypy are convinced it's Optional[Node].
Since this is really OK, we'll quiet the type checker down by stashing
the instance attribute in self._root instead, and providing a root
property accessor that is annotated to return Node instead of
Optional[Node]. We can tell mypy to ignore what looks like a potential
None here to allow callers to treat the result as a Node.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The documentation says DT.__init__ takes any iterable for the
include_path, but this leads to bad results when you pass it something
other than a 'real' sequence (list/tuple/etc), like a generator:
>>> dt = DT('/tmp/foo.dts', (x for x in ['a', 'b', 'c']))
>>> repr(dt)
"DT(filename='/tmp/foo.dts', include_path=<generator object ...>)"
Make a copy in list form just to avoid things like this.
Add a test for this and relax the regular expression in the existing
test case related to this.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Some of these are also tripping up a python 2 / python 3 warning
in mypy in the way that '{}'.format(b'foo') works, which we silence by
explicitly requesting the python 3 behavior.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The way that _init_tokens() is manipulating globals() defeats static
analyses of the file that are trying to infer a type for the 'tok_id'
variable in assignment expressions like 'tok_id = _T_INCLUDE'.
To make it easier on the analyser, define the token types as an
enum.IntEnum named _T. This means we can write e.g. '_T.INCLUDE'
instead of '_T_INCLUDE', avoiding line length increases in the lexing
code.
While we're here, use '==' and '!=' instead of 'is' and 'is not'
when comparing a tok_id that is obtained from an re.Match.lastindex
with a _T.FOO value.
This is now necessary since an int object and a _T object definitely
don't point to the same memory. It worked previously because CPython
interns all integer instances from -5 to 256, but that's an
implementation detail and not a language feature. Since we're getting
the ints from an re.Match.lastindex instead of putting the exact
_T_FOO values into some list, this code probably should not strictly
speaking have been using 'is'.
Explicitly initialize the global _token_re also, to make it more
visible for static analysis.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Continue annotating the module. In some cases mypy will miss that
_err() calls means the function will not return, so we return an
unnecessary local variable to appease it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add a _MarkerType enum. A subsequent commit will use it for type
annotations in the Property class.
Fix an incorrect type in a comment while we're here.
We continue to use 'marker_type is _MarkerType.FOO' instead of
'marker_type == _MarkerType.FOO' because we are adding those actual
_MarkerType.FOO objects to each property, so 'is' comparison
is legitimate.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
A step along the way towards typing the whole module.
Fix an incorrect (or at best misleading) comment while we're here,
which was noticed by the type checker when I originally annotated
'props' as a Dict[str, bytes].
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Marking this NoReturn helps the type checker figure out that functions
which call it are only returning valid values or failing to
return. (It unfortunately doesn't always work as mypy's control flow
analysis seems to treat a direct 'raise DTError(...)' differently than
calling _err() in some situations, but it helps.)
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Move the DTError, Node, Type, and Property definitions to the top.
This way, class definitions occur before methods which use those
classes. This will be useful to avoid string literals in type
annotations that will be added later. Some can't be avoided due to
circular dependencies, but this will help.
Adjust whitespace.
No functional changes expected.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Instead of hard-coding constants, use an IntEnum.
These is still a subclass of 'int', but is both easier to import and
easier to read during debugging.
For example, compare:
>>> Type.BYTES
<Type.BYTES: 1>
with:
>>> TYPE_BYTES
1
However, 'Type.BYTES == 1' is still True, and the enum values
otherwise behave like you would expect.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We are now in the process of extracting edtlib and dtlib into a
standalone source code library that we intend to share with other
projects.
Links related to the work making this standalone:
https://pypi.org/project/devicetree/https://python-devicetree.readthedocs.io/en/latest/https://github.com/zephyrproject-rtos/python-devicetree
This standalone repo includes the same features as what we have in
Zephyr, but in its own 'devicetree' python package with PyPI
integration, etc.
To avoid making this a hard fork, move the code that's being made
standalone around in Zephyr into a new scripts/dts/python-devicetree
subdirectory, and handle the package and sys.path changes in the
various places in the tree that use it.
From now on, it will be possible to update the standalone repository
by just recursively copying scripts/dts/python-devicetree's contents
into it and committing the results.
This is an interim step; do NOT 'pip install devicetree' yet.
The code in the zephyr repository is still the canonical location.
(In the long term, people will get the devicetree package from PyPI
just like they do the 'yaml' package today, but that won't happen for
the foreseeable future.)
This commit is purely intended to avoid a hard fork for the standalone
code, and no functional changes besides the package structure and
location of the code itself are expected.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
