Commit Graph

521 Commits

Author SHA1 Message Date
Daniel Leung
03eded1ed6 kernel: mm: rename Z_VIRT_RAM_* to K_MEM_VIRT_*
Renames:
  Z_VIRT_RAM_START to K_MEM_VIRT_RAM_START
  Z_VIRT_RAM_SIZE to K_MEM_VIRT_RAM_SIZE
  Z_VIRT_RAM_END to K_MEM_VIRT_RAM_END

This is part of a series to move memory management related
stuff from Z_ namespace into its own namespace.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-06-12 21:13:26 -04:00
Daniel Leung
3fd66de508 kernel: mm: rename Z_PHYS_RAM_* to K_MEM_PHYS_*
Renames:
  Z_PHYS_RAM_START to K_MEM_PHYS_RAM_START
  Z_PHYS_RAM_SIZE to K_MEM_PHYS_RAM_SIZE
  Z_PHYS_RAM_END to K_MEM_PHYS_RAM_END

This is part of a series to move memory management related
stuff from Z_ namespace into its own namespace.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-06-12 21:13:26 -04:00
Daniel Leung
b6f951b99e kernel: mm: rename Z_VM_OFFSET to K_MEM_VM_OFFSET
This is part of a series to move memory management related
stuff from the Z_ namespace into its own namespace.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-06-12 21:13:26 -04:00
Daniel Leung
def364ab08 kernel: mm: rename Z_BOOT_* to K_MEM_BOOT_*
Rename Z_BOOT_VIRT_TO_PHYS() and Z_BOOT_PHYS_TO_VIRT() to
K_MEM_BOOT_VIRT_TO_PHYS() and K_MEM_BOOT_PHYS_TO_VIRT()
respectively. This is part of a series to move memory management
functions away from the Z_ namespace and into its own namespace.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-06-12 21:13:26 -04:00
frei tycho
4c2938a295 kernel: added missing parenthesis
- added missing parenthesis around macro argument expansion

Signed-off-by: frei tycho <tfrei@baumer.com>
2024-06-07 12:59:46 +02:00
Peter Mitsis
d8a4c8a90c kernel: Add CONFIG_IPI_OPTIMIZE
The CONFIG_IPI_OPTIMIZE configuration option allows for the flagging
and subsequent signaling of IPIs to be optimized.

It does this by making each bit in the kernel's pending_ipi field
a flag that indicates whether the corresponding CPU might need an IPI
to trigger the scheduling of a new thread on that CPU.

When a new thread is made ready, we compare that thread against each
of the threads currently executing on the other CPUs. If there is a
chance that that thread should preempt the thread on the other CPU
then we flag that an IPI is needed for that CPU. That is, a clear bit
indicates that the CPU absolutely will not need to reschedule, while a
set bit indicates that the target CPU must make that determination for
itself.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-06-04 22:35:54 -04:00
Yong Cong Sin
6a3cb93d88 arch: remove the use of z_arch_esf_t completely from internal
Created `GEN_OFFSET_STRUCT` & `GEN_NAMED_OFFSET_STRUCT` that
works for `struct`, and remove the use of `z_arch_esf_t`
completely.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-06-04 14:02:51 -05:00
Yong Cong Sin
e54b27b967 arch: define struct arch_esf and deprecate z_arch_esf_t
Make `struct arch_esf` compulsory for all architectures by
declaring it in the `arch_interface.h` header.

After this commit, the named struct `z_arch_esf_t` is only used
internally to generate offsets, and is slated to be removed
from the `arch_interface.h` header in the future.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-06-04 14:02:51 -05:00
Yong Cong Sin
bbe5e1e6eb build: namespace the generated headers with zephyr/
Namespaced the generated headers with `zephyr` to prevent
potential conflict with other headers.

Introduce a temporary Kconfig `LEGACY_GENERATED_INCLUDE_PATH`
that is enabled by default. This allows the developers to
continue the use of the old include paths for the time being
until it is deprecated and eventually removed. The Kconfig will
generate a build-time warning message, similar to the
`CONFIG_TIMER_RANDOM_GENERATOR`.

Updated the includes path of in-tree sources accordingly.

Most of the changes here are scripted, check the PR for more
info.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-05-28 22:03:55 +02:00
Flavio Ceolin
4d85f3d91c pm: Deprecate z_pm_save_idle_exit
Deprecate z_pm_save_idle_exit and promote pm_system_resume.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2024-05-27 02:10:03 -07:00
Nicolas Pitre
b34e94f362 kernel: demand_paging: fix arch_page_location_get() documentation
Symbols from enum arch_page_location are defined as
ARCH_PAGE_LOCATION_* and not ARCH_PAGE_FAULT_*.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-05-24 07:47:49 -04:00
Nicolas Pitre
e9a47d932c kernel: mmu: shrink and align struct z_page_frame
The struct z_page_frame is marked __packed to avoid extra padding as
such padding may represent significant memory waste when lots of page
frames are used. However this is a bad strategy.

The code contained this somewhat dubious comment and code in
free_page_frame_list_put():

	/* The structure is packed, which ensures that this is true */
	void *node = pf;
	sys_slist_append(&free_page_frame_list, node);

This is bad for many reasons:

- type checking is completely bypassed;

- if the sys_snode_t node member is no longer located at the front of
  struct z_page_frame then the code will still compile and possibly run
  but be broken with memory corruption as a likely outcome;

- the sys_slist_append() code is completely unaware of the packed
  attribute which breaks architectures with alignment restrictions.

Let's improve code efficiency as well as memory usage by removing the
packed attribute and manually packing the flags in the unused virtual
address bits. This way the page frame array remains naturally aligned,
data access becomes optimal and the actual array size gets even smaller.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-05-13 16:04:40 -04:00
Nicolas Pitre
57305971d1 kernel: mmu: abstract access to page frame flags and address
Introduce z_page_frame_set() and z_page_frame_clear() to manipulate
flags. Obtain the virtual address using the existing
z_page_frame_to_virt(). This will make changes to the page frame
structure easier.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-05-13 16:04:40 -04:00
Hess Nathan
6d417d52c2 coding guidelines: comply with MISRA Rule 12.1.
added parentheses verifying lack of ambiguities

Signed-off-by: Hess Nathan <nhess@baumer.com>
2024-05-12 13:37:27 -04:00
Hess Nathan
e05c4a8786 coding guidelines: comply with MISRA Rule 11.8
- modified parameter types to receive a const pointer when a
  non-const pointer is not needed

- avoided redundant casts

Signed-off-by: Hess Nathan <nhess@baumer.com>
2024-05-10 14:45:14 -05:00
Andy Ross
fd340ebf31 sched: Optimize dummy thread usage on SMP
Nicolas Pitre points out that since these thread structs are just
dummies for the context swtiching, they can be presumed to be "write
only" and thus there's no point in having one per CPU, everyone can
share the same one.

The only gotcha is that we never really documented (nor really have a
place to document) that rule, so it's not theoretically impossible for
an architecture to read back what it might have written underneath
arch_switch().  Leave this in a separate commit for bisection
purposes, but the risk seems very low.

Signed-off-by: Andy Ross <andyross@google.com>
2024-05-02 13:55:03 -04:00
Andy Ross
f0fd54cb31 kernel/sched: Fix free-memory write when ISRs abort _current
After a k_thread_abort(), the resulting thread struct is documented as
unused/free memory that may be re-used (for example, to respawn a new
thread).

But in the special case of aborting the current thread from within an
ISR, that wasn't quite happening.  The scheduler cleanup would
complete, but the architecture layer would still try to context switch
away from the aborted thread on exit, and that can include writes to
the now-reused thread struct!  The specifics will depend on
architecture (some do a full context save on entry, most don't), but
in the case of USE_SWITCH=y it will at the very least write the
switch_handle field.

Fix this simply, with a per-cpu "switch dummy" thread struct for use
as a target for context switches like this.  There is some non-trivial
memory cost to that; thread structs on many architectures are large.

Pleasingly, this also addresses a known deadlock on SMP: because the
"spin in ISR" step now happens as the very last stage of
k_thread_abort() handling, the existing scheduler lock works to
serialize calls such that it's impossible for a cycle of threads to
independently decide to spin on each other: at least one will see
itself as "already aborting" and break the cycle.

Fixes #64646

Signed-off-by: Andy Ross <andyross@google.com>
2024-05-02 13:55:03 -04:00
Hess Nathan
527e712448 coding guidelines: comply with MISRA Rule 20.9
- avoid to use undefined macros in #if expressions

Signed-off-by: Hess Nathan <nhess@baumer.com>
2024-05-01 19:48:19 +01:00
Hess Nathan
32af724fbb coding guidelines: comply with MISRA C:2012 Rule 11.2
avoid convert pointers to incomplete type using the pointer to first item

Signed-off-by: Hess Nathan <nhess@baumer.com>
2024-05-01 10:53:20 -04:00
Alberto Escolar Piedras
ea26bcf8d3 Revert "kernel/sched: Fix free-memory write when ISRs abort _current"
This reverts commit 61c70626a5.

This PR introduced 2 regressions in main CI:
71977 & 71978
Let's revert it by now to get main's CI passing again.

Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
2024-04-26 10:10:24 +00:00
Alberto Escolar Piedras
c9ec937d71 Revert "sched: Optimize dummy thread usage on SMP"
This reverts commit 20611f13ca.

This PR introduced 2 regressions in main CI:
71977 & 71978
Let's revert it by now to get main's CI passing again.

Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
2024-04-26 10:10:24 +00:00
Andy Ross
20611f13ca sched: Optimize dummy thread usage on SMP
Nicolas Pitre points out that since these thread structs are just
dummies for the context swtiching, they can be presumed to be "write
only" and thus there's no point in having one per CPU, everyone can
share the same one.

The only gotcha is that we never really documented (nor really have a
place to document) that rule, so it's not theoretically impossible for
an architecture to read back what it might have written underneath
arch_switch().  Leave this in a separate commit for bisection
purposes, but the risk seems very low.

Signed-off-by: Andy Ross <andyross@google.com>
2024-04-25 15:12:02 +02:00
Andy Ross
61c70626a5 kernel/sched: Fix free-memory write when ISRs abort _current
After a k_thread_abort(), the resulting thread struct is documented as
unused/free memory that may be re-used (for example, to respawn a new
thread).

But in the special case of aborting the current thread from within an
ISR, that wasn't quite happening.  The scheduler cleanup would
complete, but the architecture layer would still try to context switch
away from the aborted thread on exit, and that can include writes to
the now-reused thread struct!  The specifics will depend on
architecture (some do a full context save on entry, most don't), but
in the case of USE_SWITCH=y it will at the very least write the
switch_handle field.

Fix this simply, with a per-cpu "switch dummy" thread struct for use
as a target for context switches like this.  There is some non-trivial
memory cost to that; thread structs on many architectures are large.

Pleasingly, this also addresses a known deadlock on SMP: because the
"spin in ISR" step now happens as the very last stage of
k_thread_abort() handling, the existing scheduler lock works to
serialize calls such that it's impossible for a cycle of threads to
independently decide to spin on each other: at least one will see
itself as "already aborting" and break the cycle.

Fixes #64646

Signed-off-by: Andy Ross <andyross@google.com>
2024-04-25 15:12:02 +02:00
Anas Nashif
4593f0d71c kernel: priority queues: declare as static inlines
After the move to C files we got some drop in the performance when
running latency_measure. This patch declares the priority queue
functions as static inlines with minor optimizations.

The result for one metric (on qemu):

3.6 and before the anything was changed:

  Get data from LIFO (w/ ctx switch): 13087 ns

after original change (46484da502):

  Get data from LIFO (w/ ctx switch): 13663 ns

with this change:

  Get data from LIFO (w/ ctx switch): 12543 ns

So overall, a net gain of ~ 500ns that can be seen across the board on many
of the metrics.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-22 16:40:11 -04:00
Daniel Leung
378131c266 kernel: add options to cleanup after aborting current thread
This adds the mechanism to do cleanup after k_thread_abort()
is called with the current thread. This is mainly used for
cleaning up things when the thread cannot be running, e.g.,
cleanup the thread stack.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-04-10 07:44:27 -04:00
Anas Nashif
20b2c98add kernel: move nothread support to own file
Do not build threading support when CONFIG_MULTITHREADING=n is set and
move needed calls to a new file with the changes needed instead of the
ifdef party in sched.c

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-06 14:22:08 +03:00
Anas Nashif
8a88cd4805 kernel: thread: move thread states to header
Move state string defines into thread header.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-01 18:47:36 -04:00
Anas Nashif
5c170c7046 kernel: thread: rename is_preempt
Trivila rename to thread_is_preempt.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-01 18:47:36 -04:00
Anas Nashif
17c874f4fc kernel: thread: rename is_metairq
Trivial rename of is_metairq to thread_is_metairq.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-01 18:47:36 -04:00
Anas Nashif
37df485463 kernel: split timeslicing/ipi code out of sched.c
Move both timeslicing and IPI code to own files.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-01 18:47:36 -04:00
Anas Nashif
31bc210bbc kernel: sched: remove unused prototype: z_is_thread_time_slicing
A prototype not used anywhere.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-01 18:47:36 -04:00
Anas Nashif
ebb503ff7b kernel: move thread related helper function kthread.h
Move some helper functions to inernal kthread.h, to offload crowded
sched.c

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-01 18:47:36 -04:00
Simon Hein
bcd1d19322 kernel: add closing comments to config endifs
Add a closing comment to the endif with the configuration
information to which the endif belongs too.
To make the code more clearer if the configs need adaptions.

Signed-off-by: Simon Hein <Shein@baumer.com>
2024-03-25 18:03:31 -04:00
Daniel Leung
3664ed64c3 arch: move arch_interface.h under zephyr/arch
arch_interface.h is for architecture and should not be
under sys/. So move it under include/zephyr/arch/.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-03-25 09:58:35 +00:00
TaiJu Wu
1f5f0cf838 sched: Remove multi-level queue priority limit
Modified bitmask to  bitmask array, it can make multilevel queue remove
32 bit prioriry limit.

We can scan bitmask array to find which queue have ready thread.

Only need the number of queues as priority because the priority
is checked on create_thread.

Signed-off-by: TaiJu Wu <tjwu1217@gmail.com>
2024-03-12 19:37:40 -04:00
Peter Mitsis
9f7695dda0 kernel: Remove unused z_pend_curr_irqlock()
The routine z_pend_curr_irqlock() is no longer used anywhere.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-03-07 11:51:06 -05:00
Anas Nashif
0d8da5ff93 kernel: rename scheduler spinlock variable and make it private
rename sched_spinlock to _sched_spinglock to maintain it is privacy and
to avoid any misuse.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-06 19:27:28 -05:00
Anas Nashif
868f099d61 kernel: sched: z_set_prio -> z_thread_prio_set
Rename private function to make it clear what priority we are setting
and to be consistent across the code.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-06 19:27:28 -05:00
Anas Nashif
6c003bdbcf kernel: remove unused code in headers
List of functions defined in headers and not being used anywhere.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-06 19:27:28 -05:00
Anas Nashif
9e83413542 kernel: split thread monitor
Move thread monitor related functions, not enabled in most cases outside
of thread.c and cleanup headers.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-06 19:27:28 -05:00
Anas Nashif
5e591c38f1 kernel: do not export z_thread_priority_set
This function is only being used by a test, so instead of reimplementing
a syscall in the test, provide a Kconfig option to provide the
functionality that only works with tests and remove some of the
duplication and extra code.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-06 19:27:28 -05:00
Anas Nashif
3ca50f5060 kernel: move z_init_static_threads to where it is being used
Move out of thread and put directly in init.c where it is being used.
Also remove definition from kernel.h, this is an internal function and
should not be in a public header.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-06 19:27:28 -05:00
Anas Nashif
8791012ed1 kernel: move essential flag related routines out
The functions to manipulate the essential flag indeed operate on
threads, but they are misplaced in the thread implementation file. Put
them alongside other routines setting other thread flags and cleanup
headers a bit.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-06 19:27:28 -05:00
Anas Nashif
46484da502 kernel: move priority queue handling to own file/header
clean up headers under include/ and move handling of priority queue to
own file/header.
No need for the header  include/zephyr/kernel/internal/sched_priq.h
anymore. Move the relevant structures where they are being used in
kernel_structs.h.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-02 15:06:45 +01:00
Peter Mitsis
ee9c44fee6 linker: Round TLS sizes up in linker script
Instead of rounding up both __tdata_size and __tbss_size at runtime,
perform the calculation when the image is built.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-02-25 22:28:56 -05:00
Nguyen Minh Thien
8188be57d3 kernel: fix spelling errors
Fix spelling errors found in comment of the kernel source code.

Signed-off-by: Nguyen Minh Thien <nguyenmthien@live.com>
2024-02-25 20:53:37 -05:00
Krzysztof Chruściński
25173f71cd pm: device_runtime: Extend with synchronous runtime PM
In many cases suspending or resuming of a device is limited to
just a few register writes. Current solution assumes that those
operations may be blocking, asynchronous and take a lot of time.
Due to this assumption runtime PM API cannot be effectively used
from the interrupt context. Zephyr has few driver APIs which
can be used from an interrupt context and now use of runtime PM
is limited in those cases.

Patch introduces a new type of PM device - synchronous PM. If
device is specified as capable of synchronous PM operations then
device runtime getting and putting is executed in the critical
section. In that case, runtime API can be used from an interrupt
context. Additionally, this approach reduces RAM needed for
PM device (104 -> 20 bytes of RAM on ARM Cortex-M).

Signed-off-by: Krzysztof Chruściński <krzysztof.chruscinski@nordicsemi.no>
2024-02-01 15:03:42 +01:00
Daniel Leung
2cdd44801e kernel: move z_init_cpu to private kernel headers
z_init_cpu() is a private kernel API so move it under
kernel/include.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-01-17 11:57:20 -05:00
Daniel Leung
a7dccc4475 kernel: mmu: mitigate range check overflow issue
It is possible that address + size will overflow the available
address space and the pointer wraps around back to zero. Some
of these have been fixed in previous commits. This fixes
the remaining ones with regard to Z_PHYS_RAM_START/_END,
and Z_VIRT_RAM_START/_END.

Fixes #65542

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2023-12-20 11:37:17 -05:00
Johan Hedberg
3fbf12487c kernel: Introduce a way to specify minimum system heap size
There are several subsystems and boards which require a relatively large
system heap (used by k_malloc()) to function properly. This became even
more notable with the recent introduction of the ACPICA library, which
causes ACPI-using boards to require a system heap of up to several
megabytes in size.

Until now, subsystems and boards have tried to solve this by having
Kconfig overlays which modify the default value of HEAP_MEM_POOL_SIZE.
This works ok, except when applications start explicitly setting values
in their prj.conf files:

$ git grep CONFIG_HEAP_MEM_POOL_SIZE= tests samples|wc -l
     157

The vast majority of values set by current sample or test applications
is much too small for subsystems like ACPI, which results in the
application not being able to run on such boards.

To solve this situation, we introduce support for subsystems to specify
their own custom system heap size requirement. Subsystems do
this by defining Kconfig options with the prefix HEAP_MEM_POOL_ADD_SIZE_.
The final value of the system heap is the sum of the custom
minimum requirements, or the value existing HEAP_MEM_POOL_SIZE option,
whichever is greater.

We also introduce a new HEAP_MEM_POOL_IGNORE_MIN Kconfig option which
applications can use to force a lower value than what subsystems have
specficied, however this behavior is disabled by default.

Whenever the minimum is greater than the requested value a CMake warning
will be issued in the build output.

This patch ends up modifying several places outside of kernel code,
since the presence of the system heap is no longer detected using a
non-zero CONFIG_HEAP_MEM_POOL_SIZE value, rather it's now detected using
a new K_HEAP_MEM_POOL_SIZE value that's evaluated at build.

Signed-off-by: Johan Hedberg <johan.hedberg@intel.com>
2023-12-20 11:01:42 +01:00
Daniel Leung
fa561ccd59 kernel: mmu: no need to expose z_free_page_count
z_free_page_count is only used in one file, so there is
no need to expose it, even to other part of kernel.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2023-12-12 18:46:21 +00:00
Daniel Leung
34c6b17680 doc: fixed already renamed _Cstart to z_cstart
The function _Cstart has already been renamed to z_cstart,
so change the remaining references of it in various docs.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2023-11-30 21:01:47 -05:00
Daniel Leung
c972ef1a0f kernel: mm: move kernel mm functions under kernel includes
This moves the k_* memory management functions from sys/ into
kernel/ includes, as there are kernel public APIs. The z_*
functions are further separated into the kernel internal
header directory.

Also made a quick change to doxygen to group sys_mem_* into
the OS Memory Management group so they will appear in doc.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2023-11-20 09:19:14 +01:00
Flavio Ceolin
f9c7a5e6fb kernel: random: Rename early random get function
Rename z_early_boot_rand_get with z_early_rand_get to get consistent
with other early functions.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2023-10-13 10:03:53 +03:00
Daniel Leung
2c2d53c7e5 kernel: remove deprecate wording on arch_kernel_init()
The wording on deprecating arch_kernel_init() in favor of prep_c()
has never been materialized. Various architectures are using it to
perform initialization. So remove the wording.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2023-10-09 10:15:49 +02:00
Daniel Leung
dd6a7eb77d kernel: demand_paging: add doc to enum arch_page_location
This adds doc to enum arch_page_location.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2023-10-09 10:15:49 +02:00
Daniel Leung
77dc74136c kernel: tls: fix doc on arch_tls_stack_setup()
There was a typo there so fix it.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2023-10-09 10:15:49 +02:00
Anas Nashif
a1c7bfbc63 kernel: remove unused z_init_thread_base from kernel.h
This API is internal and not used in any way in kernel.h, so move it
back to where it is needed.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-09-30 18:43:28 +02:00
Anas Nashif
e19f21cb27 kernel: move z_is_thread_essential out of public kernel header
This is a private API to the kernel, so move out of kernel.h

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-09-30 18:43:28 +02:00
Anas Nashif
f0c7fbf0f1 kernel: move sched_priq.h to internal/ folder
This header is internal to the kernel and shall not be included directly.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-09-30 18:43:28 +02:00
Peter Mitsis
e6f1090553 kernel: Integrate object core statistics
Integrates object core statistics framework into the following
kernel objects:
  sys_mem_blocks, k_mem_slab
  threads, _cpu, z_kernel

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2023-09-30 08:04:14 +03:00
Anas Nashif
8dc2746c0e kernel: z_handle_obj_poll_events is internal not kernel.h material
This internal kernel API is misplaced in a public kernel header. Just
make it available to the code using it in the kernel.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-09-17 07:59:30 -04:00
Peter Mitsis
c7255cf374 kernel: Remove references to _EXPIRED
The _EXPIRED macro is no longer necessary. It is a relic of an older
timeout processing algorithm from several years ago.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2023-09-15 19:16:20 -04:00
Anas Nashif
fcf50ed6e7 kernel: move timeout_q.h to kernel/include
This is a private kernel header with private kernel APIs, it should not
be exposed in the public zephyr include directory.

Once sample remains to be fixed (metairq_dispatch), which currently uses
private APIs from that header, it should not be the case.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-09-12 12:55:36 -04:00
Anas Nashif
8634c3b444 kernel: move wait_q.h header to be internal
This header does not expose any public APIs, so move it under
kernel/include and change files including it.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-09-12 12:55:36 -04:00
Peter Mitsis
bd5839ec9e kernel: Fix wrap-around check in kernel/mmu.h
Fixes the buffer wrap-around check so that it will not be ignored
by the GNU C compiler.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2023-08-01 09:51:33 +02:00
Florian Grandel
e256b7d244 kernel: spinlock: LOCKED -> K_SPINLOCK
Let the kernel use the new K_SPINLOCK macro and remove the alias.

Signed-off-by: Florian Grandel <fgrandel@code-for-humans.de>
2023-07-10 09:27:21 +02:00
Florian Grandel
816d09c453 kernel: spinlock: expose LOCKED macro as public API
While the LOCKED pattern is universally useful it can be misused. This
change therefore exposes the LOCKED pattern with extensive usage
documentation to reduce the risk of abuse or unintended deadlock.

Signed-off-by: Florian Grandel <fgrandel@code-for-humans.de>
2023-07-10 09:27:21 +02:00
Evgeniy Paltsev
16b8191be0 SMP: fix build failure if SMP=y and SYS_CLOCK_EXISTS=n
Fix build failure for the SMP configurations without sysclock.

Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
2023-06-22 06:17:27 -04:00
Gerard Marull-Paretas
48b201cc53 device: make device dependencies optional
Device dependencies are not always required, so make them optional via
CONFIG_DEVICE_DEPS. When enabled, the gen_device_deps script will run so
that dependencies are collected and part of the final image. Related
APIs will be also made available. Since device dependencies are used in
just a few places (power domains), disable the feature by default. When
not enabled, a second linking pass will not be required.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-06-21 09:32:05 +02:00
Gerard Marull-Paretas
5982d83e2a device: s/struct device.handles/struct device.deps
Rename struct device `handles` member to `deps`, in line with previous
renamings in the device API.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-06-21 09:32:05 +02:00
Andy Ross
c3046f417a kernel/sched: Use new barrier and spin APIs
The switch_handle field in the thread struct is used as an atomic flag
between CPUs in SMP, and has been known for a long time to technically
require memory barriers for correct operation.  We have an API for
that now, so put them in:

* The code immediately before arch_switch() needs a write barrier to
  ensure that thread state written by the scheduler is seen to happen
  before the outgoing thread is flagged with a valid switch handle.

* The loop in z_sched_switch_spin() needs a read barrier at the end,
  to make sure the calling context doesn't load state from before the
  other CPU stored the switch handle.

Also, that same spot in switch_spin was spinning with interrupts held,
which means it needs a call to arch_spin_relax() to avoid a FPU state
deadlock on some architectures.

Signed-off-by: Andy Ross <andyross@google.com>
2023-05-26 17:09:35 -04:00
Andy Ross
b89e427bd6 kernel/sched: Rename/redocument wait_for_switch() -> z_sched_switch_spin()
This trick turns out also to be needed by the abort/join code.
Promote it to a more formal-looking internal API and clean up the
documentation to (hopefully) clarify the exact behavior and better
explain the need.

This is one of the more... enchanted bits of the scheduler, and while
the trick is IMHO pretty clean, it remains a big SMP footgun.

Signed-off-by: Andy Ross <andyross@google.com>
2023-05-26 17:09:35 -04:00
Flavio Ceolin
df4be07b26 kernel: mmu: Fix Xtensa memory alignment issue
z_page_frame can't be packed on Xtensa due memory alignment
constraints. When this is struct is packed it is 5 bytes long it will
cause an memory alignment problem on Xtensa.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2023-05-23 08:54:29 +02:00
Peter Mitsis
e1aa1396d8 kernel: Remove unused absolute symbols
Removes unused absolute symbols that are defined via the
GEN_ABSOLUTE_SYM() macro in the kernel directory.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2023-04-18 10:51:28 -04:00
Aastha Grover
5537776898 kernel: Add z_sched_wake_thread API
This API wakes up a given thread and is also called from
z_thread_timeout()

Signed-off-by: Aastha Grover <aastha.grover@intel.com>
2023-03-09 09:22:21 +01:00
Andy Ross
f3afd5a4c9 kernel/sched: Use kernel timeouts for timeslice expirations
Rework the fragile and ad-hoc computation of timeslice expirations
into per-CPU struct _timeout objects with regular callbacks.  The
expiration callbacks themselves simply set a per-cpu flag (they might
run on any CPU), which gets checked at the end of the timer ISR on
every CPU.

This simplifies logic and removes a bunch of code.  It also fixes at
least three bugs:

1. As @npitre discovered: On SMP, the number of ticks announced on any
given CPU is going to be a subset of all expired ticks.  This broke
the accounting of timeslice ticks, and effectively meant that
timeslicing only worked on SMP on systems where one CPU could hog all
the announcements, and only on that CPU.

2. The bootstrap path to arm the timer driver after setting the first
timeout in an empty list couldn't take into account
sys_clock_elapsed() ticks, as it didn't know whether it was being
called underneath an existing announce loop.  Now this code is no
longer responsible for knowing anything about time slicing at all.

3. Also on SMP, there was a case where two CPUs timeslicing
simultaneously could stomp on each others' timeouts in
z_set_timeout_expiry(), as neither had a way of knowing what the
other's state was.  CPUs could miss their own expiration and have to
wait for the slice expiration on the other CPU.  Now, timeouts are
global objects with simple expiration times, and there's no need for
that function at all.

Signed-off-by: Andy Ross <andyross@google.com>
2023-03-09 09:21:12 +01:00
Peter Mitsis
a06f9ac418 kernel: Remove unused derived offset symbols
Some of the offset symbols that are derived from the macro
GEN_OFFSET_SYM() are not used anywhere in the Zephyr codebase.
Remove them as part of a cleanup effort.

Instances of an associated GEN_OFFSET_SYM() have also been
removed when the resulting macro is no longer referenced.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2023-02-22 16:01:32 +01:00
Peter Mitsis
fa23eb49b3 kernel: Remove unused generated offset symbols
Some of the offset symbols generated via the macro GEN_OFFSET_SYM()
are not used anywhere in the Zephyr codebase. Remove them as part of
a cleanup effort.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2023-02-22 16:01:32 +01:00
Peter Mitsis
ca58339e16 kernel: Add routine to walk a wait queue
Adds a routine to safely walk a specified wait queue and invoke a
custom callback function on each waiting thread.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2023-02-11 06:45:13 +09:00
Kumar Gala
c778eb2a56 smp: Move arrays to use CONFIG_MP_MAX_NUM_CPUS
Move to use CONFIG_MP_MAX_NUM_CPUS for array size declarations instead
of CONFIG_MP_NUM_CPUS.

Signed-off-by: Kumar Gala <kumar.gala@intel.com>
2022-10-17 14:40:12 +09:00
Nicolas Pitre
1f362a81f1 riscv: fix crash resulting from touching the initial stack's guard area
The interrupt stack is used as the system stack during kernel
initialization while IRQs are not yet enabled. The sp register is
set to z_interrupt_stacks + CONFIG_ISR_STACK_SIZE.

CONFIG_ISR_STACK_SIZE only represents the desired usable stack size.
This does not take into account the added guard area. Result is a stack
whose pointer is much closer to the trigger zone than expected when
CONFIG_PMP_STACK_GUARD=y, and the SMP configuration in particular pushes
it over the edge during many CI test cases.

Worse: during early init we're not quite ready to handle exceptions
yet and complete havoc ensues with no meaningful debugging output.

Make sure the early assembly code locates the actual top of the stack
by generating a constant with its true size.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-09-28 07:53:56 +00:00
Hu Zhenyu
57487622f5 kernel: Init the base.slice_ticks for dummy_thread
Fixes #46324
Set dummy_thread->base.slice_ticks to 0 when
CONFIG_TIMESLICE_PER_THREAD is set. To avoid
_current_cpu->slice_ticks be a big number.

Signed-off-by: Hu Zhenyu <zhenyu.hu@intel.com>
2022-08-04 19:44:24 -04:00
Simon Hein
02cfbfea51 kernel: comply to coding guidelines MISRA C:2012 Rule 14.4
MISRA C:2012 Rule 14.4 (The controlling expression of an if statement
and the controlling expression of an iteration-statement shall have
essentially Boolean type.)

Use `bool' instead of `int' to represent Boolean values.
Use `do { ... } while (false)' instead of `do { ... } while (0)'.
Use comparisons with zero instead of implicitly testing integers.

This commit is a subset of the original commit:
5d02614e34a86b549c7707d3d9f0984bc3a5f22a

Signed-off-by: Simon Hein <SHein@baumer.com>
2022-07-21 06:16:16 -04:00
Stephanos Ioannidis
360f810704 kernel: Migrate to K_KERNEL_PINNED_STACK_ARRAY_DECLARE
This commit updates all deprecated `K_KERNEL_PINNED_STACK_ARRAY_EXTERN`
macro usages to use the `K_KERNEL_PINNED_STACK_ARRAY_DECLARE` macro
instead.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2022-06-20 10:25:52 +02:00
Gerard Marull-Paretas
cffefc818d kernel: migrate includes to <zephyr/...>
In order to bring consistency in-tree, migrate all kernel code to the
new prefix <zephyr/...>. Note that the conversion has been scripted,
refer to zephyrproject-rtos#45388 for more details.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2022-05-09 09:26:20 +02:00
Bradley Bolen
88ba97fea4 arch: arm: aarch32: cortex_a_r: Add shared FPU support
This adds lazy floating point context switching.  On svc/irq entrance,
the VFP is disabled and a pointer to the exception stack frame is saved
away.  If the esf pointer is still valid on exception exit, then no
other context used the VFP so the context is still valid and nothing
needs to be restored.  If the esf pointer is NULL on exception exit,
then some other context used the VFP and the floating point context is
restored from the esf.

The undefined instruction handler is responsible for saving away the
floating point context if needed.  If the handler is in the first
irq/svc context and the current thread uses the VFP, then the float
context needs to be saved.  Also, if the handler is in a nested context
and the previous context was using the FVP, save the float context.

Signed-off-by: Bradley Bolen <bbolen@lexmark.com>
2022-05-05 12:03:27 +09:00
Flavio Ceolin
d02a1e9879 pm: Only resize power domains
Instead of resizing all devices handles, we just resize devices that are
power domains. This means that a power domain has to be declared as
compatbile with "power-domain" in device tree node.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2022-04-18 17:25:01 -07:00
Nazar Kazakov
f483b1bc4c everywhere: fix typos
Fix a lot of typos

Signed-off-by: Nazar Kazakov <nazar.kazakov.work@gmail.com>
2022-03-18 13:24:08 -04:00
Andy Ross
3e696896bf kernel: Add "per thread" timeslice mechanism
Zephyr's timeslice implementation has always been somewhat primitive.
You get a global timeslice that applies broadly to the whole bottom of
the priority space, with no ability (beyond that one priority
threshold) to tune it to work on certain threads, etc...

This adds an (optionally configurable) API that allows timeslicing to
be controlled on a per-thread basis: any thread at any priority can be
set to timeslice, for a configurable per-thread slice time, and at the
end of its slice a callback can be provided that can take action.
This allows the application to implement things like responsiveness
heuristics, "fair" scheduling algorithms, etc... without requiring
that facility in the core kernel.

Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2022-03-09 13:49:44 -05:00
Nicolas Pitre
678b76e4b0 kernel/init.c: allow for memset/memcpy alternatives during early boot
Zeroing the BSS and copying data to RAM with regular memset/memcpy may
cause problems when those functions are assuming a fully initialized
system for their optimizations to work e.g. some instructions require
an active MMU, but turning the MMU on needs the .bss section to be
cleared first, etc.

Commit c5b898743a ("aarch64: Fix alignment fault on z_bss_zero()")
provides a detailed explanation of such a case.

Replacing z_bss_zero() with an architecture specific one is problematic
as the former may see new sections added to it that would be missed by
the later. The same reasoning goes for z_data_copy().

Let's make maintenance much easier by providing weak versions of
memset/memcpy that can be overridden by architecture-specific safe
versions when needed.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-21 21:00:12 -05:00
Krzysztof Chruscinski
1da97e1374 kernel: Add function for calculating stack usage
Extracting stack usage calculation from k_thread_stack_space_get to
z_stack_space_get so it can be used also for interrupt stack.

Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
2022-02-21 20:57:17 -05:00
Daniel Leung
88cfd3343d kernel: arch: no need for #ifdef MMU in header
There is no need to use conditional compilation for the function
prototypes in the kernel architecture header file. So remove it.
Added bouns is that these functions can appear in documentation
without explicitly enabled in pre-defines during doc build.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2022-01-18 19:18:30 -05:00
Peter Mitsis
4eb1dd02cc kernel: extend CPU runtime stats
Extends the CPU usage runtime stats to track current, total, peak
and average usage (as bounded by the scheduling of the idle thread).
This permits a developer to obtain more system information if desired
to tune the system.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2022-01-10 10:38:06 -05:00
Peter Mitsis
5deaffb2ee kernel: update z_sched_thread_usage()
This commit does two things to the z_sched_thread_usage(). First,
it updates the API so that it accepts a pointer to the runtime
stats instead of simply returning the usage cycles. This gives it
the flexibility to retrieve additional statistics in the future.

Second, the runtime stats are only updated if the specified thread
is the current thread running on the current core.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2022-01-10 10:38:06 -05:00
Berend Ozceri
b208e5811e kernel/swap: Initialize dummy thread's resource pool
The resource pool of the short-lived dummy thread "stub" may be
inherited by other threads created during system initialization. This
commit initializes this resource pool to NULL or the system pool to
ensure that a well-defined resource pool propagates to other threads
that inherit it from the dummy thread.

Fixes #41482.

Signed-off-by: Berend Ozceri <berend@recogni.com>
2022-01-06 11:57:18 -05:00
Daniel Leung
650a629b08 debug: gdbstub: remove start argument from z_gdb_main_loop()
Storing the state where this is the first GDB break can be done
in the main GDB stub code. There is no need to store the state
in architecture layer.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2021-11-30 15:24:00 -05:00
Flavio Ceolin
623ed5ae29 pm: Remove invalid comments
Remove comments referencing an old function / behavior.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2021-11-29 19:37:55 -05:00
Krzysztof Chruscinski
c0808e3f59 logging: Minimal mode configuration cleanup
Remove LOG_MINIMAL kconfig option which was confusing
since LOG_MODE_MINIMAL existed. LOG_MINIMAL was used to
force minimal mode but because of invalid dependencies
it was leading to issues.

Refactored code to use LOG_MODE_MINIMAL everywhere and
renamed LOG_MINIMAL to LOG_DEFAULT_MINIMAL which has impact
on defualt logging mode (which still can be later changed
in conf file or in menuconfig).

Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
2021-11-20 11:58:40 -05:00
Andy Ross
f169c5bc13 kernel: Swap RUNTIME_STATS implementation
Clean up RUNTIME_STATS to separate the API from the individual data
backends.  Use the SCHED_THREAD_USAGE tracking instead of the original
for execution_cycles.  Move the kconfig for that into the runtime
stats menu, since it's part of the family now.

Also remove a lot of needless #if's around the declarations.  Unused
structs and uncalled functions don't need to be explicitly hidden.  An
attempt to access a non-existent field (e.g. "execution_cycles" if
that isn't configured) provides all the build time validation we need.

Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2021-11-08 21:32:20 -05:00