Commit Graph

3195 Commits

Author SHA1 Message Date
Guennadi Liakhovetski
037cb87276 llext: export spinlock debugging symbols to extensions
With spinlock debugging enabled LLEXTs need additional symbols
exported by the kernel.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
2025-02-11 10:11:31 +01:00
Andy Ross
f6239c52ae kernel/sched: Panic after aborting essential thread, not before
The essential thread check and panic happens at the top of
k_thread_abort().  This is arguably a performance bug: the system is
going to blow up anyway no matter where we put the test, we shouldn't
add instructions to the path taken by systems that DON'T blow up.

But really it's more of a testability/robustness glitch: if you have a
fatal error handler that wants to catch this panic (say, a test using
ztest_set_fault_valid()), then the current code will panic and
early-exit BEFORE THE THREAD IS DEAD.  And so it won't actually die,
and will continue on causing mayhem when presumably the handler code
expected it to have been aborted.

It's sort of an unanswerable question as to what the "right" behavior
is here (the system is, after all, supposed to have panicked!).  But
this seems preferable for definable practical reasons.

Kill the thread, then panic.  Unless it's _current, in which case
panic as late as possible for maximum coverage of the abort path.

Fixes: #84460

Signed-off-by: Andy Ross <andyross@google.com>
2025-02-10 22:26:10 +01:00
Ryan McClelland
37e4af63a9 kernel: thread: fix warning of always false
K_KERNEL_STACK_RESERVED can be 0 which can give a warning with
-Wtype-limits. Only perform the check if ARCH_KERNEL_STACK_RESERVED
is set. Also remove the the unncessary sets in arch.h where it's
manually set to 0, it defaults to 0 anyways.

Signed-off-by: Ryan McClelland <ryanmcclelland@meta.com>
2025-02-10 22:25:32 +01:00
Peter Mitsis
e55ac3ef65 kernel: Improve ordering in SMP k_thread_suspend()
The routine k_thread_suspend() has a fast path for non-SMP when
suspending the current thread. When SMP is enabled, it is expected
that the compiler drop the entire fast path checks because the whole
expression would always evaluate to false. However, the compiler has
been observed to only drop whole fast path check when the
"!IS_ENABLED(CONFIG_SMP)" condition appears at the beginning of the
fast path check.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-02-07 02:23:45 +01:00
Peter Mitsis
11083fc049 kernel: Initialize kheap spinlock
Initializes the kheap spinlock when the kheap is initialized.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-02-04 09:19:09 +01:00
Peter Mitsis
c63b42d478 kernel: Fix k_wakeup() exit paths
z_reschedule() already has a check to determine if it is called from
the context of an ISR--no need to duplicate it in k_wakeup().
Furthermore, if the target thread is not sleeping, there is no need
to reschedule and we can do a fast return.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-02-03 19:51:20 +01:00
Peter Mitsis
48939cb44b kernel: msgq: Allow for fast returns
1. Fixes a performance issue in k_msgq_put() to allow for a fast return
path when handling a poll event does nothing.

2. Allows for a fast return path in k_msgq_purge() when no threads were
awakened.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-02-03 19:51:20 +01:00
Peter Mitsis
790d9dad59 kernel: Allow for fast return on k_sem_reset()
Do a fast return when resetting a semaphore if no threads were awakened.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-02-03 19:51:20 +01:00
Peter Mitsis
6656c02fe4 kernel: queue: Allow for fast return
Updates the queue code to both allow for a fast return path in a
few routines when the operation did not wake or signal another
thread.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-02-03 19:51:20 +01:00
Peter Mitsis
c6ed03f66a kernel: futex: Allow for fast return on k_futex_wake()
Updates k_futex_wake() to allow for a fast return path when no threads
were waiting.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-02-03 19:51:20 +01:00
Peter Mitsis
f5c343fc93 kernel: condvar: broadcast does not always need reschedule
When doing a condition variable broadcast, a full reschedule
is only needed if at least one thread was awakened.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-02-03 19:51:20 +01:00
Peter Mitsis
6e3f57118f kernel: Propagate z_handle_obj_poll_events() return
Propagates the return value from z_handle_obj_poll_events()
within the message queue, pipes, queue and semaphore objects.
This allows the kernel object code to determine whether it
needs to perform a full reschedule, or if it can perform a
more optimized exit strategy.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-02-03 19:51:20 +01:00
Peter Mitsis
70520e7a4d kernel: Change z_handle_obj_poll_events() return type
Changes the return type of z_handle_obj_poll_events() so that it
returns true if there were polling events to handle (false
otherwise).

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-02-03 19:51:20 +01:00
Björn Bergman
b45e970388 kernel: fix void* to k_thread_entry_t conversion
Fix a void* to k_thread_entry_t conversion (that is silent in GCC but
not so in some other tools) in _is_valid_prio()

Signed-off-by: Björn Bergman <bjorn.bergman@iar.com>
2025-01-29 12:51:41 +01:00
Peter Mitsis
90e2498982 kernel: Add timeout_lock usage notes
Adds a note about the timeout_lock to aid future developers
in following the rules to help prevent deadlocks involving the
timeout and scheduler spinlocks.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-01-28 18:14:22 +01:00
Peter Mitsis
995ad43851 kernel: Streamline z_is_thread_ready()
The check for an active timeout in z_is_thread_ready() was originally
added to cover the case of a sleeping thread. However, since there is
now a bit in the thread state that indicates if the thread is sleeping
we can drop that superfluous check.

Making this change necessitates moving k_wakeup()'s call to
z_abort_thread_timeout() so that it is within the locked
_sched_spinlock section to ensure that we do not end up with
a stray thread timeout in the timeout list.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-01-28 18:14:22 +01:00
Erik Tamlin
07f5a6fa18 k_pipe: fix trace point for blocking writes
Fix the trace point in k_pipe wait_for.

Signed-off-by: Erik Tamlin <erik.tamlin@percepio.com>
2025-01-28 14:13:14 +01:00
Peter Mitsis
568b63859a kernel: Remove clearing of swap_ok in do_swap()
Removes an unnecessary clearing of the current CPU's swap_ok field
in do_swap() as that clearing is already done at the end of next_up()
which was just called by z_swap_next_thread() a little earlier.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-01-28 07:57:20 +01:00
Peter Mitsis
bfe0b74aad kernel: Do not mark thread as queued in k_yield()
SMP does not need to mark the current thread as queued in
k_yield() as that will naturally get done in do_swap().

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-01-28 07:57:20 +01:00
Kalle Kietäväinen
a2eb78c03b kernel: sched: Fix meta-IRQ preemption tracking for the idle thread
When the PM subsystem is enabled, the idle thread locks the scheduler for
the duration the system is suspended. If a meta-IRQ preempts the idle
thread in this state, the idle thread is tracked in `metairq_preempted`.
However, when returning from the preemption, the idle thread is not removed
from `metairq_preempted`, unlike all the other threads. As a result, the
scheduler keeps running the idle thread even if there are higher priority
threads ready to run.

This change treats the idle thread the same way as all other threads when
returning from a meta-IRQ preemption.

Fixes #64705

Signed-off-by: Kalle Kietäväinen <kalle.kietavainen@silabs.com>
2025-01-27 13:26:20 +01:00
Nicolas Pitre
f93c1ed291 kernel/pipe: squash compiler warning
The compiler complains that:

```
zephyr/kernel/include/kernel_internal.h:121:29:
error: 'reader' may be used uninitialized [-Werror=maybe-uninitialized]
  121 |         thread->swap_retval = value;
      |         ~~~~~~~~~~~~~~~~~~~~^~~~~~~
zephyr/kernel/pipe.c: In function 'copy_to_pending_readers':
zephyr/kernel/pipe.c:92:26: note: 'reader' was declared here
   92 |         struct k_thread *reader;
      |                          ^~~~~~
```

The static analyzer fails to see through the `LOCK_SCHED_SPINLOCK`
construct that the `reader` pointer is always initialized.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2025-01-22 05:39:01 +01:00
Nicolas Pitre
0c0644bd68 kernel/pipe: disable direct-to-readers copy with CONFIG_KERNEL_COHERENCE
Systems that enabled this option don't have their stacks in coherent
memory. Given our pipe_buf_spec is stored on the stack, and readers may
also have their destination buffer on their stack too, it is not worth
going to the trouble of supporting direct-to-readers copy with them.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2025-01-21 19:44:57 +01:00
Nicolas Pitre
3da90f9d49 kernel/pipe: add missing calls to z_reschedule()
We are waking up threads but failed to let them run if they are
higher priority. Add missing calls to z_reschedule().

Also wake up all pending writers as we don't know how many there might
be. It is more efficient to wake them all when the ring buffer is full
before reading from it rather than waking them one by one whenever there is
more room in it.

Thanks to Peter Mitsis for noticing those issues.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2025-01-21 19:44:57 +01:00
Nicolas Pitre
29ae9e3435 kernel/pipe: implement direct-to-pending-readers data copy
If there are pending readers, it is best to perform a single data copy
directly into their final destination buffer rather than doing one copy
into the ring buffer just to immediately copy the same data out of it.

Incidentally, this allows for supporting pipes with no ring buffer at all.

The pipe implementation being deprecated has a similar capability so better
have it here too.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2025-01-21 19:44:57 +01:00
Nicolas Pitre
4af80d72bc kernel/pipe: remove extra hardware timer query on thread wakeup
Dispense with the call to sys_timepoint_expired() by leveraging
swap_retval to distinguish between notifications and timeouts when
z_pend_curr() returns.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2025-01-21 19:44:57 +01:00
Nicolas Pitre
46617644e7 kernel/pipe: code flow cleanup
Simplify the logic, avoid repeated conditionals, avoid superfluous
scheduler calls, make the code more efficient and easier to read.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2025-01-21 19:44:57 +01:00
David Brown
19a376aa88 kernel: poll: Allow 0 events for submitted work
Change:
    commit cc6317d7ac
    Author: Jukka Rissanen <jukka.rissanen@linux.intel.com>
    Date:   Fri Nov 1 14:03:32 2019 +0200

        kernel: poll: Allow 0 event input

Allows `k_poll` to be user with 0 events, which is useful for allowing just
a sleep without having to create artificial events.

Allow the same for `k_work_submit_to_queue()` and `k_work_submit()`.

Signed-off-by: David Brown <david.brown@linaro.org>
2025-01-21 19:44:07 +01:00
Måns Ansgariusson
efcc734b79 tracing: k_pipe: Add tracing support for reworked k_pipe API
Add tracing support for the reworked k_pipe API.

Signed-off-by: Måns Ansgariusson <Mansgariusson@gmail.com>
2025-01-17 19:43:44 +01:00
Måns Ansgariusson
6782c5381d tests: kernel: pipe: Add new test cases for pipe API rework
This commit adds new test cases for the pipe API rework.
* basic.c: Sanity check for pipe operations.
* concurrency.c: Test pipe operations with multiple threads.
* stress.c: Test pipe operations under stress conditions.

And moves the old pipe test cases to the deprecated folder.

Signed-off-by: Måns Ansgariusson <Mansgariusson@gmail.com>
2025-01-17 19:43:44 +01:00
Måns Ansgariusson
84584c579e kernel: k_pipe: Add object_core support for the k_pipe rework
This patch adds object_core support for the k_pipe api rework.

Signed-off-by: Måns Ansgariusson <Mansgariusson@gmail.com>
2025-01-17 19:43:44 +01:00
Måns Ansgariusson
a9ab8cb779 feat: enable polling support for k_pipe interface
This commit adds polling support to the newly rewritten k_pipe interface.
Changes include:

* Removed ifdef CONFIG_POLL from kernel/poll.c to let both implementations
  coexist.
* Added the needed datastructures to the new k_pipe struct.
* k_pipe_write(..) now notifies the poll subsystem that data is available.

Signed-off-by: Måns Ansgariusson <Mansgariusson@gmail.com>
2025-01-17 19:43:44 +01:00
Måns Ansgariusson
c8a2a080ac kernel: Rewrite k_pipe_* API
The `k_pipe_*` API has been reworked to provide a more consistent and
intuitive interface. The new API aims to provide a simple to use byte
stream interface that is more in line with the POSIX pipe API.
The previous API has been deprecated and will be removed in a future
release.

Signed-off-by: Måns Ansgariusson <Mansgariusson@gmail.com>
2025-01-17 19:43:44 +01:00
Nicolas Pitre
44d5d8aef2 kernel: uninline z_dummy_thread_init()
This function is getting quite involved and it also gained more callers
lately. This is not performance critical so Uninline it to save on
binary size.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2025-01-15 07:19:40 +01:00
Tom Hughes
0ec126c6d1 kernel: Add missing marshalling header for k_reschedule
Without this header, compiling the kernel.poll test with
-Werror=unused-function fails.

Signed-off-by: Tom Hughes <tomhughes@chromium.org>
2025-01-13 20:24:16 +01:00
Nicolas Pitre
bc6eded597 kernel: mark z_smp_current_get() with the const attribute
Repeated references to _current won't produce a different result as the
executing thread instance is always the same. Use the const attribute to
let the compiler know it may reuse a previously obtained value. This offset
the penalty for moving z_smp_current_get() out of line and provides yet
more binary size reduction.

This change is isolated in its own commit to ease bisecting in case some
unexpected misbehavior is eventually observed.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2025-01-10 07:49:08 +01:00
Nicolas Pitre
7a3124d866 kernel: move current thread pointer management to core code
Define the generic _current directly and get rid of the generic
arch_current_get().

The SMP default implementation is now known as z_smp_current_get().
It is no longer inlined which saves significant binary size (about 10%
for some random test case I checked).

Introduce z_current_thread_set() and use it in place of
arch_current_thread_set() for updating the current thread pointer
given this is not necessarily an architecture specific operation.
The architecture specific optimization, when enabled, should only care
about its own things and not have to also update the generic
_current_cpu->current copy.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2025-01-10 07:49:08 +01:00
Nicolas Pitre
46aa6717ff Revert "arch: deprecate _current"
Mostly a revert of commit b1def7145f ("arch: deprecate `_current`").

This commit was part of PR #80716 whose initial purpose was about providing
an architecture specific optimization for _current. The actual deprecation
was sneaked in later on without proper discussion.

The Zephyr core always used _current before and that was fine. It is quite
prevalent as well and the alternative is proving rather verbose.
Furthermore, as a concept, the "current thread" is not something that is
necessarily architecture specific. Therefore the primary abstraction
should not carry the arch_ prefix.

Hence this revert.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2025-01-10 07:49:08 +01:00
Peter Mitsis
bdb04dbfba kernel: Alter z_abort_thread_timeout() return type
No caller of the internal kernel routine z_abort_thread_timeout()
uses its return value anymore.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-01-09 04:04:36 +01:00
Peter Mitsis
f4f3b9378f kernel: Inline halt_thread() and z_thread_halt()
Inlining these routines helps to improve the
performance of k_thread_suspend()

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-01-07 18:24:09 +01:00
Peter Mitsis
d774594547 kernel: thread suspend/resume bail paths are unlikely
Gives a hint to the compiler that the bail-out paths in both
k_thread_suspend() and k_thread_resume() are unlikely events.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-01-07 18:24:09 +01:00
Peter Mitsis
af14e120a5 kernel: Simplify clear_halting() on UP systems
There is no need for clear_halting() to do anything on UP systems.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-01-07 18:24:09 +01:00
Peter Mitsis
85a9cffd0f kernel: cache priority queue index on UP multiq
Even though calculating the priority queue index in the priority
multiq is quick, caching it allows us to extract an extra 2% in
terms of performance as measured by the thread_metric cooperative
benchmark.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-01-07 18:24:09 +01:00
Peter Mitsis
ea6adb6726 kernel: Add custom scheduler yield routines
Adds customized yield implementations based upon the selected
scheduler (dumb, multiq or scalable). Although each follows the
same broad outline, some of them allow for additional tweaking
to extract maximal performance. For example, the multiq variant
improves the performance of k_yield() by about 20%.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-01-07 18:24:09 +01:00
Peter Mitsis
30f667bceb kernel: Add routines for _THREAD_QUEUED bit
Adds routines for setting and clearing the _THREAD_QUEUED
thread_state bit.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-01-07 18:24:09 +01:00
Peter Mitsis
472c71d3f4 kernel: Add method to dequeue from a dlist
Dequeuing from a doubly linked list is similar to removing an item
except that it does not re-initialize the dequeued node.

This comes in handy when sorting a doubly linked list (where the
node gets removed and re-added). In that circumstance, re-initializing
the node is required. Furthermore, the compiler does not always
'understand' this. Thus, when performance is critical, dequeuing
may be preferred to removing.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-01-07 18:24:09 +01:00
Peter Mitsis
d1c2fc0667 kernel: inline z_sched_prio_cmp()
Inlines z_sched_prio_cmp() to get better performance.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-01-07 18:24:09 +01:00
Peter Mitsis
c6693bfdae kernel: Clean up priority_q.h
Minor cleanups include ...
 1. Eliminating unnecessary if-defs and forward declarations
 2. Co-locating routines of the same queue type

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-01-07 18:24:09 +01:00
Dong Wang
dd5f11cb04 kernel: timeout: ensure next timeout is set when aborting the first timeout
This ensures that the system clock is correctly updated when the first
timeout is aborted, preventing unexpected early wake-up by the system clock
programmed previously.

Signed-off-by: Dong Wang <dong.d.wang@intel.com>
2025-01-06 22:36:07 +01:00
Simone Orru
37fd711a45 kernel: Add k_heap_calloc
Add the k_heap_calloc function. The
equivalent of the standard calloc for
the Zephyr kernel.

Signed-off-by: Simone Orru <simone.orru@secomind.com>
2024-12-30 17:32:03 +01:00
Yong Cong Sin
2f2dd9407c kernel/swap: remove redundant ARG_UNUSED from do_swap()
The `lock` arg is used multiple times in the function, making the
`ARG_UNUSED(lock);` redundant, remove it.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
2024-12-24 13:53:49 +01:00
Peter Mitsis
669f2c489a kernel: Add k_reschedule()
The routine k_reschedule() allows an application to manually force
a schedule point. Although similar to k_yield(), it has different
properties. The most significant difference is that k_yield() if
invoked from a cooperative thread will voluntarily give up execution
control to the next thread of equal or higher priority while
k_reschedule() will not.

Applications that play with EDF deadlines via k_thread_deadline_set()
may need to use k_reschedule() to force a reschedule.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-12-24 04:22:51 +01:00
Flavio Ceolin
3e75c03cb2 security: Add default stack protection level
STACK_CANARIES was enabling canaries in all functions using the compiler
flag -fstack-protector-all. This became confuse with the addition of the
options STRONG and EXPLICIT.

This commit adds the missing option (default level) and disambiguous the
options mapping them close to the compiler flags.

Now we have the following options:

STACK_CANARIES            -> fstack-protector
STACK_CANARIES_STRONG     -> fstack-protector-strong
STACK_CANARIES_ALL        -> fstack-protector-all
STACK_CANARIES_EXPLICIT   -> fstack-protector-explicit

Note that from now on STACK_CANARIES_ALL is the symbol that adds canaries
for all functions.

Signed-off-by: Flavio Ceolin <flavio.ceolin@gmail.com>
2024-12-20 12:37:20 +01:00
Flavio Ceolin
0236f7c9aa security: Add option for explicit stack canaries
Add option to enable stack canaries only when explicitely
declared. It adds a new function attribute, __stack_protect, that
can be used to enable stack protection in a function.

Signed-off-by: Flavio Ceolin <flavio.ceolin@gmail.com>
2024-12-20 12:37:20 +01:00
Flavio Ceolin
82ace41da4 security: Additional option for stack canaries
Previously, when stack canaries were enabled, Zephyr applied this
protection to all functions. This commit introduces a new option that
allows stack canary protection to be applied selectively to specific
functions based on certain criteria.

Signed-off-by: Flavio Ceolin <flavio.ceolin@gmail.com>
2024-12-20 12:37:20 +01:00
Peter Mitsis
35435928c2 kernel: Decouple sleep from suspend
Sleeping and suspended are now orthogonal states. That is, a thread
may be both sleeping and suspended and the two do not interact. One
repercussion of this is that suspending a thread will no longer
abort its timeout.

Threads are now created in the 'sleeping' state instead of a
'suspended' state. This dovetails nicely with the start delay that
can be given to a newly created thread--it is as though the very
first operation that a thread with a start delay is a sleep.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-12-18 18:17:03 +01:00
Peter Mitsis
5710e034e7 kernel: Introduce _THREAD_SLEEPING state bit
At the present time, Zephyr does has overlap between sleeping and
suspending. Not only should sleeping and suspended be orthogonal
states, but we should ensure users always employ the correct API.
For example, to wake a sleeping thread, k_wakeup() should be used,
and to resume a suspended thread, k_thread_resume() should be used.
However, at the present time k_thread_resume() can be used on a
thread that called k_sleep(K_FOREVER). Sleeping should have nothing
to do with suspension.

This commit introduces the new _THREAD_SLEEPING thread state along
with some prep-work to facilitate the decoupling of the sleeping and
suspended thread states.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-12-18 18:17:03 +01:00
TaiJu Wu
fc47180850 kernel: k_msgq_put should trigger reschedule
Make `k_msgq_put` as scheduler point to resolve
high priority thread can not preempt low priority
thread.

Signed-off-by: TaiJu Wu <tjwu1217@gmail.com>
2024-12-14 01:04:43 +01:00
Peter Mitsis
815d669971 kernel: Optimize k_mem_slab_free()
Optimizes k_mem_slab_free() for the case where the slab's memory is
not exhausted.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-12-14 01:03:28 +01:00
Peter Mitsis
869acdbbab kernel: Add CONFIG_MEM_SLAB_POINTER_VALIDATE
Makes the validation of both allocated memory slab pointer and the
memory slab pointer to free configurable.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-12-14 01:03:28 +01:00
Måns Ansgariusson
7545e8d582 tracing: kernel: Add support for stopping workqueues
Adds tracing support for stopping workqueues.

Signed-off-by: Måns Ansgariusson <Mansgariusson@gmail.com>
2024-12-12 11:07:39 +01:00
Måns Ansgariusson
82a9bc4589 kernel: Add support for stopping workqueues
This patch adds support for stopping workqueues. This is useful for freeing
resources from workqueues when subsystems/modules is deactivated or
cleaning up the system between tests in ztest to reach a fully normalized
state.

The patch adds a new function k_work_queue_stop() that releases the
workqueues thread and stack when a workqueue is unwanted.
k_work_queue_stop(...) should be viewed as a counterpart to
k_work_queue_start(...).

This would allow to:
k_work_queue_start(...);

k_work_drain(..., true);
k_work_queue_stop(...);

Signed-off-by: Måns Ansgariusson <Mansgariusson@gmail.com>
2024-12-12 11:07:39 +01:00
Peter Mitsis
909ff45f0c kernel: arch: move arch_swap() declaration
Moves the arch_swap() declaration out of kernel_arch_interface.h
and into the various architectures' kernel_arch_func.h. This
permits the arch_swap() to be inlined on ARM, but extern'd on
the other architectures that still implement arch_swap().

Inlining this function on ARM has shown at least a +5% performance
boost according to the thread_metric benchmark on the disco_l475_iot1
board.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-12-11 21:33:49 +01:00
Chris Friedt
95896bf17c kernel: dynamic: simplify dynamic stack allocation
Merge the functions z_thread_stack_alloc_dyn() and
stack_alloc_dyn(), simplifying the flow for dynamic thread stack
allocation.

Signed-off-by: Chris Friedt <cfriedt@tenstorrent.com>
2024-12-11 21:33:00 +01:00
Akaiwa Wataru
feefb7dd35 kernel/sched: correct k_sleep() return value calculation
Fix the issue: https://github.com/zephyrproject-rtos/zephyr/issues/79863

The expected_wakeup_ticks and sys_clock_tick_get_32() are uint32_t values,
and may wrap around individually.
If the expected_wakeup_ticks has a wraparound and sys_clock_tick_get_32()
doesn't, so expected_wakeup_ticks < sys_clock_tick_get_32(), the API return
value will be corrupted.

The API return value, that is the remaining time, should be calculated in
32bit-unsigned-integer manner, and any wraparound will be treated properly.

Signed-off-by: Akaiwa Wataru <akaiwa@sonas.co.jp>
2024-12-03 02:37:03 +01:00
Andy Ross
7cdf40541b kernel/sched: Eliminate PRESTART thread state
Traditionally threads have been initialized with a PRESTART flag set,
which gets cleared when the thread runs for the first time via either
its timeout or the k_thread_start() API.

But if you think about it, this is no different, semantically, than
SUSPENDED: the thread is prevented from running until the flag is
cleared.

So unify the two.  Start threads in the SUSPENDED state, point
everyone looking at the PRESTART bit to the SUSPENDED flag, and make
k_thread_start() be a synonym for k_thread_resume().

There is some mild code size savings from the eliminated duplication,
but the real win here is that we make space in the thread flags byte,
which had run out.

Signed-off-by: Andy Ross <andyross@google.com>
2024-11-27 10:38:05 -05:00
Andy Ross
6877b6d8e5 kernel/thread: Fix assumptions in k_thread_state_str()
This table just blindly assumed that the values of _THREAD_xxx state
bits wouldn't change.  That's dangerous.  (That's right, I tried to
change them and broke it.)

Signed-off-by: Andy Ross <andyross@google.com>
2024-11-27 10:38:05 -05:00
James Roy
ef8f037e78 device: Optimize device_get_binding search efficiency
Optimize 'device_get_binding' function search so that
when a matching device is found, it returns NULL if
it is not ready.

Signed-off-by: James Roy <rruuaanng@outlook.com>
2024-11-27 08:18:30 +01:00
Anas Nashif
f90ce01d4a kernel: sched: use arch_current_thread instead of _current
_current is deprecated, use arch_current_thread()
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-11-25 19:06:57 -05:00
Andy Ross
b3ff9ae82b kernel/sched: Optimize handling for suspend(_current)
k_thread_suspend() is an async API intended to stop any thread in any
state from any context.  Some apps just want to use it to "suspend
myself", which is a much (!) simpler operation.  Detect that specific
usage as a performance case.

Signed-off-by: Andy Ross <andyross@google.com>
2024-11-26 00:12:28 +01:00
Yong Cong Sin
b1def7145f arch: deprecate _current
`_current` is now functionally equals to `arch_curr_thread()`, remove
its usage in-tree and deprecate it instead of removing it outright,
as it has been with us since forever.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
2024-11-23 20:12:24 -05:00
Yong Cong Sin
d26c712258 arch: add new interfaces to set/get the current thread of current CPU
Add the following arch-specific APIs:
- arch_curr_thread()
- arch_set_curr_thread()

which allow SMP architectures to implement a faster "get current
thread pointer" than the default provided by the kernel. The 'set'
function is required for the 'get' to work, more on that later.

When `CONFIG_ARCH_HAS_CUSTOM_CURRENT_IMPL` is selected, calls to
`_current` & `k_sched_current_thread_query()` will be redirected to
`arch_curr_thread()`, which ideally should translate into a single
instruction read, avoiding the current
"lock > read CPU > read current thread > unlock" path in SMP
architectures and thus greatly improves the read performance.

However, since the kernel relies on a copy of the "current thread"s on
every CPU for certain operations (i.e. to compare the priority of the
currently scheduled thread on another CPU to determine if IPI should be
sent), we can't eliminate the copy of "current thread" (`current`) from
the `struct _cpu` and therefore the kernel now has to invoke
`arch_set_curr_thread()` in addition to what it has been doing. This
means that it will take slightly longer (most likely one instruction
write) to change the current thread pointer on the current
CPU.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
2024-11-23 20:12:24 -05:00
Volodymyr Fialko
9eee2eaee6 riscv: pmp: enable stackguard without multithreading
Without multithreading only two stacks present: ISR and main.
As any stack they also could overflow, so it make sense to add stack
guard for them also.

Remove stack guard dependency on multithreading and mark
`Z_RISCV_STACK_GUARD_SIZE` bytes at the beginning of stack as read-only
region with PMP entry.

Signed-off-by: Volodymyr Fialko <vfialko@marvell.com>
2024-11-20 08:25:49 -05:00
Lingao Meng
6d6d4565d3 kernel: workq: Fix function format to avoid CI Warning
formating

Signed-off-by: Lingao Meng <menglingao@xiaomi.com>
2024-11-20 10:15:20 +00:00
Lingao Meng
836fa88cbd kernel: work: Add missing ASSERT for args
Add missing ASSERT for dwork & queue.

Signed-off-by: Lingao Meng <menglingao@xiaomi.com>
2024-11-20 10:15:20 +00:00
Lingao Meng
2a3b28a3f5 kernel: work: Remove meanless foreach for work
Since call this function both in
```C
	k_spinlock_key_t key = k_spin_lock(&lock);

	bool need_flush = work_flush_locked(work, flusher);

	k_spin_unlock(&lock, key);
```

So, there are no flag_get change.

Signed-off-by: Lingao Meng <menglingao@xiaomi.com>
2024-11-20 10:15:20 +00:00
Robin Kastberg
6532c2f0bb kernel: make z_is_idle_thread_entry take code ptr
This is causing errors on IAR toolchain.

Signed-off-by: Robin Kastberg <robin.kastberg@iar.com>
2024-11-18 14:02:56 -05:00
Daniel Leung
01f8e0fa2d demand_paging: eviction: add kconfig CONFIG_EVICTION_TRACKING
This adds a new kconfig for eviction algorithm which needs page
tracking. When enabled, k_mem_paging_eviction_add()/_remove()
and k_mem_paging_eviction_accessed() must be implemented.
If an algorithm does not do page tracking, there is no need to
implement these functions, and no need for the kernel MMU code
to call into empty functions. This should save a few function
calls and some CPU cycles.

Note that arm64 unconditionally calls those functions so
forces CONFIG_EVICTION_TRACKING to be enabled there.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-11-18 13:16:44 -05:00
Corey Wharton
76bceb9ed2 kernel: mem_slab: always validate memory address on free
Allowing an invalid address to be "freed" when asserts are disabled
is dangerous and can lead to a very hard class of bugs (and potential
security issues) to troubleshoot. This change always validates the
address before adding it to the free list and calls k_panic() if
asserts are not enabled.

Signed-off-by: Corey Wharton <xodus7@cwharton.com>
2024-11-16 15:54:56 -05:00
Chris Friedt
9504034733 sys: util: use BITS_PER_BYTE macro instead of the magic number 8
Obviously, everyone knows that there are 8 bits per byte, so
there isn't a lot of magic happening, per se, but it's also
helpful to clearly denote where the magic number 8 is referring
to the number of bits in a byte.

Occasionally, 8 will refer to a field size or offset in a
structure, MMR, or word. Occasionally, the number 8 will refer
to the number of bytes in a 64-bit value (which should probably
be replaced with `sizeof(uint64_t)`).

For converting bits to bytes, or vice-versa, let's use
`BITS_PER_BYTE` for clarity (or other appropriate `BITS_PER_*`
macros).

Signed-off-by: Chris Friedt <cfriedt@tenstorrent.com>
2024-11-16 15:22:35 -05:00
Tom Burdick
2b5012a5d9 kernel: Move run queue initialization
Move the initialization of the priority q for running out of sched.c to
remove one more ifdef from sched.c. No change in functionality but
better matches the rest of sched.c and priority_q.h such that the
ifdefry needed is done in in priority_q.h.

Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2024-11-16 15:20:15 -05:00
Mikhail Kushnerov
a995d9d76f kernel: fix k_sleep in no multi-threading mode
Fix k_sleep implementation for no multi-threading mode.

Absolute value of timeout expiration was fed to the k_busy_wait()
function instead of delta value. That caused bug like incrementing of
sleep time in geometric progression (while actual function argument is
constant) during program running.

Signed-off-by: Mikhail Kushnerov <m.kushnerov@yadro.com>
2024-11-16 14:07:41 -05:00
Yong Cong Sin
de347a4e07 init: support per-core init hook
Allow SoC to implement their custom per-core initialization function by
selecting `CONFIG_SOC_PER_CORE_INIT_HOOK` and implement
`soc_per_core_init_hook()`.

Signed-off-by: Maxim Adelman <imax@meta.com>
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
2024-11-16 14:04:25 -05:00
Daniel DeGrasse
6023d6a142 arch: common: fix copy for ramfunc region during XIP init
ramfunc region is copied into RAM from FLASH region during XIP init. We
copy from the loadaddr of the region, and were previously loading to the
symbol __ramfunc_start. This is incorrect when using an MPU with
alignment requirements, as the __ramfunc_start symbol may have padding
placed before it in the region. The __ramfunc_start symbol still needs
to be aligned in order to be used by the MPU though, so define a new
symbol __ramfunc_region_start, and use that symbol when copying the
__ramfunc region from FLASH to RAM.

Fixes #75296

Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
2024-11-06 10:19:08 -08:00
Anas Nashif
6261e2a673 Revert "kernel: banner: Expose tainted builds"
This reverts commit 6d4031f96c.

Those makes majority of builds og platforms with blobs tainted although
the blob were not used or compiled in. So it is very misleading.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-10-29 14:17:21 -05:00
Peter Mitsis
cedd36106b kernel: Begin abstracting out _sched_spinlock
In a uniprocessor system, _sched_spinlock may not need to be
held in all the same cases that it does in a multiprocessor
system. Removing those unnecessary usages can lead to better
performance on UP systems. In the case of uncontested taking
and giving of a semaphore, this can be as much as a +14%
performance gain.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-10-21 18:38:00 -05:00
Peter Mitsis
f6a76c32b7 kernel: inline z_unpend_first_thread()
Inlining z_unpend_first_thread() has been observed to give a
+8% and +16% performance boost to the thread_metric benchmark's
message processing and synchronization tests respectively.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-10-21 18:38:00 -05:00
Peter Mitsis
0bf44f2352 kernel: Re-order should_preempt() checks
Re-orders the checks in should_preempt() tests so that the
z_is_thread_timeout_active() check is done last.

This change has been observed to give a +7% performance boost on
the thread_metric benchmark's preemptive scheduling test.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-10-21 18:38:00 -05:00
Reto Schneider
6d4031f96c kernel: banner: Expose tainted builds
Due to the (potentially) hard to understand effects of blobs, it seems
prudent to make their presence more noticeable.

With this change, whenever blobs are present in the Zephyr work space,
the hello world sample output looks like this:

> *** Booting Zephyr OS build (tainted) v3.7.0-4569-gd4f8765ef20e ***
> Hello World! esp32c3_devkitm/esp32c3

Before, it looked like this:

> *** Booting Zephyr OS build v3.7.0-4568-g69c47471d187 ***
> Hello World! esp32c3_devkitm/esp32c3

Signed-off-by: Reto Schneider <reto.schneider@husqvarnagroup.com>
2024-10-21 15:58:27 +02:00
Peter Mitsis
c70a619a2f kernel: Remove unused z_ready_thread_locked()
Removing the routine z_ready_thread_locked() as it is not
used anywhere. It was a leftover artefact from development
that previously escaped cleanup.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-10-15 19:08:30 -04:00
Peter Mitsis
cc415bc139 kernel: Apply 'unlikely' attribute
Applies the 'unlikely' attribute to various kernel objects that
use z_unpend_first_thread() to optimize for the non-blocking path.

This boosts the thread_metric synchronization benchmark numbers
on the frdm_k64f board by about 10%.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-10-15 04:06:32 -04:00
Anas Nashif
121cb49a46 kernel: sched: inline update_cache
This improves context switching by 7% when measured using the
thread_metric benchmark.

Before:
**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 120
Time Period Total:  5451879

After:
**** Thread-Metric Preemptive Scheduling Test **** Relative Time: 30
Time Period Total:  5853535

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-10-10 20:21:04 -04:00
Peter Mitsis
318b49570a tests: scheduler queue benchmarks
Implements a set of tests designed to show how the performance of the
three scheduler queue implementations (DUMB, SCALABLE and MULTIQ)
varies with respect to the number of threads in the ready queue.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-10-07 20:16:20 -04:00
Chris Friedt
b6aed5c505 kernel: dynamic: use 4k stack size for x86
x86 architectures require a dynamic stack size that is a multiple
of 4096 bytes due to mmu restrictions.

For example, this test would previously fail when using the
default dynamic stack size of 1024 bytes for 32-bit
platforms.

```
west build -p auto -b qemu_x86/atom/nopae -t run \
  tests/posix/common/ -- -DCONFIG_USERSPACE=y
```

It would pass with an additional argument
```
west build -p auto -b qemu_x86/atom/nopae -t run \
  tests/posix/common/ -- -DCONFIG_USERSPACE=y \
  -DCONFIG_DYNAMIC_THREAD_STACK_SIZE=4096
```

Add a special default for x86 when using dynamic thread stacks.

The x86 default removes the need for `boards/qemu_x86*.conf`,
with the exception of `qemu_x86_tiny`.

qemu_x86_tiny did not have sufficient memory (or configuration)
to run the non-userspace tests, so bump up the available ram
from 256k to 512k for this test and clone the .conf from the
demand paging tests.

Eventually, the common posix test should be split into more
concise functional categories.

Signed-off-by: Chris Friedt <cfriedt@tenstorrent.com>
2024-10-05 14:13:18 -04:00
Yong Cong Sin
52a202309b zephyr: bulk update to DT_NODE_HAS_STATUS_OKAY
Change instances of:

DT_NODE_HAS_STATUS(<node_id>, okay)

to

DT_NODE_HAS_STATUS_OKAY(<node_id>)

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
2024-10-03 17:06:52 +01:00
Daniel Flodin
746c59c82a arch: kernel: lib: toolchain: Standardize TLS keyword
Up until now, the `__thread` keyword has been used for declaring
variables as Thread local storage. However, `__thread` is a GNU
specific keyword which thus limits compatibility with other
toolchains (for instance IAR).

This PR intoduces a new macro `Z_THREAD_LOCAL` which expands to the
corresponding C11, C23 or C++11 standard keyword based on the standard
that is specified during compilation, else it uses the old `__thread`
keyword.

Signed-off-by: Daniel Flodin <daniel.flodin@iar.com>
2024-09-23 10:01:48 +02:00
Daniel Leung
f5163d8940 arch: add interface to dump privileged stack in coredump
This adds a interface to allow coredump to dump privileged
stack which is defined in architecture specific way.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-09-21 11:29:39 +02:00
Anas Nashif
e9474302da init: enhance kconfig docs for custom hooks
Improve docs for custom soc/board hooks.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-09-20 13:15:31 +02:00
Pisit Sawangvonganan
5c8a2c0dbf style: kernel: remove unnecessary return statements
For code clarity, remove unnecessary `return` statements
in functions with a void return type they don't affect control flow.

Signed-off-by: Pisit Sawangvonganan <pisit@ndrsolution.com>
2024-09-20 11:06:55 +02:00
Peter Mitsis
eda5814dbc kernel: Simplify k_thread_cpu_pin()
Simplifies the k_thread_cpu_pin() implementation to leverage the
existing cpu_mask_mod() infrastructure.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-09-20 09:02:38 +02:00
Yong Cong Sin
b55f3c1c4f kernel: remove CONFIG_MP_NUM_CPUS
`CONFIG_MP_NUM_CPUS` has been deprecated for more than 2
releases, it's time to remove it.

Updated all usage of `CONFIG_MP_NUM_CPUS` to
`CONFIG_MP_MAX_NUM_CPUS`

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
2024-09-19 18:28:37 +01:00