The reason for arch_num_cpus() is to be able to dynamically adapt to
the actual number of available CPUs at run time.
In the z_sched_init() case, it is not the number of active CPUs that
we need but rather the total number of potential CPUs, and that is
represented by CONFIG_MP_MAX_NUM_CPUS not arch_num_cpus().
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Add the `zephyr,pm-device-runtime-auto` flag to `pm.yaml` and
`struct pm_device`.
This flag is intended to signify to the boot system that device runtime
PM should be automatically enabled on the device after the init function
has run.
Only run `pm_device_runtime_auto_enable` function on a device if
initialisation succeeded. This prevents actions being run on devices
that are not ready.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
Make sliceable() the actual condition for a sliceable thread. Avoid
creating a slice timeout for non sliceable threads. Always reset
slice_expired even if the next thread is not sliceable. Fold
slice_expired_locked() into z_time_slice() to avoid the hidden
unlock/lock. Change `curr` to `thread` as this is not necessarily
the current thread (yet) being set. Make variables static.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Updates events to prevent a timeout from corrupting the list of
threads that needs to be waken up.
Signed-off-by: Aastha Grover <aastha.grover@intel.com>
Fixes race condition for k_event_post_internal() in an
SMP environment while walking the waitq. Uses z_sched_waitq_walk()
to safely walk the waitq by using a sched_spinlock.
It should be noted that since walking the wait queue is an
operation of indeterminant length, there exists the possibility
that the sched_spinlock (which is a highly used and contended-for
lock) may be locked for an indeterminant amount of time. However,
it is expected that few threads will be waiting on any given kernel
event object, which should ameliorate this risk.
Fixes#54317
Signed-off-by: Aastha Grover <aastha.grover@intel.com>
Moving timeslice events to timeouts isn't quite enough on SMP, as it's
still possible for systems that don't broadcast their timer interrupts
to end up handling an expiration for a foreign CPU. There, we need an
IPI, and a symmetric call to z_time_slice() (which is itempotent and
fast) in the IPI ISR.
Signed-off-by: Andy Ross <andyross@google.com>
Rework the fragile and ad-hoc computation of timeslice expirations
into per-CPU struct _timeout objects with regular callbacks. The
expiration callbacks themselves simply set a per-cpu flag (they might
run on any CPU), which gets checked at the end of the timer ISR on
every CPU.
This simplifies logic and removes a bunch of code. It also fixes at
least three bugs:
1. As @npitre discovered: On SMP, the number of ticks announced on any
given CPU is going to be a subset of all expired ticks. This broke
the accounting of timeslice ticks, and effectively meant that
timeslicing only worked on SMP on systems where one CPU could hog all
the announcements, and only on that CPU.
2. The bootstrap path to arm the timer driver after setting the first
timeout in an empty list couldn't take into account
sys_clock_elapsed() ticks, as it didn't know whether it was being
called underneath an existing announce loop. Now this code is no
longer responsible for knowing anything about time slicing at all.
3. Also on SMP, there was a case where two CPUs timeslicing
simultaneously could stomp on each others' timeouts in
z_set_timeout_expiry(), as neither had a way of knowing what the
other's state was. CPUs could miss their own expiration and have to
wait for the slice expiration on the other CPU. Now, timeouts are
global objects with simple expiration times, and there's no need for
that function at all.
Signed-off-by: Andy Ross <andyross@google.com>
Some of the offset symbols that are derived from the macro
GEN_OFFSET_SYM() are not used anywhere in the Zephyr codebase.
Remove them as part of a cleanup effort.
Instances of an associated GEN_OFFSET_SYM() have also been
removed when the resulting macro is no longer referenced.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Some of the offset symbols generated via the macro GEN_OFFSET_SYM()
are not used anywhere in the Zephyr codebase. Remove them as part of
a cleanup effort.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Commit 3e729b2b1c ("kernel/timer: Correctly clamp period argument")
increased the lower limit to 1 so that it wouldn't conflict with a
K_NO_WAIT. But in doing so it enforced a minimum period of 2 ticks.
And the subtraction must obviously be avoided if the period is zero, etc.
Instead of doing this masquerade in k_timer_start(), let's move the
subtraction and clamping in z_timer_expiration_handler() right before
registering a new timeout. It makes the code cleaner, and then it is
possible to have single-tick periods again.
Whith this, timer_jitter_drift in tests/kernel/timer/timer_behavior does
pass with any CONFIG_SYS_CLOCK_TICKS_PER_SEC value, even when the tick
period is equal or larger than the specified timer period for the test
which failed the test before.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The call to unschedule_locked() would return true ("successfully
unscheduled") even in the case where the underlying z_abort_timeout()
failed (because the callback was already unpended and
in-progress/complete/about-to-be-run, remember that timeout callbacks
are unsynchronized), leading to state bugs and races against the
callback behavior.
Correctly detect that case and propagate the error to the caller.
Fixes#51872
Signed-off-by: Andy Ross <andyross@google.com>
Fixes sporadic data access violations that were occuring when pipes
were being used from an ISR. The ISR was incorrectly using the pipe
descriptor belonging to the interrupted thread. This led to corrupted
pipe meta-data. The solution proposed here is to perform a run-time
check and if use a pipe descriptor on the ISR's stack if called from
an ISR.
For additional information, see:
https://github.com/zephyrproject-rtos/zephyr/issues/52812
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Adds a spin lock/unlock barrier pair after a pipe thread wakes.
After the list of waiting threads is generated, it is possible for
threads on that list to timeout and be removed from the wait queue.
However, since that list was generated before the timeout occurred,
the timed-out thread must wait until the copying is done (the
pipe's spin-lock has been released).
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
By the time the working list of readers/writers is processed, it is
possible that waiting reader/writer being processed had timed out
and is no longer on the wait queue. As such, we can not blindly
wake the next thread as that next thread might not be the thread we
had just been processing.
To address this, the calls to z_sched_wake() have been replaced
with z_unpend_thread() and z_ready_thread() so that a specific
thread can be safely targeted for waking.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Uses the new z_sched_waitq_walk() routine to walk the pipe's wait
queue to build a list of waiting threads that will be used for
the data transfer.
This method is preferred over the previous as it ensures that
wait queue is safely traversed.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Adds a routine to safely walk a specified wait queue and invoke a
custom callback function on each waiting thread.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
When a timer is restarted from a high priority interrupt, it may
happen that the timer is re-added to the timeout list right after
it is removed from that list prior to execution of its expiration
handler but before that execution actually occurs. This leads to
an assertion failure reported for `z_add_timeout()` because then
that function, called from `z_timer_expiration_handler()` for
periodic timers, turns out to be adding a timeout that is already
added to the timeout list.
This commit detects such situation in `z_timer_expiration_handler()`
and makes that function exit immediately when that occurs (as the
timer was restared, its expiration handler should not be executed).
Signed-off-by: Andrzej Głąbek <andrzej.glabek@nordicsemi.no>
Most of the time, z_cstart() is running on an arbitrary region
of memory as stack, where the necessary stack setup has not been
performed. This prevents stack protection to work correctly,
as the stack canary has not been populated. So mark z_cstart()
to have no stack protection at all inside the function to avoid
raising exception during boot.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This commit updates all in-tree code to use `CONFIG_CPP` instead of
`CONFIG_CPLUSPLUS`, which is now deprecated.
Signed-off-by: Stephanos Ioannidis <stephanos.ioannidis@nordicsemi.no>
At least one static analysis tool is flagging a potential NULL
derefence in sys_clock_announce()'s tick processing loop where the
routine 'first()' is concerned. In practice, this does not occur as
...
1. The code in question is protected by a spinlock.
2. 'first()' does not change the contents of anything.
The code has consequently been tweaked to prevent similar such false
positives in the future.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Accurate timekeeping is something that is often taken for granted.
However, reliability of timekeeping code is critical for most core
and subsystem code. Furthermore, Many higher-level timekeeping
utilities in Zephyr work off of ticks but there is no way to modify
ticks directly which would require either unnecessary delays in
test code or non-ideal compromises in test coverage.
Since timekeeping is so critical, there should be as few barriers
to testing timekeeping code as possible, while preserving
integrity of the kernel's public interface.
With this, we expose `sys_clock_tick_set()` as a system call only
when `CONFIG_ZTEST` is set, declared within the ztest framework.
Signed-off-by: Chris Friedt <cfriedt@meta.com>
The following testcases fail with qemu_cortex_r5 caused by main stack
overflow.
tests/kernel/workq/work_queue/kernel.workqueue
tests/ztest/base/testing.ztest.base.verbose_0_userspace
The main stack size is 512 for qemu_cortex_r5(a Cortex-A/R aarch32
platform) with CONFIG_ZTEST=y. The Cortex-M platforms are already set to
1024. Likely 512 will fail for most aarch32 platforms soon.
Fix the issue by increasing the CONFIG_MAIN_STACK_SIZE to 1024.
Also, remove 'default 1024 if TEST_ARM_CORTEX_M' since Cortex-M is no
longer an exception of default 1024.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
- Logging supports printing 64-bit values now. Cast to unsigned long and
use %lu all times.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
In z_phys_unmap(), the call to virt_region_free() is not using
aligned virtual address and space. This can result in freeing
smaller region that allocated given that inputs to z_phys_unmap()
may not be aligned. So use the already calculated aligned
virtual address and size as input to virt_region_free().
Note that the assertion and if-block in virt_region_free() to
check whether the to-be-unmapped region is within the virtual
memory region needs to be trimmed by one byte at the end.
The assertion and if-block are checking against the region
end address but (start + size) is just one byte over the end.
So subtract one.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The C++ standard requires the main() function to have the return type
of 'int' and does not allow the main() to be defined with the 'void'
return type. Moreover, GCC goes as far as to emit a hard error when the
'::main()' has the return type of `void`.
This commit introduces an option to instruct the Zephyr kernel to call
the 'int main(void)' instead of the 'void main(void)' in case a Zephyr
application defines main() in a C++ source file.
Signed-off-by: Stephanos Ioannidis <stephanos.ioannidis@nordicsemi.no>
Move runtime code to use arch_num_cpus() instead of CONFIG_MP_NUM_CPUS
and use CONFIG_MP_MAX_NUM_CPUS for ifdef and BUILD_ASSERT macros.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Cleanup the mess of duplicate function definitions, unnecessary
variables and duplicate strings. All banner strings are now constant in
ROM. Also fixes a double space between the end of the version string and
the trailing `***` when there is no boot delay.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
The BOOT_DELAY option does nothing in code if MULTITHREADING is not
enabled. Move the dependency to Kconfig instead.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
Change for loops of the form:
for (i = 0; i < CONFIG_MP_NUM_CPUS; i++)
...
to
unsigned int num_cpus = arch_num_cpus();
for (i = 0; i < num_cpus; i++)
...
We do the call outside of the for loop so that it only happens once,
rather than on every iteration.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Continue to phase out MP_NUM_CPUS, change Kconfig to be
MP_MAX_NUM_CPUS and make MP_MAX_NUM_CPUS the main Kconfig symbol.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
The dummy thread doesn't include a TLS area, so any thread local variables
will fail to work if used in the switched_out tracing hook. Skip the hook
in this case, as it's not really accurate anyways; the dummy thread is
only used to set context for the initial switch for each core.
Signed-off-by: Keith Packard <keithp@keithp.com>
Using char pointers for %p should be avoided in log messages. It will
cause issues in configurations where logging strings are removed from
the binary and they are not inspected when cbprintf packages from
logging string are built. In that case any char pointers are treated as
strings and copied into the pacakge body.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Introduce a Kconfig (MP_MAX_NUM_CPUS) and an api arch_num_cpus() to
allow for systems that might determine the number of CPUs available to
Zephyr at runtime.
CONFIG_MP_MAX_NUM_CPUS is intented to be use for any array initialization
and such that need to occur at build time. For most systems
arch_num_cpus() will just report the value of CONFIG_MP_MAX_NUM_CPUS.
The intent is to phase out CONFIG_NP_NUM_CPUS.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Warnings being treated as errors when building :
Error this 'for' clause does not guard...
[-Werror=misleading-indentation]
Signed-off-by: Francois Ramu <francois.ramu@st.com>
The _SYS_INIT_LEVEL* definitions were used to indicate the index entry
into the levels array defined in init.c (z_sys_init_run_level). init.c
uses this information internally, so there is no point in exposing this
in a public header. It has been replaced with an enum inside init.c. The
device shell was re-using the same defines to index its own array. This
is a fragile design, the shell needs to be responsible of its own data
indexing. A similar situation happened with some unit tests.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
The function in charge of calling all init function was defined in
device.c, had a public prototype and was just used in init.c. Since this
is really an internal function tied to Kernel init code, move it to
init.c and make it static, there's no need to expose it publicly.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
The `ARCH` init level was added to solve a specific problem, call init
code (SYS_INIT/devices) before `z_cstart` in the `intel_adsp` platform.
The documentation claims it runs before `z_cstart`, but this is only
true if the SoC/arch takes care of calling:
```c
z_sys_init_run_level(_SYS_INIT_LEVEL_ARCH);
```
Which is only true for `intel_adsp` nowadays. So in practice, we now
have a platform specific init level. This patch proposes to do things in
a slightly different way. First, level name is renamed to `EARLY`, to
emphasize it runs in the early stage of the boot process. Then, it is
handled by the Kernel (inside `z_cstart()` before calling
`arch_kernel_init()`). This means that any platform can now use this
level. For `intel_adsp`, there should be no changes, other than
`gcov_static_init()` will be called before (I assume this will allow to
obtain coverage for code called in EARLY?).
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
For historical reasons[1] suspending threads would release the
scheduler lock between pend() (which places the current thread onto a
wait queue) and z_swap() (which effects the context swtich). This
process happens with the caller's lock held, so local interrupts are
masked. But on SMP this opens a tiny race where another CPU could
grab the pended thread and switch to it while we were still executing
on its stack!
Fix this by elevating the "lock swap" code that already exists in the
(portable/switch-based) z_swap() code one level so that it happens in
z_pend_curr() also. Now we hold the scheduler lock between pend and
the final context switch.
Note that this technique can't work for the older z_swap_irqlock()
implementation, which exists to vestigially support a few bits of arch
code (mostly direct interrupts) that don't work on SMP anyway.
Address with an assert to prevent future misuse.
[1] z_swap() is a historical API implemented in per-arch assembly for
older architectures (like ARM32!). It was designed to be called
with what at the time was a global IRQ lock, so it doesn't
understand the idea of a separate scheduler lock. When we finally
get all archictures on arch_switch() this design can be cleaned up
quite a bit.
Signed-off-by: Andy Ross <andyross@google.com>
We have cases where some devices needs to be initialized very early and
before c_start is call, i.e. to setup very early console or to setup
memory. Traditionally this would be hardcoded as part of the soc layer
and not using device model or the init levels.
This patch adds a new level ARCH, which will be called in early
architecture code and before we jump to the kernel code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
k_heap_aligned_alloc was not handling K_FOREVER timeout
correctly due to unsigned return value. Added explicit
K_FOREVER handling of end time.
Fixes#50611.
Signed-off-by: Jay Shoen <jay.shoen@perceive.io>
The interrupt stack is used as the system stack during kernel
initialization while IRQs are not yet enabled. The sp register is
set to z_interrupt_stacks + CONFIG_ISR_STACK_SIZE.
CONFIG_ISR_STACK_SIZE only represents the desired usable stack size.
This does not take into account the added guard area. Result is a stack
whose pointer is much closer to the trigger zone than expected when
CONFIG_PMP_STACK_GUARD=y, and the SMP configuration in particular pushes
it over the edge during many CI test cases.
Worse: during early init we're not quite ready to handle exceptions
yet and complete havoc ensues with no meaningful debugging output.
Make sure the early assembly code locates the actual top of the stack
by generating a constant with its true size.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Obtaining the CPU outside of the spin locks on SMP would
result in an assert failing on __ASSERT(!z_smp_mobile())
which makes sense as the current cpu may change.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
The requirement for k_yield() to handle "yielding" in the idle thread
was removed a while back, but it missed a spot where we'd try to yield
in the fallback loop on bringup platforms that lack an IPI. This now
crashes, because yield now unconditionally tries to reschedule the
current thread, which doesn't work for idle threads that don't live in
the run queue.
Just make it a busy loop calling swap(), even simpler.
Fixes#50119
Signed-off-by: Andy Ross <andyross@google.com>
When building with CONFIG_SCHED_CPU_MASK_PIN_ONLY=y, CPU mask
is fixed and cannot be changed while thread is running.
The current code asserts if thread state is anything but PREPARED.
We do however have interface like k_work_queue_start() where a thread is
started as part of the queue start. To allow user to set the pinned CPU
for the work queue thread, it needs to be possible to suspend the
thread, set the mask, and then call k_thread_resume(). This seems to be
a valid sequence, so relax the assert check to reflect this.
Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
k_poll does not currently allow polling on pipes. This adds support
for doing so on buffered pipes.
Signed-off-by: Jeremy Herbert <jeremy.006@gmail.com>
When a cache API function is called from userspace, this results on
ARM64 in an OOPS (bad syscall error). This is due to at least two
different factors:
- the location of the cache handlers is preventing the linker to
actually find the handlers
- specifically for ARM64 and ARC some cache handling functions are not
implemented (when userspace is not used the compiler simply optimizes
out these calls)
Fix the problem by:
- moving the userspace cache handlers to a their logical and proper
location (in the drivers directory)
- adding the missing handlers for ARM64 and ARC
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
Many device pointers are initialized at compile and never changed. This
means that the device pointer can be constified (immutable).
Automated using:
```
perl -i -pe 's/const struct device \*(?!const)(.*)= DEVICE/const struct
device *const $1= DEVICE/g' **/*.c
```
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
There's no point to doing this when the host OS clears all memory at
mapping time. And as it turns out, the __bss_end symbol it was
relying on actually comes from the host toolchain's linker, not our
own linker scripts (making it semi-dangerous to rely on). And it's
not present in clang/lld output anyway.
Signed-off-by: Andy Ross <andyross@google.com>
This new implementation of pipes has a number of advantages over the
previous.
1. The schedule locking is eliminated both making it safer for SMP
and allowing for pipes to be used from ISR context.
2. The code used to be structured to have separate code for copying
to/from a wating thread's buffer and the pipe buffer. This had
unnecessary duplication that has been replaced with a simpler
scatter-gather copy model.
3. The manner in which the "working list" is generated has also been
simplified. It no longer tries to use the thread's queuing node.
Instead, the k_pipe_desc structure (whose instances are on the
part of the k_thread structure) has been extended to contain
additional fields including a node for use with a linked list. As
this impacts the k_thread structure, pipes are now configurable
in the kernel via CONFIG_PIPES.
Fixes#47061
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Say threadA holds a mutex and threadB tries
to lock it with a timeout, a race would occur
if threadA unlock that mutex after threadB
got unpended by sys_clock and before it gets
scheduled and calls k_spin_lock.
This patch fixes this issue by checking the
mutex's status again after k_spin_lock calls.
Fixes#48056
Signed-off-by: Qi Yang <qi.yang@cmind-semi.com>
Fixes#46324
Set dummy_thread->base.slice_ticks to 0 when
CONFIG_TIMESLICE_PER_THREAD is set. To avoid
_current_cpu->slice_ticks be a big number.
Signed-off-by: Hu Zhenyu <zhenyu.hu@intel.com>
Fixes an issue in sys_clock_tick_get() that could lead to drift in
a k_timer handler. The handler is invoked in the timer ISR as a
callback in sys_tick_announce().
1. The handler invokes k_uptime_ticks().
2. k_uptime_ticks() invokes sys_clock_tick_get().
3. sys_clock_tick_get() must call elapsed() and not
sys_clock_elapsed() as we do not want to count any
unannounced ticks that may have elapsed while
processing the timer ISR.
Fixes#46378
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Updates sys_clock_announce() such that the <announce_remaining> update
calculation is done after the callback. This prevents another core from
entering the timeout processing loop before the first core leaves it.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
There is no easy way to clear event bits without
the potential for a race to exist between producer(s)
and consumer(s). The result of this race is that events
can be lost through the various resetting mechanisms
available (flag to k_event_wait(), or k_event_set()).
Add k_event_set_masked() which permits bits to be set or cleared.
This allows consumers to clear just the bits that they have read
without (accidentally) discarding any new bits.
Update unit tests to verify the functionality.
Partly Fixes#46117.
Signed-off-by: Andrew Jackson <andrew.jackson@amd.com>
Although there is nothing wrong with the existing code,
it doesn't permit individual bits to be set (or cleared).
This makes further changes slightly awkward.
Use a mask to restrict the bits set in an event.
Signed-off-by: Andrew Jackson <andrew.jackson@amd.com>
MISRA C:2012 Rule 14.4 (The controlling expression of an if statement
and the controlling expression of an iteration-statement shall have
essentially Boolean type.)
Use `bool' instead of `int' to represent Boolean values.
Use `do { ... } while (false)' instead of `do { ... } while (0)'.
Use comparisons with zero instead of implicitly testing integers.
This commit is a subset of the original commit:
5d02614e34a86b549c7707d3d9f0984bc3a5f22a
Signed-off-by: Simon Hein <SHein@baumer.com>
irq_lock() returns an unsigned integer key.
Generated by spatch using semantic patch
scripts/coccinelle/irq_lock.cocci
Signed-off-by: Johann Fischer <johann.fischer@nordicsemi.no>
Adds memory usage runtime stats routines that parallel those used
by both the heap and mem_blocks. This helps maintain some level of
of consistency across the different memory types.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Move scripts needed by the build system and not designed to be run
individually or standalone into the build subfolder.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Update the two locations that use two `SYS_INIT` macros with the same
initilisation functions to use `SYS_INIT_NAMED`.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
Add a minimal EFI console driver to support printf, this console driver
only supports console output. Otherwise the printf will not work.
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
Adds compatibility with Intel ADSP GDB from Zephyr SDK and
from Cadence toolchain to coredump_gdbserver.py.
Adds CAVS 15-25 (APL) register definitions. Implements
handle_register_single_read_packet to serve ADSP GDB
p packets.
Prevents BSA from changing between stack dump printout
and coredump by taking lock. Observed to be necessary for
accurate results on slower simulated platforms.
Signed-off-by: Lauren Murphy <lauren.murphy@intel.com>
Logging v1 has been removed and log_strdup wrapper function is no
longer needed. Removing the function and its use in the tree.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
This commit updates all deprecated `K_KERNEL_PINNED_STACK_ARRAY_EXTERN`
macro usages to use the `K_KERNEL_PINNED_STACK_ARRAY_DECLARE` macro
instead.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Files including <zephyr/kernel.h> do not have to include
<zephyr/zephyr.h>, a shim to <zephyr/kernel.h>.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Rename the symbols used to denote the locations of the global
constructor lists and modify the Zephyr start-up code accordingly.
On POSIX systems this ensures that the native libc init code won't
find any constructors to run before Zephyr loads.
Fixes#39347, #36858
Signed-off-by: David Palchak <palchak@google.com>
Use a new environment variable,
ZEPHYR_TOOLCHAIN_SUPPORTS_THREAD_LOCAL_STORAGE, to set the value for
TOOLCHAIN_SUPPORTS_THREAD_LOCAL_STORAGE instead of setting it to 'n' for
all non-Zephyr toolchains. In particular, the Debian arm-none-eabi
toolchain has TLS support and with this option, can be used to build
Zephyr with thread local variables.
Signed-off-by: Keith Packard <keithp@keithp.com>
Documentation specifies that aborting/terminating/exiting essential
threads is a system panic condition, but we didn't actually implement
that and allowed it as for other threads. At least one app wants to
exploit this documented behavior as a "watchdog" kind of condition,
and that seems reasonable. Do what we say we're supposed to do.
This also includes a small fix to a test, which seemed like it was
written to exercise exactly this condition. Except that it failed to
detect whether or not a system fatal error was actually signaled and
was (incorrectly) indicating "success". Check that we actually enter
the handler.
Fixes#45545
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The function k_thread_runtime_stats_all_get() now populates the
current_cycles field in the thread runtime stats structure.
Resets the number of cycles in the CPU's current usage window once
the idle thread is scheduled.
Fixes the average_cycles calcuation.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
For a library which already provides a multi-thread aware errno, use
that instead of creating our own internal value.
Signed-off-by: Keith Packard <keithp@keithp.com>
This adds the internal function z_work_submit_to_queue(), which
submits the work item to the queue but doesn't force the thread to yield,
compared to the public function k_work_submit_to_queue().
When called from poll.c in the context of k_work_poll events, it ensures
that the thread does not yield in the context of the spinlock of object
that became available.
Fixes#45267
Signed-off-by: Lucas Dietrich <ld.adecy@gmail.com>
In order to bring consistency in-tree, migrate all kernel code to the
new prefix <zephyr/...>. Note that the conversion has been scripted,
refer to zephyrproject-rtos#45388 for more details.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Implements a function that application and driver code can use to check
whether it is valid to yield (or block) in the current context. This
check is required for functions that can feasibly be run from multiple
contexts. The primary intended use case is power management transition
functions, which can be run by application code explicitly or
automatically in the idle thread by system PM.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
This adds lazy floating point context switching. On svc/irq entrance,
the VFP is disabled and a pointer to the exception stack frame is saved
away. If the esf pointer is still valid on exception exit, then no
other context used the VFP so the context is still valid and nothing
needs to be restored. If the esf pointer is NULL on exception exit,
then some other context used the VFP and the floating point context is
restored from the esf.
The undefined instruction handler is responsible for saving away the
floating point context if needed. If the handler is in the first
irq/svc context and the current thread uses the VFP, then the float
context needs to be saved. Also, if the handler is in a nested context
and the previous context was using the FVP, save the float context.
Signed-off-by: Bradley Bolen <bbolen@lexmark.com>
Do not allow changing the CPU which a thread is pinned when it is
already being executed. This allows further optimizations in some
platforms with incoherent memory since we can safely assume that the
thread will run in the same CPU and avoid invalidate / flush the
cache during context switches.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
The k_timer utility was written to assume that the kernel timeout
handler would never be delayed by more than a tick, so it can naively
reschedule the next interrupt with a simple delay.
Unfortunately real platforms have glitchy hardware and high tick
rates, and on intel_adsp we're seeing this promise being broken in
some circumstances.
It's probably not a good idea to try to plumb the timer driver
interface up into the IPC layer to do this correction, but thankfully
the existing absolute timeout API provides the tools we need (though
it does require that CONFIG_TIMEOUT_64BIT be enabled).
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The original design intent with arch_sched_ipi() was that
interprocessor interrupts were fast and easily sent, so to reduce
latency the scheduler should notify other CPUs synchronously when
scheduler state changes.
This tends to result in "storms" of IPIs in some use cases, though.
For example, SOF will enumerate over all cores doing a k_sem_give() to
notify a worker thread pinned to each, each call causing a separate
IPI. Add to that the fact that unlike x86's IO-APIC, the intel_adsp
architecture has targeted/non-broadcast IPIs that need to be repeated
for each core, and suddenly we have an O(N^2) scaling problem in the
number of CPUs.
Instead, batch the "pending" IPIs and send them only at known
scheduling points (end-of-interrupt and swap). This semantically
matches the locations where application code will "expect" to see
other threads run, so arguably is a better choice anyway.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The work queue has a semi/non-standard reschedule point implemented
using k_yield(), with a check to see if the current thread is
preemptible. Just call z_reschedule_unlocked(), it has this check
internally and is the intended API for this.
Really, this is only a half fix. Ideally the schedule point and the
lock release should be atomic[1] via the more idiomatic
z_reschedule(). But that would take some surgery, so let's go with
the simpler cleanup first.
This also avoids having to duplicate logic that gets added to
reschedule points by an upcoming patch.
[1] So that they represent a condition variable and don't race at the
end. In this case the race is present but benign, since the only thing
we really want to know is that the queue thread gets a chance to run.
The only cost is an occasional duplicated/needless context switch if
two threads are racing on a submit.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Removes an unnecessary schedule lock/unlock pair from k_mutex_unlock().
Rationale: Given that only the current thread (which would also be the
mutex owner) will be able to modify the mutex object AND that a
recursive unlock ought never trigger any reschedule (as it does not
touch the pend queue), then performing a schedule lock is not needed
prior to testing for a recursive unlock.
Furthermore, even if it is not a recursive unlock, then a schedule lock
is superfluous as the existing spinlock provides sufficient protection.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
When threads are in more than one state at a time, k_thread_state_str()
returns a string that lists each of its states delimited by a '+'.
This in turn necessitates a change to the API that includes both a
pointer to the buffer to use for the string and the size of the buffer.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Add an API that clears cpu mask from a thread and sets it to a specific
CPU.
This is the equivelent of:
k_thread_cpu_mask_clear(&thread);
k_thread_cpu_mask_enable(&thread, cpu_idx);
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Instead of resizing all devices handles, we just resize devices that are
power domains. This means that a power domain has to be declared as
compatbile with "power-domain" in device tree node.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Add API to add devices to a power domain in runtime. The number of
devices that can be added is defined in build time.
The script gen_handles.py will check the number defined in
`CONFIG_PM_DEVICE_POWER_DOMAIN_DYNAMIC` to resize the handles vector,
adding empty slots in the supported sector to be used later.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Commit b1182bf83b ("kernel/timeout: Serialize handler callbacks on
SMP") introduced an important fix to timeout handling on
multiprocessor systems, but it did it in a clumsy way by holding a
spinlock across the entire timeout process on all cores (everything
would have to spin until one core finished the list). The lock also
delays any nested interrupts that might otherwise be delivered, which
breaks our nested_irq_offload case on xtensa+SMP (where contra x86,
the "synchronous" interrupt is sensitive to mask state).
Doing this right turns out not to be so hard: take the timeout lock,
check to see if someone is already iterating
(i.e. "announce_remaining" is non-zero), and if so just increment the
ticks to announce and exit. The original cpu will then complete the
full timeout list without blocking any others longer than needed to
check the timeout state.
Fixes#44758
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
On multiprocessor systems, it's routine to enter sys_clock_announce()
in parallel (the driver will generally announce zero ticks on all but
one cpu).
When that happens, each call will independently enter the loop over
the timeout list. The access is correctly synchronized, so the list
handling is correct. But the lock is RELEASED around the invocation
of the callback, which means that the individual callbacks may
interleave between cpus. That means that individual
application-provided callbacks may be executed in parallel, which to
the app is indistinguishable from "out of order".
That's surprising and error-prone. Don't do it. Place a secondary
outer spinlock around the announce loop (but not the timeslicing
handling) to correctly serialize the timeout handling on a single cpu.
(It should be noted that this was discovered not because of a timeout
callback race, but because the resulting simultaneous calls to
sys_clock_set_timeout from separate cores seems to cause extremely
high latency excursions on intel_adsp hardware using the cavs_timer
driver. That hardware issue is still poorly understood, but this fix
is desirable regardless.)
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The idle thread got an index suffix in #23536 to make it easier to
identify different idle threads on different cores. This looks out of
place on single-core devices when the idle thread is listed next to
other kernel threads, such as main.
Remove the idle thread index on single-core platforms, and replace all
references to this format in tests and documentation.
Signed-off-by: Trond Einar Snekvik <Trond.Einar.Snekvik@nordicsemi.no>
This is an attempt at formally distinguishing and supporting the case
described in 40795 where an architecture doesn't preserve/restore the
complete thread state upon entering/exiting interrupt exception state.
This is mainly about promoting the current behavior from the accepted
workaround to a formal API specification. This workaround is currently
used on ARM64 but RISC-V requires it too.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
