CLIC supports mintstatus.MIL (RO) and mcause.MPIL (RW) for the current
interrupt level and the previous interrut level before a trap. Each ISR
must execute MRET to set mcause.MPIL back to mintstatus.MIL.
This commit introduces CONFIG_CLIC_SUPPORT_INTERRUPT_LEVEL to handle
mcause.MPIL for interrupt preemption in nested ISR, and uses
CONFIG_RISCV_ALWAYS_SWITCH_THROUGH_ECALL to ensure ISR always switch out
with MRET.
e.g.
With CONFIG_RISCV_ALWAYS_SWITCH_THROUGH_ECALL=n, a context-switch in
ISR may skip MRET in this flow:
IRQ -> _isr_wrapper -> z_riscv_switch() -> retrun to arch_switch()
Signed-off-by: Jimmy Zheng <jimmyzhe@andestech.com>
When RISCV_SOC_HAS_ISR_STACKING is used, it may
be needed to initialize custom hw stacked esf members.
Some initial values may need to be aligned with
hw stacking mechanism to avoid any side effects.
Signed-off-by: Łukasz Stępnicki <lukasz.stepnicki@nordicsemi.no>
Mostly a revert of commit b1def7145f ("arch: deprecate `_current`").
This commit was part of PR #80716 whose initial purpose was about providing
an architecture specific optimization for _current. The actual deprecation
was sneaked in later on without proper discussion.
The Zephyr core always used _current before and that was fine. It is quite
prevalent as well and the alternative is proving rather verbose.
Furthermore, as a concept, the "current thread" is not something that is
necessarily architecture specific. Therefore the primary abstraction
should not carry the arch_ prefix.
Hence this revert.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
`_current` is now functionally equals to `arch_curr_thread()`, remove
its usage in-tree and deprecate it instead of removing it outright,
as it has been with us since forever.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Up until now, the `__thread` keyword has been used for declaring
variables as Thread local storage. However, `__thread` is a GNU
specific keyword which thus limits compatibility with other
toolchains (for instance IAR).
This PR intoduces a new macro `Z_THREAD_LOCAL` which expands to the
corresponding C11, C23 or C++11 standard keyword based on the standard
that is specified during compilation, else it uses the old `__thread`
keyword.
Signed-off-by: Daniel Flodin <daniel.flodin@iar.com>
This implements arch_thread_priv_stack_space_get() so this can
be used to figure out how much privileged stack space is used.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds the bits to initialize the privileged stack when
a thread is transitioning to user mode. This prevents
information leaking if the stack is reused, and also aids
in calculating stack space usage during system calls.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Simple rename to align the kernel naming scheme. This is being
used throughout the tree, especially in the architecture code.
As this is not a private API internal to kernel, prefix it
appropriately with K_.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Simple rename to align the kernel naming scheme. This is being
used throughout the tree, especially in the architecture code.
As this is not a private API internal to kernel, prefix it
appropriately with K_.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This reverts commit f0b458a619.
This is a pointless change that simply increases footprint.
Existing code already supports compilation without multithreading.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Allow builds which has CONFIG_MULTITHREADING disabled.
This is reduce code footprint which is handy for
constrained targets as bootloaders.
Signed-off-by: Marek Matej <marek.matej@espressif.com>
Enable single-threading support for the riscv architecture.
Add z_riscv_switch_to_main_no_multithreading function for
supporting single-threading.
The single-threading does not work with enabling PMP_STACK_GUARD.
It is because single-threading does not use context-switching.
But the privileged mode transition that PMP depends on implicitly
presupposes using context-switching. It is a contradiction.
Thus, disable PMP_STACK_GUARD when MULTITHREADING is not enabled.
Signed-off-by: TOKITA Hiroshi <tokita.hiroshi@fujitsu.com>
FPU context switching is always performed on demand through the FPU
access exception handler. Actual task switching only grants or denies
FPU access depending on the current FPU owner.
Because RISC-V doesn't have a dedicated FPU access exception, we must
catch the Illegal Instruction exception and look for actual FP opcodes.
There is no longer a need to allocate FPU storage on the stack for every
exception making esf smaller and stack overflows less likely.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Currently it is assumed that Zephyr CPU numbers match their hartid
value one for one. This assumption was relied upon to efficiently
retrieve the current CPU's `struct _cpu` pointer.
People are starting to have systems with a mix of different usage for
each CPU and such assumption may no longer be true.
Let's completely decouple the hartid from the Zephyr CPU number by
stuffing each CPU's `struct _cpu` pointer in their respective scratch
register instead. `arch_curr_cpu()` becomes more efficient as well.
Since the scratch register was previously used to store userspace's
exception stack pointer, that is now moved into `struct _cpu_arch`
which implied minor user space entry code cleanup and rationalization.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
This reverts the bulk of commit c8bfc2afda ("riscv: make
arch_is_user_context() SMP compatible") and replaces it with a flag
stored in the thread local storage (TLS) area, therefore making TLS
mandatory for userspace support on RISC-V.
This has many advantages:
- The tp (x4) register is already dedicated by the standard for this
purpose, making TLS support almost free.
- This is very efficient, requiring only a single instruction to clear
and 2 instructions to set.
- This makes the SMP case much more efficient. No need for funky
exception code any longer.
- SMP and non-SMP now use the same implementation making maintenance
easier.
- The is_user_mode variable no longer requires a dedicated PMP mapping
and therefore freeing one PMP slot for other purposes.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
5f65dbcc9dab3d39473b05397e05.
The tp (x4) register is neither caller nor callee saved according to
the RISC-V standard calling convention. It only has to be set on thread
context switching and is otherwise read-only.
To protect the kernel against a possible rogue user thread, the tp is
also re-set on exception entry from u-mode.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
A separate privileged stack is used when CONFIG_GEN_PRIV_STACKS=y. The
main stack guard area is no longer needed and can be made available to
the application upon transitioning to user mode. And that's actually
required if we want a naturally aligned power-of-two buffer to let the
PMP map a NAPOT entry on it which is the whole point of having this
CONFIG_PMP_POWER_OF_TWO_ALIGNMENT option in the first place.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The StackGuard area is used to save the esf and run the exception code
resulting from a StackGuard trap. Size it appropriately.
Remove redundancy, clarify documentation, etc.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
In order to bring consistency in-tree, migrate all arch code to the new
prefix <zephyr/...>. Note that the conversion has been scripted, refer
to zephyrproject-rtos#45388 for more details.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Add the appropriate hooks effectively replacing the old implementation
with the new one.
Also the stackguard wasn't properly enforced especially with the
usermode combination. This is now fixed.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Set TP in exception context so that it gets loaded into the CPU when
first running the thread. Set TP for secondary cores to related idle TLS
area.
Signed-off-by: Keith Packard <keithp@keithp.com>
This is painful. There is no way for u-mode code to know if we're
currently executing in u-mode without generating a fault, besides
stealing a general purpose register away from the standard ABI
that is. And a global variable doesn't work on SMP as this must be
per-CPU and we could be migrated to another CPU just at the right
moment to peek at the wrong CPU variable (and u-mode can't disable
preemption either).
So, given that we'll have to pay the price of an exception entry
anyway, let's at least make it free to privileged threads by using
the mscratch register as the non-user context indicator (it must
be zero in m-mode for exception entry to work properly). In the
case of u-mode we'll simulate a proper return value in the
exception trap code. Let's settle on the return value in t0
and omit the volatile to give the compiler a chance to cache
the result.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The move to arch_switch() is a prerequisite for SMP support.
Make it optimal without the need for an ECALL roundtrip on every
context switch. Performance numbers from tests/benchmarks/sched:
Before:
unpend 107 ready 102 switch 188 pend 218 tot 615 (avg 615)
After:
unpend 107 ready 102 switch 170 pend 217 tot 596 (avg 595)
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
This is a per-thread register that gets updated only when context
switching. No need to load and save it on every exception entry.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Complete revamp of the exception entry code, including syscall handling.
Proper syscall frame exception trigger. Many correctness fixes, hacks
removal, etc. etc.
I tried to make this into several commits, but this stuff is all
inter-related and a pain to split.
The diffstat summary:
14 files changed, 250 insertions(+), 802 deletions(-)
Binary size (before):
text data bss dec hex filename
1104 0 0 1104 450 isr.S.obj
64 0 0 64 40 userspace.S.obj
Binary size (after):
text data bss dec hex filename
600 0 0 600 258 isr.S.obj
36 0 0 36 24 userspace.S.obj
Run of samples/userspace/syscall_perf (before):
*** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 ***
Main Thread started; qemu_riscv32
Supervisor thread started
User thread started
Supervisor thread(0x80010048): 384 cycles 509 instructions
User thread(0x80010140): 77312 cycles 77437 instructions
Run of samples/userspace/syscall_perf (after):
*** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 ***
Main Thread started; qemu_riscv32
Supervisor thread started
User thread started
Supervisor thread(0x80010048): 384 cycles 509 instructions
User thread(0x80010138): 7040 cycles 7165 instructions
Yes, that's more than a 10x speed-up!
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
This reverts commit be28de692c.
The purpose of this commit will be reintroduced later on top of
a cleaner codebase.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Enable `arch_switch()` as preparation for SMP support. This patch
doesn't try to keep support for old style context swap - only switch
based swap is supported, to keep things simple.
A fair amount of refactoring was done in this patch, specially regarding
the code that decides what to do about the ISR. In RISC-V, ECALL
instructions are used to signalize several events, such as user space
system calls, forced syscall, IRQ offload, return from syscall and
context switch. All those handled by the ISR - which also handles
interrupts. After refactor, this "dispatching" step is done at the
beginning of ISR (just after saving generic registers).
As with other platforms, the thread object itself is used as the thread
"switch handle" for the context swap.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Currently, is_user_mode is 8-byte in riscv64 and it breaks a 4-byte PMP
region protecting it. Because is_user_mode is a single flag, we could
just fix it's size to 4-byte in both riscv32 and riscv64.
Signed-off-by: Jim Shu <cwshu09@gmail.com>
Thread init related to PMP & userspace contains 5 parts:
1. User/supervisor thread clear PMP context
2. User thread clear it's context
3. User/supervisor thread assign to different entry
4. Supervisor thread assign mstatus.MPRV for M-mode PMP protection
5. User/supervisor thread setup PMP regions of stack guard if enabled
Signed-off-by: Jim Shu <cwshu@andestech.com>
This reverts commit 7b09d031fa. Because
context save of GP register is removed, we don't need to initialize GP
at thread init. GP will be a constant value so that it could only be
initialized at program start.
Signed-off-by: Jim Shu <cwshu@andestech.com>
Plus added implementation for esp32c3 SoC.
Signed-off-by: Felipe Neves <ryukokki.felipe@gmail.com>
Signed-off-by: Felipe Neves <felipe.neves@espressif.com>
The stack frame size, used for context switch, is rounded up to 16-bytes
alignment. Therefore, we need round down the pointer of top of the
pre-populated stack frame so that the preserved stack frame size is also
rounded up to 16-bytes alignment.
Fixes#29535
Signed-off-by: Shih-Wei Teng <swteng@andestech.com>
In unshared FP mode, only 1 thread can use FPU but kernel doesn't know
which one, so riscv arch would enable FPU of each thread.
Signed-off-by: Jim Shu <cwshu@andestech.com>
This patch introduce new API to enable FPU of thread. This is pair of
existed k_float_disable() API. And also add empty arch_float_enable()
into each architectures that have arch_float_disable(). The arc and
riscv already implemented arch_float_enable() so I do not touch
these implementations.
Motivation: Current Zephyr implementation does not allow to use FPU
on main and other system threads like as work queue. Users need to
create an other thread with K_FP_REGS for floating point programs.
Users can use FPU more easily if they can enable FPU on running
threads.
Signed-off-by: Katsuhiro Suzuki <katsuhiro@katsuster.net>
The IRQ handler has had a major changes to manage syscall, reschedule
and interrupt from user thread and stack guard.
Add userspace support:
- Use a global variable to know if the current execution is user or
machine. The location of this variable is read only for all user
thread and read/write for kernel thread.
- Memory shared is supported.
- Use dynamic allocation to optimize PMP slot usage. If the area size
is a power of 2, only one PMP slot is used, else 2 are used.
Add stack guard support:
- Use MPRV bit to force PMP rules to machine mode execution.
- IRQ stack have a locked stack guard to avoid re-write PMP
configuration registers for each interruption and then win some
cycle.
- The IRQ stack is used as "temporary" stack at the beginning of IRQ
handler to save current ESF. That avoid to trigger write fault on
thread stack during store ESF which that call IRQ handler to
infinity.
- A stack guard is also setup for privileged stack of a user thread.
Thread:
- A PMP setup is specific to each thread. PMP setup are saved in each
thread structure to improve reschedule performance.
Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com>
Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
Adds the necessary bits to initialize TLS in the stack
area and sets up CPU registers during context switch.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The core kernel computes the initial stack pointer
for a thread, properly aligning it and subtracting out
any random offsets or thread-local storage areas.
arch_new_thread() no longer needs to make any calculations,
an initial stack frame may be placed at the bounds of
the new 'stack_ptr' parameter passed in. This parameter
replaces 'stack_size'.
thread->stack_info is now set before arch_new_thread()
is invoked, z_new_thread_init() has been removed.
The values populated may need to be adjusted on arches
which carve-out MPU guard space from the actual stack
buffer.
thread->stack_info now has a new member 'delta' which
indicates any offset applied for TLS or random offset.
It's used so the calculations don't need to be repeated
if the thread later drops to user mode.
CONFIG_INIT_STACKS logic is now performed inside
z_setup_new_thread(), before arch_new_thread() is called.
thread->stack_info is now defined as the canonical
user-accessible area within the stack object, including
random offsets and TLS. It will never include any
carved-out memory for MPU guards and must be updated at
runtime if guards are removed.
Available stack space is now optimized. Some arches may
need to significantly round up the buffer size to account
for page-level granularity or MPU power-of-two requirements.
This space is now accounted for and used by virtue of
the Z_THREAD_STACK_SIZE_ADJUST() call in z_setup_new_thread.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
MISRA-C wants the parameter names in a function implementaion
to match the names used by the header prototype.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
arch_new_thread() passes along the thread priority and option
flags, but these are already initialized in thread->base and
can be accessed there if needed.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The optional SOC_CONTEXT carries processor state registers that need to
be initialized properly to avoid uninitialized memory read as processor
state.
In particular on the RV32M1 the extra soc context stores a state for
special loop instructions, and loading non zero values will have the
core assume it is in a loop.
Signed-off-by: Karsten Koenig <karsten.koenig.030@gmail.com>
This commit renames the Kconfig `FP_SHARING` symbol to `FPU_SHARING`,
since this symbol specifically refers to the hardware FPU sharing
support by means of FPU context preservation, and the "FP" prefix is
not fully descriptive of that; leaving room for ambiguity.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
This commit renames the Kconfig `FLOAT` symbol to `FPU`, since this
symbol only indicates that the hardware Floating Point Unit (FPU) is
used and does not imply and/or indicate the general availability of
toolchain-level floating point support (i.e. this symbol is not
selected when building for an FPU-less platform that supports floating
point operations through the toolchain-provided software floating point
library).
Moreover, given that the symbol that indicates the availability of FPU
is named `CPU_HAS_FPU`, it only makes sense to use "FPU" in the name of
the symbol that enables the FPU.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
This change adds full shared floating point support for the RISCV
architecture with minimal impact on threads with floating point
support not enabled.
Signed-off-by: Corey Wharton <coreyw7@fb.com>
This operation is formally defined as rounding down a potential
stack pointer value to meet CPU and ABI requirments.
This was previously defined ad-hoc as STACK_ROUND_DOWN().
A new architecture constant ARCH_STACK_PTR_ALIGN is added.
Z_STACK_PTR_ALIGN() is defined in terms of it. This used to
be inconsistently specified as STACK_ALIGN or STACK_PTR_ALIGN;
in the latter case, STACK_ALIGN meant something else, typically
a required alignment for the base of a stack buffer.
STACK_ROUND_UP() only used in practice by Risc-V, delete
elsewhere.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
