Allow SoC to implement their custom per-core initialization function by
selecting `CONFIG_SOC_PER_CORE_INIT_HOOK` and implement
`soc_per_core_init_hook()`.
Signed-off-by: Maxim Adelman <imax@meta.com>
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
For code clarity, remove unnecessary `return` statements
in functions with a void return type they don't affect control flow.
Signed-off-by: Pisit Sawangvonganan <pisit@ndrsolution.com>
This adds a kconfig to enable making the interrupts
non-preemptible by other interrupts. Enabling this will set
the INTLEVEL to the max non-debug level before clearing
the EXCM bit.
Signed-off-by: Christopher J. Champagne <christopher.j.champagne@intel.com>
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Newer Xtensa toolchain needs to include xtensa-types.h so that
macros in tie.h can be used without compilation errors.
The exact verison needing this is unknown but first encountered
in RJ-2023.2. Tested with older toolchain and that did not
cause any compilation errors so just include xtensa-types.h
if xt-clang is used. Haven't seen newer toolchains being
generated with xcc, so skip that for now.
Note that Zephyr SDK and the public HAL in Zephyr do not provide
this file.
Signed-off-by: Anthony Giardina <anthony.giardina@intel.com>
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
arch_kernel_init() was misused for all architecture initialization code
that is done in prep_c and prior to cstart on other architectures.
arch_kernel_init() is late in the init process and comes after EARLY
init level, making xtensa have a very special boot path.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Check that the stack frame pointer is valid before dumping
any registers while handling exceptions. If the pointer is
invalid, anything it points to will probably be also be
invalid. Accessing them may result in another access
violation.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
There actually is no triple faults on Xtensa. Once PS.EXCM is
set, it keeps going through double exception vector for any
new exceptions. However, our exception code needs to unmask
PS.EXCM to enable register window operations. So after that,
any new exceptions will go through the kernel or user vectors
depending on PS.UM. If there is continuous faults, it may
keep ping-ponging between double and kernel/user exception
vectors that may never get resolved. Since we stash DEPC
during double exception, and the stashed one is only cleared
once the double exception has been processed, we can use
the stashed DEPC value to detect if the next exception could
be considered a triple fault. If such a case exists, simply
jump to an infinite loop, or quit the simulator, or invoke
debugger.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds a new function xtensa_mem_kernel_has_access() to
determine if a memory region can be accessed by kernel threads.
This allows checking for valid mapped memory before accessing
them to avoid relying on page faults to detect invalid access.
Also fixed an issue with arch_buffer_validate() on MPU where
it may return okay even if the incoming memory region has no
corresponding entry in the MPU table.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
If there are any TLB misses during interrupt handling,
the user, kernel and double exception vector will be triggered
for the miss and the DEPC and EXCCAUSE overwritten as the TLB
missse are be handled in the assembly code and execution
returned to the original vector code. Because of this, both
DEPC and EXCCAUSE being read in the C handler are not the ones
that triggered the original exception (for example, level-1
interrupt). So stash both DEPC and EXCCAUSE such that
the original cause of exception is visible in the C handler.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Make `struct arch_esf` compulsory for all architectures by
declaring it in the `arch_interface.h` header.
After this commit, the named struct `z_arch_esf_t` is only used
internally to generate offsets, and is slated to be removed
from the `arch_interface.h` header in the future.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Namespaced the generated headers with `zephyr` to prevent
potential conflict with other headers.
Introduce a temporary Kconfig `LEGACY_GENERATED_INCLUDE_PATH`
that is enabled by default. This allows the developers to
continue the use of the old include paths for the time being
until it is deprecated and eventually removed. The Kconfig will
generate a build-time warning message, similar to the
`CONFIG_TIMER_RANDOM_GENERATOR`.
Updated the includes path of in-tree sources accordingly.
Most of the changes here are scripted, check the PR for more
info.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
We can use some extra bits available for SW implementation to
save original permissions and avoid duplicating the kernel page tables
for the default memory domain.
Whe duplicating the page table to a new domain we just ensure
to restore the original map.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
A workaround to avoid icache corruption was added in commit be881d4cf2
("arch: xtensa: add isync to interrupt vector").
This patch implements a different workaround by adding custom logic to
idle entry on affected Intel ADSP platforms. To safely enter "waiti"
when clock gating is enabled, we need to ensure icache is both unlocked
and invalidated upon entry.
Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Simple rename to align the kernel naming scheme. This is being
used throughout the tree, especially in the architecture code.
As this is not a private API internal to kernel, prefix it
appropriately with K_.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
arch_interface.h is for architecture and should not be
under sys/. So move it under include/zephyr/arch/.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
On Intel ADSP platforms, additional "isync" is needed in interrupt
vector to synchronize icache when core is woken up from deeper
sleep state by an interrupt. This is only needed if DSP clock
gating is enabled.
Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Adds the necessary code required to unconditionally save/restore the
HiFi AE registers. The macros xchal_cp1_load and xchal_cp1_store
are defined in the Xtensa HAL.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Updates the xtensa_irq_base_save_area structure to include space
for saving/restoring the HiFi AudioEngine registers used by CP1.
The starting address of these HiFi AE registers also needs to be
referenced from assembly, so it is added to the set of symbols
symbols for which we need an offset to be auto-generated.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Some Xtensa cores do not support NMI, so XCHAL_HAVE_NMI=0 and
XCHAL_NMILEVEL won't be defined at all causing
arch/xtensa/include/xtensa-asm2-s.h to throw compilation error.
Fixes: #67855
Signed-off-by: Maciej Kusio <maciejkusio@meta.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
There is no need to sync in every xtlb invalidation. Sync only
after all tlb autofill ways invalidation.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
wsr.UPPERCASE can lead to compiler errors when UPPERCASE matches
a macro defined in the special register header file.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
rsr.UPPERCASE can lead to compiler errors when UPPERCASE matches
a macro defined in the special register header file.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
This follows the idea to remove any z_ prefix. Since MMU has
a large number of these, separate out these changes into one
commit to ease review effort.
Since these are no longer have z_, these need proper doxygen
doc. So add them too.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
z_xtensa_dump_stack() and z_xtensa_exccause() are both arch
internal functions that should not be exposed in public API.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Header files under arch/xtensa/include are considered internal
to architecture. There is really no need for two places to
house architecture internal header files.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Simply to provide some consistencies on file naming under
arch/xtensa.
These are all internally used files and are not public.
So there is no need to provide a deprecation path for
them.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
xtensa-asm2.h only contains the function declaration of
xtensa_init_stack() which is only used in one file. So
make the actual implementation a static function in that
file. Also there is really no need to expose stack init
function as arch public API. So remove xtensa-asm2.h.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Andy Ross re-implementation of MMU layer with some subtle changes,
like re-using existent macros, fix page table cache property when
direct mapping it in TLB.
From Andy's original commit message:
This is a reworked MMU layer, sitting cleanly below the page table
handling in the OS. Notable differences from the original work:
+ Significantly smaller code and simpler API (just three functions to
be called from the OS/userspace/ptable layer).
+ Big README-MMU document containing my learnings over the process, so
hopefully fewer people need to go through this in the future.
+ No TLB flushing needed. Clean separation of ASIDs, just requires
that the upper levels match the ASID to the L1 page table page
consistently.
+ Vector mapping is done with a 4k page and not a 4M page, leading to
much more flexibility with hardware memory layout. The original
scheme required that the 4M region containing vecbase be mapped
virtually to a location other than the hardware address, which makes
confusing linkage with call0 and difficult initialization
constraints where the exception vectors run at different addresses
before and after MMU setup (effectively forcing them to be PIC
code).
+ More provably correct initialization, all MMU changes happen in a
single asm block with no memory accesses which would generate a
refill.
Signed-off-by: Andy Ross <andyross@google.com>
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
During arch_kernel_init(), the interrupt stack is being
initialized. However, if the current in-use stack is
the interrupt stack, it would wipe all the data up to
that point in stack, and might result in crash. So skip
initializing the interrupt stack if the current stack
pointer is within the boundary of interrupt stack.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Swap page tables at exit of exception handler if we are going to
be restored to another thread context. Or else we would be using
the outgoing thread's page tables which is not going to work
correctly due to mapping and permissions.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When MMU is enabled, we need some scratch registers to preload
page table entries. So update gen_zsr.py to that.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Userspace support for Xtensa architecture using Xtensa MMU.
Some considerations:
- Syscalls are not inline functions like in other architectures because
some compiler issues when using multiple registers to pass parameters
to the syscall. So here we have a function call so we can use
registers as we need.
- TLS is not supported by xcc in xtensa and reading PS register is
a privileged instruction. So, we have to use threadptr to know if a
thread is an user mode thread.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Simplify the logic around xtensa_mmu_init.
- Do not have a different path to init part of kernel
- Call xtensa_mmu_init from C
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
arch_switch() is basically an alias to xtensa_switch() so
we can mark arch_switch() as ALWAYS_INLINE to avoid another
function call, especially when no optimization is used when
debugging.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds code to always map data TLB for VECBASE so that
we would be dealing with fewer data TLB misses during
exception handling. With VECBASE always mapped, there is
no need to pre-load anymore.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This moves the TLB miss handling to the C exception handler.
This also allows us to handle page faults (for example,
unmapped pages) during this time as any more exceptions
handled in the C handler will not trigger the double
exception handler but the same C handler.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Initial support for Xtensa MMU version 3. It is using a two level page
table based on fact that the page table is in the virtual space. Only
the top level (page directory) is wired mapped in the TLB to avoid
second level page miss.
The mapped memory is completely fragmented in multiple sections, maybe
we find a better way in future.
The exception handler is where we effectively map the memory, the way it
works is:
1) SW try to access some memory address
2) The address is not mapped, so the MMU will try the auto-refill,
looking the page table
3) The page table contents is not mapped (remember, just the top-level page
is mapped)
4) An exception will be triggered, in the exception we try to read the
portion of the page table that maps the original address
5) The address is not mapped, so the MMU will try again the auto-refill.
This time though, the address is mapped by the top level page that is
properly mapped. (The top-level page maps the page table itself).
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
The "cross stack call" mechanism has intermediate states where the
stack frames are not valid for our own interrupt entry code, which
causes corruption if an interrupt races at exactly the right time.
Leave interrupts masked until just before the call.
The fix is midly complicated by the fact that we RELY on nested window
exception frames to spill registers from the interruptee, so have to
do the masking with PS.INTLEVEL, which requires a register to save its
contents, which we don't have since everything needs to happen in one
4-register window. But thankfully our Zephyr-reserved EPS register is
guaranteed to be available through this process.
Fixes#57009
Signed-off-by: Andy Ross <andyross@google.com>
This adds some structs for interrupt stack frames to make it
easier to access individual elements, and ultimately getting
rid of magic array element numbers in the code. Hopefully,
this would aid in debugging where you can view the whole
struct in debugger.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Save FP user register and FP register file during context switch.
This change enables shared FP registers mode using CONFIG_FPU_SHARING.
Since there is no lazy stacking, the FPU registers will be saved regardless
of whether floating point calculations are performed in the threads when
CONFIG_FPU_SHARING is enabled. This require 72 additional bytes in the
stack memory.
Signed-off-by: Lucas Tamborrino <lucas.tamborrino@espressif.com>
The Xtensa arch has historically had state/user register accessor
macros with bare three-byte symbol names. I think this might have
been in the original Cadence-contributed arch integration, but I'm not
sure. In any case they also exist in the same names in vendor
HAL/toolchain code and are causing collisions. We never should have
had these symbols exposed in our header.
Put them under an XTENSA_ prefix to decollide.
Signed-off-by: Andy Ross <andyross@google.com>
The simulator seems to drop garbage addresses (somewhere in the ROM it
looks like) into this SR at arbitrary times. I don't know if this is
a hardware exception handler that we can't turn off, or a simulator
bug, or what. But our code that assumes it will be cleared to zero or
valid is breaking. Set it every time in every context switch for now
pending someone figuring out what's going wrong.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This commit corrects all `extern K_KERNEL_STACK_ARRAY_DEFINE` macro
usages to use the `K_KERNEL_STACK_ARRAY_DECLARE` macro instead.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
