Currently we are using mxcsr register with the bit 6 DAZ enabled.
When the denormals-are-zeros flag is set, the processor
converts all denormal source operands to a zero with the sign
of the original operand before performing any computations on them.
It causes bugs in the SIMD XMM registers computation like #38646
I suggest to disable Denormals-Are-Zeros flag and mask division-by-zero
exception.
Set value to the default 1F80H according to the Intel(R) 64 and IA-32
Architectures Software Developer's Manual.
Fix will let all x86 boards perform SIMD computation using XMM
registers in the correct way.
Fixes#38646
Signed-off-by: Maksim Masalski <maksim.masalski@intel.com>
Cleanup and preparation commit for linker script generator.
Zephyr linker scripts provides start and end symbols for each larger
areas in the linker script.
The symbols _image_text_start and _image_text_end sometimes includes
linker/kobject-text.ld. This mean there must be both the regular
__text_start and __text_end symbols for the pure text section, as well
as <group>_start and <group>_end symbols.
The symbols describing the text region which covers more than just the
text section itself will thus be changed to:
_image_text_start -> __text_region_start
_image_text_end -> __text_region_end
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Cleanup and preparation commit for linker script generator.
Zephyr linker scripts provides start and end symbols for each larger
areas in the linker script.
The symbols _image_rom_start and _image_rom_end corresponds to the group
ROMABLE_REGION defined in the ld linker scripts.
The symbols _image_rodata_start and _image_rodata_end is not placed as
independent group but covers common-rom.ld, thread-local-storage.ld,
kobject-rom.ld and snippets-rodata.ld.
This commit align those names and prepares for generation of groups in
linker scripts.
The symbols describing the ROMABLE_REGION will be renamed to:
_image_rom_start -> __rom_region_start
_image_rom_end -> __rom_region_end
The rodata will also use the group symbol notation as:
_image_rodata_start -> __rodata_region_start
_image_rodata_end -> __rodata_region_end
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Current location options for linker source files includes init and
noinit ram data, but only a noinit ram section. This makes it impossible
for application code to define an initialized RAM output section,
such as with the Z_ITERABLE_SECTION_RAM() helpers.
Adding a DATA_SECTIONS linker source option for this use case.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Due to the use of gperf to generate hash table for kobjects,
the addresses of these kobjects cannot change during the last
few phases of linking (especially between zephyr_prebuilt.elf
and zephyr.elf). Because of this, the gperf generated data
needs to be placed at the end of memory to avoid pushing symbols
around in memory. This prevents moving these generated blocks
to earlier sections, for example, pinned data section needed
for demand paging. So create placeholders for use in
intermediate linking to reserve space for these generated blocks.
Due to uncertainty on the size of these blocks, more space is
being reserved which could result in wasted space. Though, this
retains the use of hash table for faster lookup.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
We need to do a few things differently if we are to support
a virtual memory map, i.e. CONFIG_MMU where CONFIG_KERNEL_VM_BASE
is not the same as CONFIG_SRAM_BASE_ADDRESS.
- All sections must be specified with a VMA and LMA, where
VMA is the virtual address and LMA is the physical memory
location.
- All sections must be specified with ALIGN_WITH_INPUT to
keep VMAs and LMAs synchronized
To do this, the existing linker macros need some adjustment:
- GROUP_LINK_IN undefined when CONFIG_KERNEL_VM_BASE is not
the same as CONFIG_SRAM_BASE_ADDRESS.
- New macro GROUP_ROM_LINK_IN for text/rodata sections
- New macro GROUP_NOLOAD_LINK_IN for bss/noinit sections
- Implicit ALIGN_WITH_INPUT for all sections
GROUP_FOLLOWS_AT is unused anywhere in the kernel for years
now and has been removed.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
We will use this to map the kernel instead of all RAM.
The end of the kernel is always page-aligned, regardless
of CONFIG_SRAM_REGION_PERMISSIONS as it must be mapped.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
We will use this to map the kernel instead of all RAM.
The end of the kernel is always page-aligned, regardless
of CONFIG_SRAM_REGION_PERMISSIONS as it must be mapped.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
1. Exclude the CODE UNREACHABLE line while generating coverage report.
2. Exclude the memory domain deprecated API when calculating code
coverage.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
x86 and x86_64 require certain alignment in the k_thread struct
since the buffer to save/restore FPU/SSE registers requires
strict alignment.
Fixes#29589Fixes#29629
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Fix some issues in linker.ld when enabling coverage report feature
for qemu_x86_64 platform:
1. Fix the gcov record data should not be in BSS section.
2. Fix some test case crash due to lack of memory page align before
gcov rodata.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
And implement the support for intel64 which is basically the
architecture that will require it for now.
Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
There are x86 platforms where the IRQ configuration register for PCIe
is not pre-populated and the OS needs to assign a number dynamically
by writing to the register.
In order to allocate interrupts we have to know which ones have been
hard-coded in device tree. We accomplish this by collecting these
values through the IRQ_CONNECT() macro and placing them in a dedicated
linker section (in ROM).
The full set of allocated interrupts are managed through a bitmap, and
the pre-allocated values (from the linker section) are inserted into
this upon initial runtime access.
This patch introduces a new pcie_alloc_irq() API that drivers can use
to allocate interrupt line numbers. The two in-tree drivers that were
using this API (I2C and UART) are converted to use the new API.
Signed-off-by: Johan Hedberg <johan.hedberg@intel.com>
We provide an option for low-memory systems to use a single set
of page tables for all threads. This is only supported if
KPTI and SMP are disabled. This configuration saves a considerable
amount of RAM, especially if multiple memory domains are used,
at a cost of context switching overhead.
Some caching techniques are used to reduce the amount of context
switch updates; the page tables aren't updated if switching to
a supervisor thread, and the page table configuration of the last
user thread switched in is cached.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
- z_x86_userspace_enter() for both 32-bit and 64-bit now
call into C code to clear the stack buffer and set the
US bits in the page tables for the memory range.
- Page tables are now associated with memory domains,
instead of having separate page tables per thread.
A spinlock protects write access to these page tables,
and read/write access to the list of active page
tables.
- arch_mem_domain_init() implemented, allocating and
copying page tables from the boot page tables.
- struct arch_mem_domain defined for x86. It has
a page table link and also a list node for iterating
over them.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Adds the necessary bits to initialize TLS in the stack
area and sets up CPU registers during context switch.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The x86 paging code has been rewritten to support another paging mode
and non-identity virtual mappings.
- Paging code now uses an array of paging level characteristics and
walks tables using for loops. This is opposed to having different
functions for every paging level and lots of #ifdefs. The code is
now more concise and adding new paging modes should be trivial.
- We now support 32-bit, PAE, and IA-32e page tables.
- The page tables created by gen_mmu.py are now installed at early
boot. There are no longer separate "flat" page tables. These tables
are mutable at any time.
- The x86_mmu code now has a private header. Many definitions that did
not need to be in public scope have been moved out of mmustructs.h
and either placed in the C file or in the private header.
- Improvements to dumping page table information, with the physical
mapping and flags all shown
- arch_mem_map() implemented
- x86 userspace/memory domain code ported to use the new
infrastructure.
- add logic for physical -> virtual instruction pointer transition,
including cleaning up identity mappings after this takes place.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This had been copy-pasted between linker scripts, create
a central header for it.
The linker scripts for xtensa and posix have very different
structure and have been left alone.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Previously, DTS specification of physical RAM bounds did not
correspond to the actual bounds of system RAM as the first
megabyte was being skipped.
There were reasons for this - the first 1MB on PC-like systems
is a no-man's-land of reserved memory regions, but we need DTS
to accurately capture physical memory bounds.
Instead, we introduce a config option which can apply an offset
to the beginning of physical memory, and apply this to the "RAM"
region defined in the linker scripts.
This also fixes a problem where an extra megabyte was being
added to the size of system RAM.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Remove Kconfig, linker script, and related bits associated with
CUSTOM_RODATA_LD, CUSTOM_RWDATA_LD, CUSTOM_SECTIONS_LD,
SOC_NOINIT_LD, SOC_RODATA_LD, and SOC_RWDATA_LD options that have been
deprecated since Zephyr 2.2.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Rework x86 linker scripts to use DT_REG_ADDR/DT_REG_SIZE on
DT_CHOSEN(zephyr_sram) and DT_CHOSEN(zephyr_flash). As part of this
we remove the dts_fixup.h. Using DT_REG_SIZE means we don't have to
adjust the sizes by 1024.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
This operation is formally defined as rounding down a potential
stack pointer value to meet CPU and ABI requirments.
This was previously defined ad-hoc as STACK_ROUND_DOWN().
A new architecture constant ARCH_STACK_PTR_ALIGN is added.
Z_STACK_PTR_ALIGN() is defined in terms of it. This used to
be inconsistently specified as STACK_ALIGN or STACK_PTR_ALIGN;
in the latter case, STACK_ALIGN meant something else, typically
a required alignment for the base of a stack buffer.
STACK_ROUND_UP() only used in practice by Risc-V, delete
elsewhere.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The callee-saved registers have been separated out and will not
be saved/restored if exception debugging is shut off.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Implement a set of per-cpu trampoline stacks which all
interrupts and exceptions will initially land on, and also
as an intermediate stack for privilege changes as we need
some stack space to swap page tables.
Set up the special trampoline page which contains all the
trampoline stacks, TSS, and GDT. This page needs to be
present in the user page tables or interrupts don't work.
CPU exceptions, with KPTI turned on, are treated as interrupts
and not traps so that we have IRQs locked on exception entry.
Add some additional macros for defining IDT entries.
Add special handling of locore text/rodata sections when
creating user mode page tables on x86-64.
Restore qemu_x86_64 to use KPTI, and remove restrictions on
enabling user mode on x86-64.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
- In early boot, enable the syscall instruction and set up
necessary MSRs
- Add a hook to update page tables on context switch
- Properly initialize thread based on whether it will
start in user or supervisor mode
- Add landing function for system calls to execute the
desired handler
- Implement arch_user_string_nlen()
- Implement logic for dropping a thread down to user mode
- Reserve per-CPU storage space for user and privilege
elevation stack pointers, necessary for handling syscalls
when no free registers are available
- Proper handling of gs register considerations when
transitioning privilege levels
Kernel page table isolation (KPTI) is not yet implemented.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
We use a fixed value of 32 as the way interrupts/exceptions
are setup in x86_64's locore.S do not lend themselves to
Kconfig configuration of the vector to use.
HW-based kernel oops is now permanently on, there's no reason
to make it optional that I can see.
Default vectors for IPI and irq offload adjusted to not
collide.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Nothing too fancy here, we try as much as possible to
use the same register layout as the C calling convention.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Includes linker script fragments for the kernel object
tables and automatic memory partitions. The data section
is moved to the end per the requirements of
include/linker/kobject.h.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
We don't need to set up GDT data descriptors for setting
%gs. Instead, we use the x86 MSRs to set GS_BASE and
KERNEL_GS_BASE.
We don't currently allow user mode to set %gs on its own,
but later on if we do, we have everything set up to issue
'swapgs' instructions on syscall or IRQ.
Unused entries in the GDT have been removed.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
These were previously assumed to always be fatal.
We can't have the faulting thread's XMM registers
clobbered, so put the SIMD/FPU state onto the stack
as well. This is fairly large (512 bytes) and the
execption stack is already uncomfortably small, so
increase to 2K.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Promote the private z_arch_* namespace, which specifies
the interface between the core kernel and the
architecture code, to a new top-level namespace named
arch_*.
This allows our documentation generation to create
online documentation for this set of interfaces,
and this set of interfaces is worth treating in a
more formal way anyway.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This commit refactors kernel and arch headers to establish a boundary
between private and public interface headers.
The refactoring strategy used in this commit is detailed in the issue
This commit introduces the following major changes:
1. Establish a clear boundary between private and public headers by
removing "kernel/include" and "arch/*/include" from the global
include paths. Ideally, only kernel/ and arch/*/ source files should
reference the headers in these directories. If these headers must be
used by a component, these include paths shall be manually added to
the CMakeLists.txt file of the component. This is intended to
discourage applications from including private kernel and arch
headers either knowingly and unknowingly.
- kernel/include/ (PRIVATE)
This directory contains the private headers that provide private
kernel definitions which should not be visible outside the kernel
and arch source code. All public kernel definitions must be added
to an appropriate header located under include/.
- arch/*/include/ (PRIVATE)
This directory contains the private headers that provide private
architecture-specific definitions which should not be visible
outside the arch and kernel source code. All public architecture-
specific definitions must be added to an appropriate header located
under include/arch/*/.
- include/ AND include/sys/ (PUBLIC)
This directory contains the public headers that provide public
kernel definitions which can be referenced by both kernel and
application code.
- include/arch/*/ (PUBLIC)
This directory contains the public headers that provide public
architecture-specific definitions which can be referenced by both
kernel and application code.
2. Split arch_interface.h into "kernel-to-arch interface" and "public
arch interface" divisions.
- kernel/include/kernel_arch_interface.h
* provides private "kernel-to-arch interface" definition.
* includes arch/*/include/kernel_arch_func.h to ensure that the
interface function implementations are always available.
* includes sys/arch_interface.h so that public arch interface
definitions are automatically included when including this file.
- arch/*/include/kernel_arch_func.h
* provides architecture-specific "kernel-to-arch interface"
implementation.
* only the functions that will be used in kernel and arch source
files are defined here.
- include/sys/arch_interface.h
* provides "public arch interface" definition.
* includes include/arch/arch_inlines.h to ensure that the
architecture-specific public inline interface function
implementations are always available.
- include/arch/arch_inlines.h
* includes architecture-specific arch_inlines.h in
include/arch/*/arch_inline.h.
- include/arch/*/arch_inline.h
* provides architecture-specific "public arch interface" inline
function implementation.
* supersedes include/sys/arch_inline.h.
3. Refactor kernel and the existing architecture implementations.
- Remove circular dependency of kernel and arch headers. The
following general rules should be observed:
* Never include any private headers from public headers
* Never include kernel_internal.h in kernel_arch_data.h
* Always include kernel_arch_data.h from kernel_arch_func.h
* Never include kernel.h from kernel_struct.h either directly or
indirectly. Only add the kernel structures that must be referenced
from public arch headers in this file.
- Relocate syscall_handler.h to include/ so it can be used in the
public code. This is necessary because many user-mode public codes
reference the functions defined in this header.
- Relocate kernel_arch_thread.h to include/arch/*/thread.h. This is
necessary to provide architecture-specific thread definition for
'struct k_thread' in kernel.h.
- Remove any private header dependencies from public headers using
the following methods:
* If dependency is not required, simply omit
* If dependency is required,
- Relocate a portion of the required dependencies from the
private header to an appropriate public header OR
- Relocate the required private header to make it public.
This commit supersedes #20047, addresses #19666, and fixes#3056.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Move _LINKER and _ASMLANGUAGE to target.cmake because of how we pick the
linker script that might be used. This way regardless of how or where a
linker.ld gets included we will always set _LINKER & _ASMLANGUAGE (so
any header that needs check based on those defines they can,
specifically generated_dts_board.h)
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Same snippet as in our 32-bit linker file. Creates
an iterable section with all the MMU_BOOT_REGION() info.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Some linker symbols indicating the bounds of various memory
areas needed for setting up boot MMU regions are now in
place.
MMU_BOOT_REGION macros added to align the bounds of these
sections if we're using runtime page tables.
_image_rodata_start moved to the proper place, it was not
accounting for the rodata sections specified in
linker/common_rom.ld.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Program text, rodata, and data need different MMU
permissions. Split out rodata and data from the program
text, updating the linker script appropriately.
Region size symbols added to the linker script, so these
can later be used with MMU_BOOT_REGION().
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This is really just to facilitate CPU bootstrap code between
the BSP and the APs, moving the clear operation out of the way.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
