The pending_sqe logic to track where in the ring queue the concurrent
executor had left off was slightly flawed. It didn't account for starting
all sqes in the queue and ending back up at the beginning.
Instead track the last SQE in the queue, from which the next one in the
queue will the one to start next.
If we happen to sweep the last known SQE in the queue, reset it to NULL
so the next time prepare is called we start at the beginning of the queue
again.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Transactional submissions treat a sequence of sqes as a single atomic
submission given to a single iodev and expect as a reply a single
completion.
This is useful for scatter gather like APIs that exist in Zephyr already
for I2C and SPI.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
The test suites have grown to cover different units really and having
them in one file was becoming a bit much to scroll around in.
Coincidentally found a accidental reuse of a define between the spsc and
mpsc tests.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Race was possible though very unlikely between the atomic cas
and queue push/pop operations. The outcome of the race had it shown up
would have been a submission not worked on due to the timer never being
started. A small critical section fixes this and clarifies where the single
conumer part of the mpsc queue comes in despite there being multiple
contexts which may enter that section.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Noticed the tests were a bit verbose, saw a few stray printks. Drop those
as they aren't really needed and potentially cause testing issues, printk
is a potential synchronization point.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Renode platform fails the test despite it working well on qemu riscv.
Ignore this particular platform for now.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
By using an mpsc queue for each iodev, the iodev itself is shareable across
contexts. Since its lock free, submits may occur even from an ISR context.
Rather than a fixed size queue, and with it the possibility of running
out of pre-allocated spots, each iodev now holds a wait-free mpsc
queue head.
This changes the parameter of iodev submit to be a struct containing 4
pointers for the rtio context, the submission queue entry, and the mpsc
node for the iodevs submission queue.
This solves the problem involving busy iodevs working with real
devices. For example a busy SPI bus driver could enqueue, without locking,
a request to start once the current request is done.
The queue entries are expected to be owned and allocated by the
executor rather than the iodev. This helps simplify potential
tuning knobs to one place, the RTIO context and its executor an
application directly uses.
As the test case shows iodevs can operate effectively lock free
with the mpsc queue and a single atomic denoting the current task.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Adds a lock free/wait free MPSC queue to the rtio subsystem.
While the SPSC ring queue is fast and cache friendly it doesn't
work for all scenarios. Particularly the case where multiple rtio contexts
are attempting to work with a single iodev. An MPSC queue works perfectly
in this scenario.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
When a libc partition exists, all user mode threads will need access to
libc variables. Add the libc partition in this test case to allow that.
This was detected by running the test on ARM32 with thread local storage
enabled as that uses z_arm_tls_ptr which is included in the libc partition.
Signed-off-by: Keith Packard <keithp@keithp.com>
Add support for userspace with RTIO by making rtio and rtio_iodev
k_objects. As well as adding three syscalls for copying in submissions,
copying out completions, and starting tasks with submit.
For the small devices Zephyr typically runs on one of the most important
attributes tends to be low memory usage. To maintain the low footprint of
RTIO and its current executor implementations the rings are not shared with
userspace. Sharing the rings it turns out would require copying submissions
before working with them to avoid TOCTOU issues.
The API could still support shared rings in the future so that a
kernel thread could directly poll, copy, verify, and start the submitted
work. This would require a third executor implementation that maintains its
own copy of submissions similiar to how io_uring in Linux works.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Add a bunch of missing "zephyr/" prefixes to #include statements in
various test and test framework files.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Schedules I/O chains in the same order as they arrive providing a fixed
amount of concurrency. The low memory cost comes at the cost of some
computational cost that is likely to be acceptable with small amounts
of concurrency.
The code cost is about 4x higher than the simple linear executor
which isn't entirely unexpected as the logic requirements are quite a bit
more than doing the next thing in the queue.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
A DMA friendly Stream API for zephyr. Based on ideas from io_uring
and iio, a queue based API for I/O operations.
Provides a pair of fixed length ringbuffer backed queues for submitting
I/O requests and recieving I/O completions. The requests may be chained
together to ensure the next operation does not start until the current
one is complete.
Requests target an abstract rtio_iodev which is expected to wrap all
the hardware particulars of how to perform the operation. For example
with a SPI bus device, a description of what a read, and write mean
can be decided by the iodev wrapping a particular device
hanging off of a SPI controller.
The queue pair are submitted to an executor which may be a simple
inplace looping executor done in the callers execution context
(thread/stack) but other executors are expected. A threadpool executor
might for example allow for concurrent request chains to execute in
parallel. A DMA executor, in conjunction with DMA aware iodevs
would allow for hardware offloading of operations going so far as to
schedule with priority using hardware arbitration.
Both the iodev and executor are definable by a particular
SoC, meaning they can work in conjuction to perform IO operations
using a particular DMA controller or methodology if desired.
The application decides entirely how large the queues are, where
the buffers to read/write come from (some executors
may have particular demands!), and which executor to submit
requests to.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
