014a0983fb
Additional logging of kernel.timer.timer_behavior_external test case statistics for timer drift, variance, etc. as JSON-formatted records to make easier data collection and its further analysis. These log records will be processed by the Twister Harness recording feature which captures and parses timer statistics from the log output, then composes it into twister.json and recording.csv files. Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
87 lines
3.6 KiB
Plaintext
87 lines
3.6 KiB
Plaintext
# Test a timer implementations variance and long term drift
|
|
|
|
Records and calculates statistical values against a timer validating that.
|
|
|
|
1. Timer variance and standard deviation is below defined acceptable values.
|
|
2. Periodic timers do not drift in either direction from expected total time.
|
|
|
|
Timers are meant to be precise and accurate. This test validates an implementation is both.
|
|
|
|
--------------------------------------------------------------------------------
|
|
|
|
External tool testing
|
|
|
|
It's also possible to use an external tool, such as a logic analyzer, to
|
|
collect samples. The "kernel.timer.timer_behavior_external" test will toggle a
|
|
GPIO pin on every cycle - first cycle will be a rising edge. This test expects
|
|
a Python module to interface with the external tool, which will provide the
|
|
necessary statistics that the test will use to assert the test status.
|
|
|
|
GPIO pin
|
|
|
|
To enable external tool testing on a board, it must provide the compatible
|
|
"test-kernel-timer-behavior-external", with property "timeout-gpios" being the
|
|
GPIO pin that will be toggled each period.
|
|
|
|
External tool interface
|
|
|
|
In order to get data from the external tool, the test expects a Python module,
|
|
named on testcase.yaml, with the following interface:
|
|
|
|
run(seconds: float, options: str) -> {}, int
|
|
|
|
The `seconds` parameter defines for how long the data collection is expected
|
|
to run; `options` is a string defined on testcase.yaml with options known to
|
|
the external tool helper module. It should return a tuple with a dictionary
|
|
for the following time statistics, in seconds:
|
|
|
|
'mean': Mean time of each period
|
|
'stddev': Standard deviation from the mean time
|
|
'var': Variance from the mean time
|
|
'min': Minimum period registered
|
|
'max': Maximum period registered
|
|
'total_time': Total time, between first and last period.
|
|
|
|
and an integer value of how many data values are collected for the statistics.
|
|
|
|
Note that the collection may need to go a bit after the "seconds" parameter,
|
|
to account for expected drift in the test and between the DUT and the external
|
|
tool.
|
|
|
|
One can check `pytest/saleae_logic2.py` file as a sample of external tool
|
|
helper module.
|
|
|
|
Twister
|
|
|
|
For Twister execute the external tool testing, the fixture "gpio_timerout"
|
|
must be available on the device. Also, testcase.yaml
|
|
"harness_config/pytest_args" from "kernel.timer.timer_behavior_external" must
|
|
have parameters "tool", with the name of a loadable Python module and
|
|
"tool-options", a string with options passed to the Python module helper.
|
|
|
|
Check testcase.yaml for an example using the saleae_logic2 module.
|
|
|
|
Configuration options
|
|
|
|
At its heart, the external testing is actually comparing two clocks: one on
|
|
the board under test and one at the external tool (Zephyr implementation of
|
|
the timer also plays a role, and it's the real target of the test, but it
|
|
depends on the board's clock). Different clocks run at different speeds, so
|
|
they tend to drift. To be able to account for this drift, the external test
|
|
doesn't reuse CONFIG_TIMER_TEST_MAX_DRIFT and
|
|
CONFIG_TIMER_TEST_PERIOD_MAX_DRIFT_PERCENT, instead introducing
|
|
CONFIG_TIMER_EXTERNAL_TEST_MAX_DRIFT_PPM and
|
|
CONFIG_TIMER_EXTERNAL_TEST_PERIOD_MAX_DRIFT_PPM, that can be more finely tuned
|
|
for the hardware in question.
|
|
|
|
Also, CONFIG_TIMER_EXTERNAL_TEST_SYNC_DELAY is used to set a delay before a
|
|
test cycle starts, so that the external tool can be set up.
|
|
|
|
Dependencies
|
|
|
|
The external tool interface may have its own dependencies. For instance,
|
|
the saleae_logic2 module ones are listed at pytest/requirements-saleae.txt.
|
|
One can install them with pip (possibly in some virtual environment):
|
|
|
|
pip install -r pytest/requirements-saleae.txt
|