This reverts commit 331442f059.
Change was introduced because it was failing on slow core
(nrf54h20/cpuppr) however it turned out that there was a bug in the
driver and now test passes without this tweak which make the test
last twice longer.
Signed-off-by: Krzysztof Chruściński <krzysztof.chruscinski@nordicsemi.no>