From 18451bca44df36e479ae4eec27b7eeb39c0b05e9 Mon Sep 17 00:00:00 2001 From: Dmitrii Golovanov Date: Sat, 19 Oct 2024 18:51:29 +0200 Subject: [PATCH] doc: twister: Update test naming and application diagram Update Test Application diagram as well as Test Scenario and Test Case naming conventions to make them more clear and aligned to Ztest suite name included as a part of Test Case name. Signed-off-by: Dmitrii Golovanov --- .../test/figures/twister_test_project.svg | Bin 134581 -> 138361 bytes doc/develop/test/twister.rst | 65 +++++++++--------- 2 files changed, 33 insertions(+), 32 deletions(-) diff --git a/doc/develop/test/figures/twister_test_project.svg b/doc/develop/test/figures/twister_test_project.svg index f21a45c2971ae19eee8886c92f90459f19fcddcf..75440d6bda601f9651a40bb42eb000ca0b37685a 100644 GIT binary patch delta 8772 zcma(%X?RsdmfyJ#NrV6al9vRC+`LFcNS=3J$f7_4vNjQL2{s`yA#5fg2?iZ$f@9l$ zb|ccIl#B~74Swp31Ng*i;|St1u5IJjGuql{gU;6iqu_$tsNFN?)Lq^SMy7v|cdF{t zsZ*!Ue%X6p_L--$cRvEdRHZ_7=d1bzQ=edHNyF9H)9{^tX$LN8b4Zc%}t%^ z{3#Q!n!;|Eek++;>Z;g#5*X}gRvw$){~`9tb6KVHn>)MYsT~d1H+Q#ntdPf+HC*3b zxw5OWQ2;Gnm33WAeLkPn&dg)Q^Qu5&b@OguuVuZ_zpjh6rnY2XDY4UIi`iWTBV($f zvEmns1}|xAt8Z-RsBh?OZ@#XJH580s>oym%+-u%ubG3YSS?*M}wP;7`FF9MYLOML} z;OJ4Eh`p?>wZ0YkNp`Mo%p4>RPtmU{x+zwsPg_wnJe?zpX(-{dq?Wj!l}UCzcrf`?;6h*58NV1>|cEs zZBKo>=-phlaC3etyW?jN5x(Go{M3oIn}BWT+?bkk!>@oHSiLbds_WN~I@0~-dF+*; zC9M3(@u_#$))ui-(?+Le@D=?e!)9>5k@}zeKFk0A?i**@jA1GH@hOs|>Zz6|a-rY+ zrOR5`W&0PW-rhTa~5>No>5A z$0j~gz`n|gr&^zRxW8oTEV)Nd&3xgaLiTCDc>bz-w`0u>v#;rPC;zCeeO2q#bC*x4 zU=v5-vko1_e!RDkjaaKDI=dP=x+ZpXv~^dvwzW1_Hn%qUqsEnO*R2q@n6rNx+dg#c z09=}ik65i#vescmY|wdw*`c2NyuQw12QQDa|7sh>3hNT=ve#Gjjihwtvrps2Y{kAL zlMDN?|JW1%dnNrTu*-Av1P|?Vve@q~8(*NRnpw7Lbz4`Zd}Fl_RM@J)wio3|s>wDz z(U9sm{2r8RiabkpTtn8=kZ#hPgL5XWTC|E#9iR9fXqzvAEPm=!n53k4sR|(r0s^$u za96D6!6@aA)ofEeC{YX&nv$KmIL2A59FAxo8IlyVG@+qHMcLlu|q!F%Bvb{ zg!xuo(pbZed{jQrc8vsHFQMx$U$q|Ui)(&hr`X1}9~;5%+6Z}k_h`sx3qOn}6btWX z8_2vv9IJ}b3p5lfyz@m|{tSke5iZUKy9|pUu`3f)*wEwRqNcbbf4>$6=SCZS^s{hC z4ikVK`D%VlckGn?^#TZ747=ml2=?JeYD{;KVlF)+3S z-C4P;ndZfniJIQVQ*}vadqZmtYxvldG?PR&gMILEQI_xU61=1giZUcL>y!9EJ$*xd z(1F?f**X}=a^Jt0o&2N_jMSN9hXH@yI?)U~_W`J48$Lc?G6knI__~|H_Jv*cc>(H3 z(s58;4v9j@+tzwPs!MX!O+qtNm#@DRX7b%zA&)tq51FhaP1$hOkcOQe*2%MdqUlwE zoIX$>I|kqWAk?#GPv=~y1!rU&sA)(s<9w5^dl7E%*>_II6I%G{rV{B&Ca77g{dB#Y zMsp(3nHeBIxCRRO8{5I*ht|S?T+%30Y!ZrLiggdy{~6faKbOQzrMEi#@-Gu5lnuUJ z=P*B*2{Rd6H-E~!`QuQ>FJ&9f6iFIs#=8(RbyAHcNFD!90?7y--TvF!95cm}F#n2pFBhOMR8>k5c`MPa1yMo={U z0Xi;mSTOW?2nNJXKmgH*H&-D`zmV6Iz!gK%Cy6z_sTRf(56U;3z1!Ubpb*Vf@pcV>!+?E%#@}adbPKa(9EgIqk zf=nB&$PM z8%yCJoHMcD+s}sy0}WRfvDHw;d%TLq?cpmN5%R4c3tvcqd}nY}=eAN!Pbw&W71D9M9FxkV zt;)J;g`3r+hT}9bCmQpf7oox1<-)&!q$%FDmGA}vLK&y>j%xTPUl&aDWT7tjFdDOC zM!I0==*a?IFf_uYC=>2MVIll|T?l4Ln6$3@x*%W_t4~uC0`|`$7%dlhAxQj+J!r2q z*Ox8~;_a8hdA>sQiIi4|MH69dhCtZL-QJLGaokZ? zX47R1xndXrwhH@cVG_iLs!AoF~?Ga9q- zpbN2h≺@#O>vntBdPDf|?NAd`Ty)M6TCwfs$T`R(QwaZm5#%;Ep@Hp>dEaLJYkr zLhQ5C1$=)A455c#A-S%HPw_{Hxd@?2ZM>i5z*csNwQV<<(Ga*20;9XyS( zL-%%Xg2#cM_!(5;I;)L%d$K1^_XR5KqPAADk@X5z6Qy$vcn)V0GP zxw^M&6QqDIy#uUBkxtwJ8n3z=A5dtM;9wt06qU%Q`7H+}AV;LEzA#ntAJfA6B+o}`*#bq_NkDZD7{KS# zRB!Qp@D=c*Q?Xp_xfyH7lMlc!bi|Z)I4b!!1N~bDWKiTwhp7d&E$Txb!?r|0>Zn+6 z;MQoU;*cuOXhN)prv%915Rtk15*B{g{xC%=VT|Abix{D(SPvsPM=&Dzqo|8Rin`m& zEC++FD2cFC@%^%aQ)+LRZTjy*-m?{^&sOvhov2!vXa>89YA__$F}DkcC=*p6r;ifI zn1sZ)!M&IPAX`)oIU+;0$qpjORjKx^E2DLU158!*8zp25Y0y=}Vlya!)0>E>zfAS- z^T5qpA^Ag6cO6B=IJ6#{6x((}9X`DJC_enChfw$V6Su=~?~SKm0q{#7 zL9dDZCgC+-c?4P{L*siSDej$m5?%#9W&@J-QUBpFbV``q2vq6cFwwG{Ow`9VMW}oB zpo>F|^>NXGU5Dy@^Z+EtB|Qgx+UAIX6MLbG8~4JG`RXU35wn+wJlzsL7@tU|zi+-J2C$=*WI($nj&H@G`#3$SmITd}Dw40M<@u zaCCEyHy=Qf5`@w0kcCGeri6AUIyLAblBL2vO!Av)2vZ;`JcO>gm_|g68UEl4kS+U& z*v(fp+(mYQ*x?Wjft7r*gqObv*|P;9OA!X;Vq!*^;ZQSiNLm?9WS$xzM@%4chCu~Q zsXYGWi!i>*r3%7C6L%vB+o6;`pc;}greSO|iD_*9U`C815h?~}Nu4vpCDrhkh*t~_ z{z%gWtI_x@GCpoVRMUymhe}N|>C$N$KAR!BF9bQ5A7aAgLutuYh@4nM#9)_>6}dxW z7eAWNj&MvLQTX~7p^Ja>GK?>>Y+SVP{hTx;N-UqYd>ryGq}8YjvxVo0sxVu4p0Myj zlKFhjt5B5C!u!j11}K;pnG-q+REtVplO+w~Z@miY1p>{rgj(2|uNKkJ@w;mWT%^DI zHNaLZy^nYuk$37;;`!B_0D~0$LNDiacvMtLPyH5t=_kc#yrRe-&UgL}enO?v(*J}K zKC%{yyqZI>15;zuyZkLQSYELJ2C+LI9nM>RkHIXgo{s+>)>3_o+uo-p86ocv`v)`l z6@P$RBuh|Dgjv%L;0K$~H`KiaR{$Sz7$!+!Q|#fxuwk4^~_yCG3@Iw?D3Nu6vi8XHNAD=(p+7U3?kPH}qchcGm*hvIRSgxtNXscR{A@m=qp58-uem|>@O z?kc#T5W~jzg{JBmQsXE!#%n$T@y&#!1Ktn2k8ys*-4K^h>&_g90g~ic923TW=RMhQ8^8P$*n(fi_{~S4l=nZ0 zp%2@hqQ|h~B-}GRY~h&HH>ILqPx6Kjpn*Sj3bM_xTVRs=)F;PeGbL4-4vG zj$Q`cr9g1~5Ec%}m5LA+PSDLFY5{UcT;W5%g9?s~N;RF!&XgCV@p&U`P~<>C=f$mJ-Hc9EAbA>w>dZ=;9F`Hc2ri zukriu71>e|wel~|mQINx${iqah*_BHp}Z&Z)}1hnm*q&)`L@xJJHM|_ycJrarmCf; zs=c9WsoYeZm@RsYuA!oxYpACLY3NJ+8R2ztCiIt>=SL~k#d_djaXsx(4g9{5>0x!J zPY*kxPu^6Yw+{So3i6ujgy#D+SF3IQ=kLELJhZ2<^~Y4Ir>iV|o{_G$ylT@-8HO7?OTX-0d&5mj=S^##Hhtpy zuQhL&xc-itZeAXIQ#-ENUgoBPcQkT>y89Es^eY}Mo7c2{Lvrf6##K$5)~s8R96o>E zb#v#92=>pNO>Smum&p{2SlK-&YVV#oT{vFr-@Il`{jH7b>YJMyH{7wVslIW2OVg4K zS2wO|857h!UmomReJp61-#aK7n8@ty{;k4HDsgmB<|h>wDD$WE?<+I2mRC@4YW^7| ze7x|S6fzeV-_tWV`=5!-;dOU%|IVQO+lfrjaxVoRFWi~=VB;gStA1xNEF2JgchQxZ z_6@gFQ2#(8n7yn|@W8GmyH+MLRhz3i2Nxe6k@?Zxb$x^99xe_J{Gc>b_MH`-G}j8~ zm>sP8ey_}LzyEF`_(_G4dGp}STCnx;Ucr<@7lS>|H)I}r`npc&xr$Q3wxWK)`Jv@O z!@*SM!m|%)n(qZWjtmd#mX&2*K62m+@$QFoW$<#J(xCicBJ-nP4=d07=6~OzQI?&o zO?qiNY2_r_=Wh`UOa&m(`Gy_Ct)tJ{}UP+ z{$L*MCCx~S)1}%Vvc$HZQ}4`{4|Pgh_FTpODKx6cb^XfZ=5dvlV~L+N(e&WN`9#_H z%MJ*(oxP>P)M2=5!zvhGn>2L2gJ_$(i)kpfQd61Lv^^dpHSC-KdLZTjJ<@R2%I0w7jGvx+}%Rxl|QQ zXio%x{3w!Ir<@s{3Bb z^WYti_bxr^;)wGk%VK|94y)Og#V)I#!`hBv#zXa#lXP^c!ZW$K=KSOgIdxtyE8Dqc zSs6{yVpDz4@MZTQa#pOAzib*xIV((G=*MYs(C~g$@P~htRoF2k93u;fe9;zlr)hA| z`sFY&dm?qgDkaJ6n6#etV4moR#u_RYCtf5yi;o_R4@VA^__#zTA0m#aAktmpSkc%u z3hyyi9Qcy)F0t~^1*#{TvGO))wuDMEcu`_NwCWH=Tlzp@Kti_ot zsITVoVgHc|94~j;lM5U_d)&36qKL{h+ZJPr=%+JmgbqjmPP5X0L9#BA?POZ_Q)WaG zw#j`=7gGL@GCTisclvmR3L;Fo>Ey~Xt3qKk-}@ z`J(M2CB%eMDn|_4;et|lyTCS#u(~(BscEM4(@ZhbqT!-pJM~cdt$nE9Tx^4_bW^sK zw7ke!ag+UJrqbNQYPK#N1{qssc7mk-^LOr-4(#Z#$xGGOsNeArO@1_iPdDYhzm~7cjDG+DhnSQOaW<*5sn>7_koTg>?elN#zly#+N9&#DGy8_ zeOH%j;`Fh5oqJf#^3y5Pl#MzxjFhb!Sy8z-ZUX5YU^z*Q9YKXFOTK4reyN++ z&jenLJH9`S79V^KWH#~Z)GOTR(ON1nb%|I>CUO{8*b!&*(Y`76zqQVvHSl+jGRFI zMwk{y4K7{JPPs`#Uf`bLr~EwGga0j2x1ah5J&jAH$2saPUYkILLvYs7Xd`7BurWKE zmu(B*#w577oZOhJ}(5$l#(F z7PbZQieO=ztjnAjkAdqTcYsCofxXIIj)H}fF;q@xN>8h0SVok>43^Y?9)sBAZ+XYg zxnF5_3Fb??E{}-**VD>8M4XsTn}G-cM3jZd8W56z#nm(D_8ctUo=MFGF0QFH?PcMD zS@Z_F{LHcsCz04Vhvr1ADRW;jVlK6eQkcPUrgIXm0N@CYN~_4?UPv92_h~6PKqc!r z(NNekF(yyMaE=a{xxWBb_QQr7=`M;AL}kwSo2en`%M1(g<;uFj?pQ{ z;vit%Jkj-azH7wuw~!mOohw38mBhD5G3pJ`3pct(xP1Zb#+4pj7o1j$Z!e<7&;(IOCUeEg(~u=b{(GjF(OZ# z-VL=)tLb*lw?mkKOVZcFtt$ZwF=8E67x=1Jm?1Ljs1b#N+K`VzL2amhBaMM|Tn30U z8)=DVr`0?Kz9V;lO+R!eO@hApN|vC)zqcH&`q|WRxEgwYQ(+UsZ8O8 z>rSpD5po@+JiK_rq?QB#ONK5n%LDR6uOn!>^~?>DQ>$)9&9A7y18lZ^gAQB*n@7Gy zEpP+8oPT1wpTzNQ`7Z4z@x%LQj0m2h8DiA^IKSc(a0HT;@=G9;{5n;L>sqtJ9^Xyr z7(}B5u8u)3Vc_QbInV9@Cpl2UNv(Hg6;`R)LtVv?r)d^LrCR)M4^?%9l^hlRBD>tK zy>xwyu50$uB(Ydf6(DvAaFgQChr!cW=?^3xI6`&ekv(LHmhG^u@&uKZN(aj%Q2AP= zfE<SG47$t_gIxZ-gLW8aq3)-HaGCInZ zEsSnjU-9Sxs*;!892@Gcfy-Ep<;7eic2ScTi<(@nD&=giRL-p{m5~f=KS19ZXfbbb zh2qQP)P+;?Br*Fm9#yhc=Y$Ws4VAjtQHG(ODEKULqA%)3P+u|baY>NfN^)c5{IWMh zFQD$DcF~@sUAS>_4EasX(#5`miu0ZU>sW#q<=6g}=8#BoF$v-`#PL_Ckbxr!M%I|) z4^jj2gKftUA}<_7CW@3YY(LEG`q3efk*kPi{nOwpFWh^O{)8CwBaK1NfYZPkifDfP z3^g!=cp)rlBjyc<6(1(xNmAqKBXlcg5V9Ww?eP3z=Ej}BqVbw~3z{LWItErw=eBd0 z5FL5D`LEy5Sn=`jT?-7w!~7hQ zR>8aGRq8v~;k3#DjQyO1;*WEC)EHjUN2SAWAwwO!jTmOEX5-zH8wNlH-MA@6AE&i7 zay%Caa=d!fh?|y=5sbh1L&wFGa=f;ceE}&Kx4cFp$9kOWI|wyDD`8*Ey@n17&X?8G z3P{^bA^!PE=V*!c?=#}hhAltK*u|--mgP#*a&2}*c8W+ho)Ee`Bz3F z#Kw*h!(XS#1Eq!!Cd0AwGZ`Oy+{6VCJyex*-aN7rQ-Uz~JY))Iaq*NV#u;ze@;iE* zRQ*x)d)kuCHm8fv-k`OdZNB<8g7nT=^bHg&f26c#`r(uBA}0|~yhS%gN%`UkXL>`_v>xpQgeB zpeelTefokDuHW)6W0I>H*WJEsb=AtI<_*=!w43?lVp&&qz$-!KF$&ka5G63@0;B^EY52+DtuY>zeoTDCnqSssl zq*R&QwCuJGOYuCd>*1?q+N6_rH*CKq|tbSiAHhBq{g$vFBoc6PQrNhLG+7- z`1BuC$u)iXm$VPRU&vxs6+eByqPt3Dgr{8;tjH6Ns^(dWIFBz&80> zE;}?-J4oWx6O=5#RAF&vgjr9HLt2x~Vi=%opil)Az0o+x_ErBcz}Tf08A`fF3^{{cUW8QA~; diff --git a/doc/develop/test/twister.rst b/doc/develop/test/twister.rst index 685c70ab886..8d943279a3d 100644 --- a/doc/develop/test/twister.rst +++ b/doc/develop/test/twister.rst @@ -223,57 +223,60 @@ Tests Tests are detected by the presence of a ``testcase.yaml`` or a ``sample.yaml`` files in the application's project directory. This test application -configuration file may contain one or more entries in the tests section each -identifying a test scenario. +configuration file may contain one or more entries in the ``tests:`` section each +identifying a Test Scenario. .. _twister_test_project_diagram: .. figure:: figures/twister_test_project.svg - :alt: Twister and a Test applications' project. + :alt: Twister and a Test application project. :figclass: align-center - Twister and a Test applications' project. + Twister and a Test application project. Test application configurations are written using the YAML syntax and share the same structure as samples. -A test scenario is a set of conditions or variables, defined in test scenario -entry, under which a set of test suites will be executed. Can be used -interchangeably with test scenario entry. +A Test Scenario is a set of conditions and variables defined in a Test Scenario +entry, under which a set of Test Suites will be built and executed. -A test suite is a collection of test cases that are intended to be used to test -a software program to ensure it meets certain requirements. The test cases in a -test suite are often related or meant to be executed together. +A Test Suite is a collection of Test Cases which are intended to be used to test +a software program to ensure it meets certain requirements. The Test Cases in a +Test Suite are either related or meant to be executed together. -The name of each test scenario needs to be unique in the context of the overall +The name of each Test Scenario needs to be unique in the context of the overall test application and has to follow basic rules: -#. The format of the test scenario identifier shall be a string without any spaces or +#. The format of the Test Scenario identifier shall be a string without any spaces or special characters (allowed characters: alphanumeric and [\_=]) consisting - of multiple sections delimited with a dot (.). + of multiple sections delimited with a dot (``.``). -#. Each test scenario identifier shall start with a section followed by a - subsection separated by a dot. For example, a test scenario that covers - semaphores in the kernel shall start with ``kernel.semaphore``. +#. Each Test Scenario identifier shall start with a section name followed by a + subsection names delimited with a dot (``.``). For example, a test scenario + that covers semaphores in the kernel shall start with ``kernel.semaphore``. -#. All test scenario identifiers within a ``testcase.yaml`` file need to be unique. For - example a ``testcase.yaml`` file covering semaphores in the kernel can have: +#. All Test Scenario identifiers within a ``testcase.yaml`` file need to be unique. + For example a ``testcase.yaml`` file covering semaphores in the kernel can have: * ``kernel.semaphore``: For general semaphore tests * ``kernel.semaphore.stress``: Stress testing semaphores in the kernel. -#. Depending on the nature of the test, an identifier can consist of at least - two sections: +#. The full canonical name of a Test Suite is: + ``/`` - * Ztest tests: The individual test cases in the ztest testsuite will be - concatenated by dot (``.``) to the identifier in the ``testcase.yaml`` file - generating unique identifiers for every test case in the suite. +#. Depending on the Test Suite implementation, its Test Case identifiers consist + of **at least three sections** delimited with a dot (``.``): - * Standalone tests and samples: This type of test should at least have 3 - sections concatnated by dot (``.``) in the test scenario identifier in the - ``testcase.yaml`` (or ``sample.yaml``) file. - The last section of the name shall signify the test case itself. + * **Ztest tests**: + a Test Scenario identifier from the corresponding ``testcase.yaml`` file, + a Ztest suite name, and a Ztest test name: + ``..`` + + * **Standalone tests and samples**: + a Test Scenario identifier from the corresponding ``testcase.yaml`` (or + ``sample.yaml``) file where the last section signifies the standalone + Test Case name, for example: ``debug.coredump.logging_backend``. The following is an example test configuration with a few options that are @@ -316,12 +319,10 @@ related to the sample and what is being demonstrated: tags: tests min_ram: 16 -The full canonical name for each test scenario is:``/`` +A Test Scenario entry in the ``tests:`` YAML dictionary has its Test Scenario +identifier as a key. -A test scenario entry is a a block or entry starting with test scenario -identifier in the YAML files. - -Each test scenario entry in the test application configuration can define the +Each Test Scenario entry in the Test Application configuration can define the following key/value pairs: .. _test_config_args: