On 8/1/24 10:27 PM, Shuah Khan wrote:
On 8/1/24 10:27, Shuah Khan wrote:
On 8/1/24 02:42, Muhammad Usama Anjum wrote:
On 7/31/24 9:57 PM, Shuah Khan wrote:
On 7/31/24 07:39, Muhammad Usama Anjum wrote:
Don't print that 88 sub-tests are going to be executed, but then skip. This is against TAP compliance. Instead check pre-requisites first before printing total number of tests.
Does TAP clearly mention this?
Yes from https://testanything.org/tap-version-13-specification.html
Skipping everything This listing shows that the entire listing is a skip. No tests were run.
TAP version 13 1..0 # skip because English-to-French translator isn't installed
One more thing on TAP compliance - we don't strive to be TAP compliant as it doesn't meet our special needs.
It is important to keep the how many tests could be run to improve testing coverage.
Refer to: https://www.kernel.org/doc/html/latest/dev-tools/ktap.html
"The Linux Kernel largely uses TAP output for test results. However, Kernel testing frameworks have special needs for test results which don’t align with the original TAP specification. Thus, a “Kernel TAP” (KTAP) format is specified to extend and alter TAP to support these use-cases. This specification describes the generally accepted format of KTAP as it is currently used in the kernel."
I appreciate the effort you are putting into improving the reports. I request that you refer to the above document and also keep in mind what would help us improve our testing over focusing just on reports and compliance.
That makes sense. I'll work on improving the testing going forward.
thanks, -- Shuah