On Thu, Jun 30, 2011 at 11:24 PM, Daniel Lezcano <daniel.lezcano@linaro.org> wrote:
When all tests will be finished I wish to switch the new way test suite
execution in lava, if it is possible.
Given that the old tests are broken at the moment and disabled, any reason we shouldn't switchover now?
 
Will the following format be ok ?

test_01/cpu0 : checking scaling_available_frequencies file ... PASS
...
That would probably translate internally to something like: 
{test_id="test_01_cpu0", message="checking scaling_available_frequencies file", result="PASS"}
Is that ok?  Seems like something we should have no trouble making sense out of later I think.  Also, the exact output is saved as an attachment as well.
 
The result for a test case is PASS or FAIL.

We support "unknown" as a result as well, if that helps you at all.  Sometimes results can be indeterminate, or unimportant (odd as that may sound, sometimes the message is really what you're after, as illustrated by some of the previous pm qa tests)

But under some circumstances, we need to do some extra work where a
failure does not mean the test case failed but the pre-requisite for the
test case is not met.
...and we also support "skip" as a result.  That seems like a correct use for it.
 You don't have to report it literally as "skip", you can call it "oink" for all we care, and just provide a translation table that converts whatever your result strings are to {pass, fail, skip, unknown}.  For example, see the test definition for ltp.
 

> deviation 0 % for 2333000 is ...                                      VERY
>> GOOD
>
> Same comments as above about having an easier to interpret format, but the
> result here: "VERY GOOD" - what does that mean?  What are the other possible
> values?  Is this simply another way of saying "PASS"?  Or should it actually
> be a measurement reported here?

Yep, I agree it is an informational message and should go to a logging
file. I will stick to a simple result 'PASS' or 'FAIL' and I will let
the user to read the documentation of the test in the wiki page to
understand the meaning of these messages (GOOD, VERY GOOD...).
keeping to the template you mentioned earlier, I wonder if we could do something like this:
deviation_0_for_2333000: VERY GOOD ... PASS
(are the numbers there consistent and useful as a test case name?  I'm assuming so here)
That would allow you to capture "VERY GOOD" as details in the message (one of the fields we keep).  Also, your test could be smart enough to know that good, or verygood = pass, while bad, verybad = fail.  Possibly I'm making a lot of assumptions here, but I think you see what I mean. 

Keeping to a consistent results format in your output is a good practice, and makes this *much* easier for capturing the data important to you.  Of course, anything is possible.  We have some tests with rather elaborate results and for those, the test definition just inherits from a base class and defines it's own parser.  If you're not feeling very pythonic, you could provide your own parser as part of the test download written in shell, c, ruby, go, whatever, that just acts as a filter, then have it just read it all in directly.  There are lots of options, but I'm of the opinion that a consistent format makes it easier on humans looking at it as much as machine parsers.  And since you have control over that, easiest to do it now. :)

Thanks,
Paul Larson