On 10 February 2011 04:30, David Gilbert <david.gilbert@linaro.org> wrote:
On 10 February 2011 12:19, Mirsad Vojnikovic
<mirsad.vojnikovic@linaro.org> wrote:
<snip
 > That I wrote:

>> I'd like to add as user stories:
>>   Dave wants to rerun a test on a particular machine to see if a
>> failure is machine specific.
>
> An initial idea we had was to run jobs based on machine type, i.e.
> BeagleBoard, not on a particular machine, i.e. BeagleBoard_ID001. The
> dispatcher would choose on which particular machine to run, depending on
> availability. I understand your point when running on a particular machine
> is desirable, but maybe this feature should be enabled for admins trying to
> track a deviating hardware? Or maybe this is a user story for dashboard, to
> have a feature comparing and presenting results from all machines of the
> same type, or even in broader aspect for chosen/all machine types we
> support?

I'm talking here of the case where the user has run a set of tests and
one is showing
up as bad and they are trying to work out why; lets say they run the
test again and it
works on a different machine; they might reasonably want to see if the
original machine fails.
Then the second subcase is that we've identified that a particular machine
always fails a particular test but no one can explain why; you've been
given the job
of debugging the test and figuring out why it always fails on that machine.
This might not be a hardware/admin issue - it might be something really subtle.

I understand what you aim at. The question is then to allow or not allow users to submit jobs to particular machine(s). I have no particular problem with allowing it, we can include it in our solution. We can have both choices: run on particular machine(s) or let the system choose one or more from given machine type(s). Anyone else, comments on this?
 

>>   Dave wants to run the same test on a set of machines to compare the
>> results.
>
> This is almost same as first. Maybe the better solution, as I wrote above,
> is to go to dashboard and compare all the existing results there instead?
> This assumes of course that there are results already reported for wanted
> hardware, which I think would be a case if looking at weekly execution
> intervals, but probably not daily. What do you think, is this reasonable
> enough or am I missing something important?

OK, there were a few cases I was thinking here:
 1)  A batch of new machines arrives in the data centre; they are apparently
identical - you want to run a benchmark on them all and make sure the variance
between them is within the expected range.
 2) Some upgrade has happened to a set of machines (e.g. new kernel/new linaro
release) rolled out to them all - do they still all behave as expected?
 3) You've got a test, it's results seem to vary wildly from run to run - is it
consistent across machines in the farm?

OK, I understand better now. For me this is still at test result level, i.e. dashboard (launch-control) should produce such kind of reports. Cannot see where this fits on scheduler level? When we give the possibility to run jobs on specific boards, it should be easy to retrieve all needed test reports from the dashboard.
 

Note these set of requirements come from using a similar testing farm. 

Dave