On 10 February 2011 12:19, Mirsad Vojnikovic mirsad.vojnikovic@linaro.org wrote: <snip
That I wrote:
I'd like to add as user stories: Dave wants to rerun a test on a particular machine to see if a failure is machine specific.
An initial idea we had was to run jobs based on machine type, i.e. BeagleBoard, not on a particular machine, i.e. BeagleBoard_ID001. The dispatcher would choose on which particular machine to run, depending on availability. I understand your point when running on a particular machine is desirable, but maybe this feature should be enabled for admins trying to track a deviating hardware? Or maybe this is a user story for dashboard, to have a feature comparing and presenting results from all machines of the same type, or even in broader aspect for chosen/all machine types we support?
I'm talking here of the case where the user has run a set of tests and one is showing up as bad and they are trying to work out why; lets say they run the test again and it works on a different machine; they might reasonably want to see if the original machine fails. Then the second subcase is that we've identified that a particular machine always fails a particular test but no one can explain why; you've been given the job of debugging the test and figuring out why it always fails on that machine. This might not be a hardware/admin issue - it might be something really subtle.
Dave wants to run the same test on a set of machines to compare the results.
This is almost same as first. Maybe the better solution, as I wrote above, is to go to dashboard and compare all the existing results there instead? This assumes of course that there are results already reported for wanted hardware, which I think would be a case if looking at weekly execution intervals, but probably not daily. What do you think, is this reasonable enough or am I missing something important?
OK, there were a few cases I was thinking here: 1) A batch of new machines arrives in the data centre; they are apparently identical - you want to run a benchmark on them all and make sure the variance between them is within the expected range. 2) Some upgrade has happened to a set of machines (e.g. new kernel/new linaro release) rolled out to them all - do they still all behave as expected? 3) You've got a test, it's results seem to vary wildly from run to run - is it consistent across machines in the farm?
Note these set of requirements come from using a similar testing farm.
Dave