On 10 February 2011 13:14, Mirsad Vojnikovic mirsad.vojnikovic@linaro.org wrote:
On 10 February 2011 04:30, David Gilbert david.gilbert@linaro.org wrote:
OK, there were a few cases I was thinking here: 1) A batch of new machines arrives in the data centre; they are apparently identical - you want to run a benchmark on them all and make sure the variance between them is within the expected range. 2) Some upgrade has happened to a set of machines (e.g. new kernel/new linaro release) rolled out to them all - do they still all behave as expected? 3) You've got a test, it's results seem to vary wildly from run to run - is it consistent across machines in the farm?
OK, I understand better now. For me this is still at test result level, i.e. dashboard (launch-control) should produce such kind of reports. Cannot see where this fits on scheduler level? When we give the possibility to run jobs on specific boards, it should be easy to retrieve all needed test reports from the dashboard.
My only reason for thinking it was a scheduling issue is that you need a way to cause the same test to happen on all the machines in a group; not necessarily at the same time.
Dave
Note these set of requirements come from using a similar testing farm.
Dave