On Tue, Jun 5, 2012 at 5:57 PM, Zach Pfeffer <
zach.pfeffer@linaro.org> wrote:
> On 5 June 2012 23:34, Alexander Sack <
asac@linaro.org> wrote:
>> On Tue, Jun 5, 2012 at 5:26 PM, Zach Pfeffer <
zach.pfeffer@linaro.org> wrote:
>>> On 5 June 2012 18:23, Alexander Sack <
asac@linaro.org> wrote:
>>>> I feel stopping and rebooting and continuing with the next test is
>>>> what we want to aim for.
>>>>
>>>> On this front I wonder if we should directly go for rebooting for
>>>> _all_ tests to ensure that every test gets executed the same runtime
>>>> environment.
>>>>
>>>> Big benefit is obviously that tests can then stop services, change
>>>> runtime state etc. as much as they like without bothering about
>>>> bringing the system back into a clean state.
>>>>
>>>> Would that be hard? Why wouldn't we do this?
>>>
>>> Seems like a good idea in theory, but in practice it may cause testing
>>> to take a long time. Plus, what constitutes a test boundary? I think
>>
>> I don't think this would really extend the testing time much. Majority
>> of time is usually spend in flashing the image. The booting and
>> running should be not of a big time sink; the little bit we loose we
>> get back from keeping the suite running "isolated".
>>
>>> if we do the fail then restart then we get the best of both worlds,
>>> we're able to run multiple tests and restart if we get the system into
>>> a really weird state.
>>
>> I would think "one suite" is a good test boundary (basically the names
>> you currently put in TEST_PLAN).
>
> Sure. I actually okay with this.
>
> If we do this though, we may want to change things so that each test
> is a tuple with a test name and a timeout for that test.
>