On Mar 9, 2012 11:40 AM, "Zach Pfeffer" <zach.pfeffer@linaro.org> wrote:

On 9 March 2012 07:33, Paul Larson <paul.larson@linaro.org> wrote:
> On Fri, Mar 9, 2012 at 7:20 AM, Alexander Sack <asac@linaro.org> wrote:
>>
>> On Fri, Mar 9, 2012 at 8:05 AM, Paul Larson <paul.larson@linaro.org>
>> wrote:
>>>
>>> On Fri, Mar 9, 2012 at 12:16 AM, Zach Pfeffer <zach.pfeffer@linaro.org>
>>> wrote:
>>>>
>>>> On 8 March 2012 23:44, Paul Larson <paul.larson@linaro.org> wrote:
>>>> >
>>>> >
>>>> > On Thu, Mar 8, 2012 at 4:07 PM, Zach Pfeffer <zach.pfeffer@linaro.org>
>>>> > wrote:
>>>> >>
>>>> >> Right. If a command in the command log causes the unit-under-test to
>>>> >> do any of those things, then the unit should be rebooted (in the case
>>>> >> of a hang) or the reboot should be sensed (in case the command caused
>>>> >> a reboot) and when the unit boots back up, LAVA would continue the
>>>> >> test on the command after the one that caused it to hang, reboot,
>>>> >> freeze, etc. LAVA should save the logs for the entire command file
>>>> >> run, including all hangs and reboots.
>>>> >
>>>> > As mentioned when we talked about this on the phone, I don't think
>>>> > this is
>>>> > the best way to approach the problem. Currently, if a test hangs,
>>>> > times out,
>>>> > reboots improperly (and thus times out because it won't get to the
>>>> > expected
>>>> > place), lava will reboot the system back into the test image.
>>>> > However, at
>>>> > this point, it will mark the previous testsuite it tried to run as a
>>>> > failed
>>>> > run.  Then, if there are other tests queued up to run, it will
>>>> > continue
>>>> > running with the next one - NOT try to re-enter the existing test and
>>>> > continue where it left off.  This is not a capability currently
>>>> > supported in
>>>> > lava-android-test.
>>>> >
>>>> > The good news is, I think there's a much more straightforward way to
>>>> > do
>>>> > this.
>>>> > I haven't really seen an example of the kinds of things you want to
>>>> > run, but
>>>> > it seems that it's just a list of several tests, possibly with
>>>> > multiple
>>>> > steps in each test.  And if it reboots, or has a problem, you want it
>>>> > to
>>>> > fail that one and continue to the next.  This is very simple to do, if
>>>> > you
>>>> > simply define each of those tests as lava-android-tests.  Then you can
>>>> > run
>>>> > them all in your test job, and I think it will do exactly what you are
>>>> > looking for here.  Also, they can then be easily reused in other test
>>>> > jobs.
>>>>
>>>> Hmm...
>>>>
>>>> Here's what I'd like to do.
>>>>
>>>> I'd like to pass a simple command file to LAVA. I'd like LAVA to
>>>> execute each test and if that test hangs or causes a reboot I'd like
>>>> LAVA to pick back up at the next test case. I'd like LAVA to collect
>>>> all the logs and send them either to LAVA or back to us for post
>>>> processing.
>>>>
>>>> I'd like to do this, because I'd like to be able to include these
>>>> command files in our builds so that people can run them themselves and
>>>> include the post processing commands for people to see what passed and
>>>> failed.
>>>>
>>>> The text file also gives me a very easy way to add and remove tests on
>>>> a per-build basis since it goes along with the build.
>>>
>>> I get that, but I don't see why you can't have it both ways.  If you want
>>> a simple set of instructions for someone to manually type at a command line
>>> to run through the tests, there's nothing preventing you from including that
>>> in a readme, script, or however you want to do that.  But those tests
>>> described in there would be really nice to convert into reusable tests in
>>> lava.  Once you have those building blocks, you can arrange them in future
>>> builds however you like.
>>>
>>
>> I think what Zach is saying here is that he wants to use a single file to
>> maintain the list of tests to run for a build. That file is then sourced
>> both by a tool used by developers for local testing as well as by LAVA.
>>
>> He most likely also would want the local tool used by devs to be a
>> standard solution provided by the lava test framework...
>>
> I know, but what I'm seeing here doesn't look like something that's possible
> right now, and I think might take some pretty serious effort and
> re-engineering of how lava works around this particular use case to make it
> work.  What I'm trying to show is that there's more than one way to think
> about this, and that with a slight change it goes from something that we
> need to discuss at the connect and get some effort behind making happen, to
> something that he can have today.  And at the same time, doing it in this
> alternative way builds up a reusable set of android tests that can be
> applied to future builds.

Paul,

Your proposal is to create a linaro-android-test for each command I
would put in a command file right?

> Thanks,
> Paul Larson
>



--
Zach Pfeffer
Android Platform Team Lead, Linaro Platform Teams
Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog