About the sync up on Monday

Zygmunt Krynicki zygmunt.krynicki at linaro.org
Fri Mar 9 18:39:07 UTC 2012


W dniu 09.03.2012 19:21, Zach Pfeffer pisze:
> On 9 March 2012 11:47, Paul Larson<paul.larson at linaro.org>  wrote:
>> I wouldn't think for every single command, no.  As mentioned earlier, I
>> suspect it could be broken up into tests comprised of one or more commands.
>> The tests are what would go into lava-android-test.
>>
>> But I'd like to get a better understanding of your perspective.  Would it be
>> possible to provide an example of one of these files so I can get a better
>> idea of what you're looking for here?
>
> Looking at:
>
> http://bazaar.launchpad.net/~linaro-validation/lava-android-test/trunk/view/head:/lava_android_test/test_definitions/monkey.py
>
> To run "monkey" I must define this:
>
> # Copyright (c) 2011 Linaro
>
> # Author: Linaro Validation Team<linaro-dev at lists.linaro.org>
> #
> # This file is part of LAVA Android Test.
> #
> #
> # This program is free software: you can redistribute it and/or modify
> # it under the terms of the GNU General Public License as published by
> # the Free Software Foundation, either version 3 of the License, or
> # (at your option) any later version.
> #
> # This program is distributed in the hope that it will be useful,
> # but WITHOUT ANY WARRANTY; without even the implied warranty of
> # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> # GNU General Public License for more details.
> #
> # You should have received a copy of the GNU General Public License
> # along with this program.  If not, see<http://www.gnu.org/licenses/>.
> import os
> import lava_android_test.testdef
> from lava_android_test.config import get_config
>
> test_name = 'monkey'
> config = get_config()
> curdir = os.path.realpath(os.path.dirname(__file__))
> monkey_sh_name = 'monkey.sh'
> monkey_sh_path = os.path.join(curdir, 'monkey', monkey_sh_name)
> monkey_sh_android_path = os.path.join(config.installdir_android,
>                                        test_name, monkey_sh_name)
>
> INSTALL_STEPS_ADB_PRE = ['push %s %s ' % (monkey_sh_path,
>                                            monkey_sh_android_path),
>                            'shell chmod 777 %s' % monkey_sh_android_path]
>
> ADB_SHELL_STEPS = [monkey_sh_android_path]
> #PATTERN = "^(?P<test_case_id>\w+):\W+(?P<measurement>\d+\.\d+)"
> PATTERN = "## Network stats: elapsed time=(?P<measurement>\d+)ms"
> FAILURE_PATTERNS = []
> #FAILURE_PATTERNS = ['\*\* Monkey aborted due to error.',
> #                    '\*\* System appears to have crashed']
>
> inst = lava_android_test.testdef.AndroidTestInstaller(
>                                      steps_adb_pre=INSTALL_STEPS_ADB_PRE)
> run = lava_android_test.testdef.AndroidTestRunner(
>                                      adbshell_steps=ADB_SHELL_STEPS)
> parser = lava_android_test.testdef.AndroidTestParser(PATTERN,
>                 appendall={'units': 'ms'}, failure_patterns=FAILURE_PATTERNS)
> testobj = lava_android_test.testdef.AndroidTest(testname=test_name,
>                          installer=inst, runner=run, parser=parser)
> Which calls:
>
> http://bazaar.launchpad.net/~linaro-validation/lava-android-test/trunk/view/head:/lava_android_test/test_definitions/monkey/monkey.sh
>
> #!/system/bin/sh
> #monkey_cmd="monkey -s 1 --pct-touch 10 --pct-motion 20 --pct-nav 20
> --pct-majornav 30 --pct-appswitch 20 --throttle 500 2147483647"
> monkey_cmd="monkey -s 1 --pct-touch 10 --pct-motion 20 --pct-nav 20
> --pct-majornav 30 --pct-appswitch 20 --throttle 500 500"
> echo execute command=${monkey_cmd}
> ${monkey_cmd}
> echo MONKEY_RET_CODE=$?
>
> ...so to run a specific instance of monkey:
>
> monkey -s 1 --pct-touch 10 --pct-motion 20 --pct-nav 20 --pct-majornav
> 30 --pct-appswitch 20 --throttle 500 2147483647
>
> I must write 20 lines of code.
>
> Lets say I want to run monkey N different ways, I can create:
>
> http://bazaar.launchpad.net/~linaro-validation/lava-android-test/trunk/view/head:/lava_android_test/test_definitions/monkey/monkey1.sh
> http://bazaar.launchpad.net/~linaro-validation/lava-android-test/trunk/view/head:/lava_android_test/test_definitions/monkey/monkey2.sh
> http://bazaar.launchpad.net/~linaro-validation/lava-android-test/trunk/view/head:/lava_android_test/test_definitions/monkey/monkey3.sh
> ...
> http://bazaar.launchpad.net/~linaro-validation/lava-android-test/trunk/view/head:/lava_android_test/test_definitions/monkey/monkeyN.sh
>
> ...or I could write something that allows me to pass the command line
> I want to monkey through the JSON binding.
>
> Lets say I want to run monkey in N ways for target A and M ways for
> target B and etc...
>
> I won't want to encode all of these as
>
> http://bazaar.launchpad.net/~linaro-validation/lava-android-test/trunk/view/head:/lava_android_test/test_definitions/monkey/monkey1_target1.sh
> http://bazaar.launchpad.net/~linaro-validation/lava-android-test/trunk/view/head:/lava_android_test/test_definitions/monkey/monkey2_target1.sh
> http://bazaar.launchpad.net/~linaro-validation/lava-android-test/trunk/view/head:/lava_android_test/test_definitions/monkey/monkey3_target1.sh
> ...
> http://bazaar.launchpad.net/~linaro-validation/lava-android-test/trunk/view/head:/lava_android_test/test_definitions/monkey/monkeyN_targetX.sh
>
> I really just want to pass a file in that says:
>
> monkey1
> monkey2
> monkey3
> monkey4

Then all you need is one monkey with all the parameters exposed.
Everything else will be specified by the dispatcher job. You can specify 
targets and any parameters.

You are attempting to solve a non-issue. We have everything needed to 
support your use case now.

Thanks
ZK

> ...and have LAVA execute that.
>
> So that's generally doable. I'll write my 20 lines to call a test
> entiry point that passes a known good file.
>
> But each one of the commands I passed in will crash the unit. I want
> to run them back to back because I want to see for each command how
> long the unit ran.
>
> Now I could just have the command file that I run, that I wrote the 20
> lines for, simply end on a test that I think will crash, but I'll
> never know for sure. To save me from guessing, the "tester" simply
> picks back up where it left off.
>
>  From a LAVA perspective this would probably be down to:
>
>      program_the_unit
>      current_command = 0
> boot_or_reboot:
>      current_command = next_command
>      running_test = 1
>      while (running_test and current_command) {
>          run_command(current_command)
> 	current_command = next_command
>      }
>      running_test = 0
>
> So we can write the linaro-android-test that runs a command file
> that's resident in the build, but I'm still stuck because if one of
> the commands hangs the test will be marked "over" and the unit will be
> reset, reprogramed and the next linaro-android-test run. So I'll need
> to write something the collects up all the runs and does the analysis
> - or LAVA could execute a test script, collect all the logs, and send
> them back. It could even accept a parser that would be able to parse
> all the logs from all the runs and take lots of nice statistics,
>
> In the end, we will need to self host all the tests we run as per
> George's request. A set of command files and parsers included with
> each build allow me to satisfy that requirement.
>
>> Thanks,
>> Paul Larson
>>
>> On Mar 9, 2012 11:40 AM, "Zach Pfeffer"<zach.pfeffer at linaro.org>  wrote:
>>>
>>> On 9 March 2012 07:33, Paul Larson<paul.larson at linaro.org>  wrote:
>>>> On Fri, Mar 9, 2012 at 7:20 AM, Alexander Sack<asac at linaro.org>  wrote:
>>>>>
>>>>> On Fri, Mar 9, 2012 at 8:05 AM, Paul Larson<paul.larson at linaro.org>
>>>>> wrote:
>>>>>>
>>>>>> On Fri, Mar 9, 2012 at 12:16 AM, Zach Pfeffer
>>>>>> <zach.pfeffer at linaro.org>
>>>>>> wrote:
>>>>>>>
>>>>>>> On 8 March 2012 23:44, Paul Larson<paul.larson at linaro.org>  wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 8, 2012 at 4:07 PM, Zach Pfeffer
>>>>>>>> <zach.pfeffer at linaro.org>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Right. If a command in the command log causes the unit-under-test
>>>>>>>>> to
>>>>>>>>> do any of those things, then the unit should be rebooted (in the
>>>>>>>>> case
>>>>>>>>> of a hang) or the reboot should be sensed (in case the command
>>>>>>>>> caused
>>>>>>>>> a reboot) and when the unit boots back up, LAVA would continue the
>>>>>>>>> test on the command after the one that caused it to hang, reboot,
>>>>>>>>> freeze, etc. LAVA should save the logs for the entire command file
>>>>>>>>> run, including all hangs and reboots.
>>>>>>>>
>>>>>>>> As mentioned when we talked about this on the phone, I don't think
>>>>>>>> this is
>>>>>>>> the best way to approach the problem. Currently, if a test hangs,
>>>>>>>> times out,
>>>>>>>> reboots improperly (and thus times out because it won't get to the
>>>>>>>> expected
>>>>>>>> place), lava will reboot the system back into the test image.
>>>>>>>> However, at
>>>>>>>> this point, it will mark the previous testsuite it tried to run as
>>>>>>>> a
>>>>>>>> failed
>>>>>>>> run.  Then, if there are other tests queued up to run, it will
>>>>>>>> continue
>>>>>>>> running with the next one - NOT try to re-enter the existing test
>>>>>>>> and
>>>>>>>> continue where it left off.  This is not a capability currently
>>>>>>>> supported in
>>>>>>>> lava-android-test.
>>>>>>>>
>>>>>>>> The good news is, I think there's a much more straightforward way
>>>>>>>> to
>>>>>>>> do
>>>>>>>> this.
>>>>>>>> I haven't really seen an example of the kinds of things you want to
>>>>>>>> run, but
>>>>>>>> it seems that it's just a list of several tests, possibly with
>>>>>>>> multiple
>>>>>>>> steps in each test.  And if it reboots, or has a problem, you want
>>>>>>>> it
>>>>>>>> to
>>>>>>>> fail that one and continue to the next.  This is very simple to do,
>>>>>>>> if
>>>>>>>> you
>>>>>>>> simply define each of those tests as lava-android-tests.  Then you
>>>>>>>> can
>>>>>>>> run
>>>>>>>> them all in your test job, and I think it will do exactly what you
>>>>>>>> are
>>>>>>>> looking for here.  Also, they can then be easily reused in other
>>>>>>>> test
>>>>>>>> jobs.
>>>>>>>
>>>>>>> Hmm...
>>>>>>>
>>>>>>> Here's what I'd like to do.
>>>>>>>
>>>>>>> I'd like to pass a simple command file to LAVA. I'd like LAVA to
>>>>>>> execute each test and if that test hangs or causes a reboot I'd like
>>>>>>> LAVA to pick back up at the next test case. I'd like LAVA to collect
>>>>>>> all the logs and send them either to LAVA or back to us for post
>>>>>>> processing.
>>>>>>>
>>>>>>> I'd like to do this, because I'd like to be able to include these
>>>>>>> command files in our builds so that people can run them themselves
>>>>>>> and
>>>>>>> include the post processing commands for people to see what passed
>>>>>>> and
>>>>>>> failed.
>>>>>>>
>>>>>>> The text file also gives me a very easy way to add and remove tests
>>>>>>> on
>>>>>>> a per-build basis since it goes along with the build.
>>>>>>
>>>>>> I get that, but I don't see why you can't have it both ways.  If you
>>>>>> want
>>>>>> a simple set of instructions for someone to manually type at a command
>>>>>> line
>>>>>> to run through the tests, there's nothing preventing you from
>>>>>> including that
>>>>>> in a readme, script, or however you want to do that.  But those tests
>>>>>> described in there would be really nice to convert into reusable tests
>>>>>> in
>>>>>> lava.  Once you have those building blocks, you can arrange them in
>>>>>> future
>>>>>> builds however you like.
>>>>>>
>>>>>
>>>>> I think what Zach is saying here is that he wants to use a single file
>>>>> to
>>>>> maintain the list of tests to run for a build. That file is then
>>>>> sourced
>>>>> both by a tool used by developers for local testing as well as by LAVA.
>>>>>
>>>>> He most likely also would want the local tool used by devs to be a
>>>>> standard solution provided by the lava test framework...
>>>>>
>>>> I know, but what I'm seeing here doesn't look like something that's
>>>> possible
>>>> right now, and I think might take some pretty serious effort and
>>>> re-engineering of how lava works around this particular use case to make
>>>> it
>>>> work.  What I'm trying to show is that there's more than one way to
>>>> think
>>>> about this, and that with a slight change it goes from something that we
>>>> need to discuss at the connect and get some effort behind making happen,
>>>> to
>>>> something that he can have today.  And at the same time, doing it in
>>>> this
>>>> alternative way builds up a reusable set of android tests that can be
>>>> applied to future builds.
>>>
>>> Paul,
>>>
>>> Your proposal is to create a linaro-android-test for each command I
>>> would put in a command file right?
>>>
>>>> Thanks,
>>>> Paul Larson
>>>>
>>>
>>>
>>>
>>> --
>>> Zach Pfeffer
>>> Android Platform Team Lead, Linaro Platform Teams
>>> Linaro.org | Open source software for ARM SoCs
>>> Follow Linaro: http://www.facebook.com/pages/Linaro
>>> http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog
>
>
>


-- 
Zygmunt Krynicki
Linaro Validation Team



More information about the linaro-android mailing list