2012.05 planning: Move existing test defintions away from LAVA test to a dedicated project
Hi, please have a look at [1] and discuss this on the mailing list.
[1] https://blueprints.launchpad.net/lava-test/+spec/lava-test-dedicated-test-re...
On 04/02/2012 07:20 AM, Zygmunt Krynicki wrote:
Hi, please have a look at [1] and discuss this on the mailing list.
[1] https://blueprints.launchpad.net/lava-test/+spec/lava-test-dedicated-test-re...
I would prefer this stay as one project, but the "test_definitions" directory is thought of as a separate component that another team, Paul's, manage (bug triage, code review, and merge)
I think keeping this simplicity is important.
W dniu 02.04.2012 16:30, Andy Doan pisze:
On 04/02/2012 07:20 AM, Zygmunt Krynicki wrote:
Hi, please have a look at [1] and discuss this on the mailing list.
[1] https://blueprints.launchpad.net/lava-test/+spec/lava-test-dedicated-test-re...
I would prefer this stay as one project, but the "test_definitions" directory is thought of as a separate component that another team, Paul's, manage (bug triage, code review, and merge)
AFAIK there is no way to implement that on launchpad without spamming everyone in the LAVA part of the team. Plus it does not address any of the other issues I've raised. It certainly is not a single project for the reasons I've outlined there.
I think keeping this simplicity is important.
I don't think this will raise the complexity, if anything it will ensure that the same level of user experience is provided to in-tree and out-of-tree tests.
Thanks ZK
So what you are suggesting is basically that *all* tests become out-of-tree tests?
Also, some of your rationale is around lava being cross-platform. Lava-test - which natively assumes ubuntu/debian is certainly not cross-platform. Also, how would this affect lava-android-test?
I actually think we need to rethink lava-android-test, lava-test, etc. This is, perhaps, a good blueprint topic for the next connect. There have been some who have even expressed interest in running host-side tests for things like bootloader testing, prompting previous discussions of yet another lava-test fork. This is kinda nuts to keep forking this thing. I'd really like to go back to a base framework and reconsider things that we thought were a priority in the original design.
I'm thinking that all of this can be driven from the host side. A transport could be established and specified and handed off to lava-test to drive the install steps, run steps, etc. It could be serial, ssh, adb, lava-client, whatever. Then parsing/bundling would happen host-side. Test definitions could possibly even be simplified a bit I think.
Thoughts?
On Mon, Apr 2, 2012 at 9:57 AM, Zygmunt Krynicki < zygmunt.krynicki@linaro.org> wrote:
W dniu 02.04.2012 16:30, Andy Doan pisze:
On 04/02/2012 07:20 AM, Zygmunt Krynicki wrote:
Hi, please have a look at [1] and discuss this on the mailing list.
[1]
https://blueprints.launchpad.net/lava-test/+spec/lava-test-dedicated-test-re...
I would prefer this stay as one project, but the "test_definitions" directory is thought of as a separate component that another team, Paul's, manage (bug triage, code review, and merge)
AFAIK there is no way to implement that on launchpad without spamming everyone in the LAVA part of the team. Plus it does not address any of the other issues I've raised. It certainly is not a single project for the reasons I've outlined there.
I think keeping this simplicity is important.
I don't think this will raise the complexity, if anything it will ensure that the same level of user experience is provided to in-tree and out-of-tree tests.
Thanks ZK
-- Zygmunt Krynicki Linaro Validation Team
linaro-validation mailing list linaro-validation@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-validation
W dniu 02.04.2012 21:45, Paul Larson pisze:
So what you are suggesting is basically that *all* tests become out-of-tree tests?
Yes. Then the UI becomes streamlined, installation issues (if any) quickly fixed and everyone plays on equal ground. Then all current tests become linaro specific (if we so desire) and are installed by default in our jobs.
Also, some of your rationale is around lava being cross-platform. Lava-test - which natively assumes ubuntu/debian is certainly not cross-platform. Also, how would this affect lava-android-test?
LAVA test is perfectly cross platform. The part that relies on debian has been optional for a while. With my recent proposals it would also be possible to plug non-ubuntu package backends there. I can easily see fedora/rpm package backend and pure-python package backends being immediately useful. This would allow lava-test to be used on all major operating systems, to run unit tests of python software.
Android is not part of this discussion but lava-android-test _can_ and _should_, similarly, work on all operating systems (including windows and osx), that have working adb client.
I actually think we need to rethink lava-android-test, lava-test, etc. This is, perhaps, a good blueprint topic for the next connect. There
We already have some blueprints on that topic. In the long term I'd like to merge both tools while unlocking a possibility of host-driven primitive-device tests that would allow bare-metal/silicon simluator/bootloader tests within the same common framework and command line interface.
have been some who have even expressed interest in running host-side tests for things like bootloader testing, prompting previous discussions of yet another lava-test fork. This is kinda nuts to keep forking this thing. I'd really like to go back to a base framework and reconsider things that we thought were a priority in the original design.
+1 but this should not affect the decision around this blueprint.
I'm thinking that all of this can be driven from the host side. A transport could be established and specified and handed off to lava-test to drive the install steps, run steps, etc. It could be serial, ssh, adb, lava-client, whatever. Then parsing/bundling would happen host-side. Test definitions could possibly even be simplified a bit I think.
You'll be happy to know that my recent work on fast models may bring us very close to that point. I'll explain later as it's already past 10PM but I'll provide a generic connection interface that has many (extensible by third party) implementation and drives the stack for both dispatcher and (possibly) lava-test.
Thanks ZK
On 3 April 2012 03:45, Paul Larson paul.larson@linaro.org wrote:
So what you are suggesting is basically that *all* tests become out-of-tree tests?
To some extent, I +1 for making more test become out-of-tree test too.
Let say integrating an android test into lava-android-test: 1. android team member create the test tools sometimes the test tools just an android shell script, but it also need to merged into lava-android-test 2. android team member understand the lava wrapper infrastructure 3. android team member need to understand python to write the python wrapper script. It's very very simple for member who know python, but will be difficult for member who does not know python. like the indent style, string format style, for/if sentence style 4. android team member need to submit a MP for the branch, and wait review/merge 5. android member need to wait lava-android-test deployed to production and then can try the test on production. some times also need to wait the modification of android build.
And all the steps above will need LAVA member to support.
If we make the test out-of-tree, then we just need to define the rules for the test tools that lava support, and android team member just need create the test tools obey the rules lava required. They can use any language the like, shell, python, C, even assemble language if he like And the monkeyrunner supported now in the lava-android-test is something like this. android team member only need to create the monkeyrunner script obey the rule and commit it into their git. then they can try the test from android build.
Thanks, Yongqin Liu
Also, some of your rationale is around lava being cross-platform. Lava-test - which natively assumes ubuntu/debian is certainly not cross-platform. Also, how would this affect lava-android-test?
I actually think we need to rethink lava-android-test, lava-test, etc. This is, perhaps, a good blueprint topic for the next connect. There have been some who have even expressed interest in running host-side tests for things like bootloader testing, prompting previous discussions of yet another lava-test fork. This is kinda nuts to keep forking this thing. I'd really like to go back to a base framework and reconsider things that we thought were a priority in the original design.
I'm thinking that all of this can be driven from the host side. A transport could be established and specified and handed off to lava-test to drive the install steps, run steps, etc. It could be serial, ssh, adb, lava-client, whatever. Then parsing/bundling would happen host-side. Test definitions could possibly even be simplified a bit I think.
Thoughts?
On Mon, Apr 2, 2012 at 9:57 AM, Zygmunt Krynicki < zygmunt.krynicki@linaro.org> wrote:
W dniu 02.04.2012 16:30, Andy Doan pisze:
On 04/02/2012 07:20 AM, Zygmunt Krynicki wrote:
Hi, please have a look at [1] and discuss this on the mailing list.
[1]
https://blueprints.launchpad.net/lava-test/+spec/lava-test-dedicated-test-re...
I would prefer this stay as one project, but the "test_definitions" directory is thought of as a separate component that another team, Paul's, manage (bug triage, code review, and merge)
AFAIK there is no way to implement that on launchpad without spamming everyone in the LAVA part of the team. Plus it does not address any of the other issues I've raised. It certainly is not a single project for the reasons I've outlined there.
I think keeping this simplicity is important.
I don't think this will raise the complexity, if anything it will ensure that the same level of user experience is provided to in-tree and out-of-tree tests.
Thanks ZK
-- Zygmunt Krynicki Linaro Validation Team
linaro-validation mailing list linaro-validation@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-validation
linaro-validation mailing list linaro-validation@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-validation
On 04/02/2012 09:57 AM, Zygmunt Krynicki wrote:
W dniu 02.04.2012 16:30, Andy Doan pisze:
On 04/02/2012 07:20 AM, Zygmunt Krynicki wrote:
Hi, please have a look at [1] and discuss this on the mailing list.
[1] https://blueprints.launchpad.net/lava-test/+spec/lava-test-dedicated-test-re...
I would prefer this stay as one project, but the "test_definitions" directory is thought of as a separate component that another team, Paul's, manage (bug triage, code review, and merge)
AFAIK there is no way to implement that on launchpad without spamming everyone in the LAVA part of the team. Plus it does not address any of the other issues I've raised. It certainly is not a single project for the reasons I've outlined there.
I think other than the cross-platform nature the main issues you raised are the email/bugs/review/merging on the project. I'd think a mail filter could help with that. Given we're already forked on android in such a different way, I don't know if I'd make cross-platform a high priority for May.
I think keeping this simplicity is important.
I don't think this will raise the complexity, if anything it will ensure that the same level of user experience is provided to in-tree and out-of-tree tests.
As a person who's had to "lava-tize this thing" its one more thing. There were so many steps in working with LAVA that I essentially refused to use LAVA for the first 4 months I was doing benchmarks. To be fair - it wasn't that bad, but that the impression people have.
Also, it gets easier for someone to run into the scenario where their test only works on some version of the framework. Also around inter-dependencies is the case when someone introduces a new test that requires functional change to the framework. Now its easy to handle in one merge proposal.
All that said - Paul's thread raises a completely separate idea that's very interesting and would probably end up accomplishing what you want. I just don't really like the idea of splitting it up based on what I currently see in the blueprint. I'm sure I could be persuaded differently.
-andy
linaro-validation@lists.linaro.org
-
Andy Doan -
Paul Larson -
YongQin Liu -
Zygmunt Krynicki