Hello Zach,
On Wed, 28 Mar 2012 12:05:46 -0500 Zach Pfeffer zach.pfeffer@linaro.org wrote:
On 27 March 2012 06:32, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
Yong Qin is working on the blueprint https://blueprints.launchpad.net/lava-android-test/+spec/modify-android-buil... to add arbitrary custom client-side scripts to Android Build. He submitted first implementation of it as https://code.launchpad.net/~liuyq0307/linaro-android-build-tools/run-custom/... and documented as https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration .
Unfortunately, I'm not thrilled with that implementation, more specifically, its "user interface" (i.e. any parts which user directly faces) by the following reasons:
- The idea behind Android Build's build config was that they're
short and easy for human to parse, essentially one glance-over would enough to get a good idea what is built here, even for outsider. Consequently, the configs should not be overloaded with details not related to building. If there's a need for integration with other systems, we have good pattern of externalizing such details and then just referring to them with a single variable in a build config.
- The whole approach in
https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration seems like trying to encode hierarchical structure in the shell syntax, which is not much supporting of that. The end result looks pretty much like representation of graph structure in raw assembler
- it's spaghetti mix of data pieces and labels, requiring long time
to wrap hand around to understand it, and cumbersome and error-prone to write.
So, I would like to propose alternative syntax solving the issues above. I probably should start with saying that if the talk is about LAVA, then using native LAVA JSON request immediately comes to mind. Well, I guess human-writability wasn't a design goal for that, so I skip it. It still makes sense to stick to general-purpose hierarchical structure syntax though. Except that JSON has 2 problems: a) it doesn't support comments natively, so we'll need to pre-process it; b) error reporting/localization may be still no ideal.
Anyway, here's my try, it is presented as a comment to https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration and then full example at https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration/pfalco...
Let's discuss if that covers our needs and constraints.
Paul, thanks for the alternative proposal.
The request was:
Provide a easy way to pass a test into LAVA from an android-build configuration.
Yong Qin's approach seems reasonable because its of the form:
MY_DEFINE="runmycommand --paramone 1 --paramtwo 2"
Well, not quite, example at https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration shows 20 cross-referencing lines for defining tests (with build configuration per se taking 5).
I don't like Paul's proposal because if you need something that complicated you should just use the LAVA python interface. I just want something simple.
Something like:
LAVA_TEST_1="runmycommand --paramone 1 --paramtwo 2" LAVA_TEST_PARSE_1="some regex"
LAVA_TEST_2="runmyothercommand --paramone 1" LAVA_TEST_PARSE_2="some other regex"
..and that's it.
Well, if it's about "parsing", it's never that simple. Yong Qin's example shows what typical parsing regexp looks like.
LAVA would then run LAVA_TEST_1 and LAVA_TEST_2 etc... and use LAVA_TEST_PARSE_1 and LAVA_TEST_PARSE_2 to parse the results nad send them back to the android-build.
All the tests we run on Android will be baked into the build, so we don't have to worry about installing them.
I don't even want to have to do this:
LAVA_TESTS="LAVA_TEST_1 LAVA_TEST_2"
...because that's adding too much structure.
But if you have too many things without structure, mess ensues. In particular, it's unclear when these tests run in respect to LAVA_TEST_PLAN.
This is really just a way to get a test in without needing to add a bunch of python code to LAVA. The flat list of commands is easy to see, and there's no tree structure to contend with. If you need more structure you'd write some python or write the structure into a script.
Well, all in all, what you described is quite different from what is presented at https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration (and implemented in 100 lines (50% of existing code size) at https://code.launchpad.net/~liuyq0307/linaro-android-build-tools/run-custom/...) , so probably lack of proper spec to start with was the issue here.
If we reduce the requirements to just supporting textual (no external URL references) content in
LAVA_TEST_${N} LAVA_TEST_PARSE_${N}
and well defined order to run them (for example, after LAVA_TEST_PLAN), then it certainly will be good enough to put into build configs(though parsing regexps would still look hairy there imho).