[Linaro-validation] Syntax for custom LAVA tests

Paul Sokolovsky paul.sokolovsky at linaro.org
Tue Mar 27 13:06:39 UTC 2012


Hello Zygmunt,

On Tue, 27 Mar 2012 13:48:30 +0200
Zygmunt Krynicki <zygmunt.krynicki at linaro.org> wrote:

> W dniu 27.03.2012 13:32, Paul Sokolovsky pisze:
> > Hello,
> > 
> > Yong Qin is working on the blueprint
> > https://blueprints.launchpad.net/lava-android-test/+spec/modify-android-build
> > to add arbitrary custom client-side scripts to Android Build. He
> > submitted first implementation of it as
> > https://code.launchpad.net/~liuyq0307/linaro-android-build-tools/run-custom/+merge/98825
> > and documented as
> > https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration .
> > 
> > Unfortunately, I'm not thrilled with that implementation, more
> > specifically, its "user interface" (i.e. any parts which user
> > directly faces) by the following reasons:
> > 
> > 1. The idea behind Android Build's build config was that they're
> > short and easy for human to parse, essentially one glance-over
> > would enough to get a good idea what is built here, even for
> > outsider. Consequently, the configs should not be overloaded with
> > details not related to building. If there's a need for integration
> > with other systems, we have good pattern of externalizing such
> > details and then just referring to them with a single variable in a
> > build config.
> > 
> > 2. The whole approach in
> > https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration
> > seems like trying to encode hierarchical structure in the shell
> > syntax, which is not much supporting of that. The end result looks
> > pretty much like representation of graph structure in raw assembler
> > - it's spaghetti mix of data pieces and labels, requiring long time
> > to wrap hand around to understand it, and cumbersome and
> > error-prone to write.
> > 
> > 
> > So, I would like to propose alternative syntax solving the issues
> > above. I probably should start with saying that if the talk is about
> > LAVA, then using native LAVA JSON request immediately comes to mind.
> > Well, I guess human-writability wasn't a design goal for that, so I
> > skip it. It still makes sense to stick to general-purpose
> > hierarchical structure syntax though. Except that JSON has 2
> > problems: a) it doesn't support comments natively, so we'll need to
> > pre-process it; b) error reporting/localization may be still no
> > ideal.
> 
> Hi, just jumping into the conversation briefly to look at small
> technical aspect. I have not really been tracking this command effort
> and I don't understand what it's about.

Yeah, me too, I just spent some time reviewing Yong Qin's examples and
code. And my main concern about it is external interface/syntax which
users face on android-build.linaro.org. I'm ok with any underlying
implementation.

> On JSON: I think that comment 2 is inaccurate. We have very precise
> syntax error reporting (down to line/column and text range on some
> errors) and even better format reporting (the javascript expression
> that pinpoints the part of json document that does not match the
> schema, same for the schema actually).

Yes, I meant a bit different thing, let me elaborate:

1. LAVA has API which accepts JSON. You say that you parse that JSON
very carefully, that's cool! But that native API syntax unlikely was
designed to be easily written by human from scratch, right?
2. So, let us on android-build side (could be reused if proved
successful) to develop another, deliberately human-optimized syntax,
and let it be still based on JSON.
3. Then, android-build side script will a) read such human-friendly test
def (example:
https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration/pfalcon),
b) transform it into native LAVA syntax (essentially, dereference
URLs and add machine-readable fields), and c) submit as such.
4. My concern with error reporting and comment support was for 3a
step. Well, I just checked that python's json module does down-to-char
error reporting, which is cool. And comments can be handled by a simple
regexp substitution, so we're all good.


So, with this plan, there won't be any code changes from LAVA team
(assuming that running arbitrary test commands already supported by
its API), but we'd for sure appreciate sanity and approach review.

> 
> Anyway, you have my full support for native json formats. I think that
> if comments are an issue I can provide a parser that simply ignores
> comments. We could then keep the human readable documents and strictly
> machine readable, schema-backed data.
> 
> Thanks
> ZK
> 
> > Anyway, here's my try, it is presented as a comment to 
> > https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration
> > and then full example at
> > https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration/pfalcon
> > 
> > 
> > Let's discuss if that covers our needs and constraints.
> > 
> > Thanks,
> > Paul
> > 
> > Linaro.org | Open source software for ARM SoCs
> > Follow Linaro: http://www.facebook.com/pages/Linaro
> > http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog
> > 
> > _______________________________________________
> > linaro-validation mailing list
> > linaro-validation at lists.linaro.org
> > http://lists.linaro.org/mailman/listinfo/linaro-validation
> 
> 



-- 
Best Regards,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog



More information about the linaro-validation mailing list