[Linaro-validation] Syntax for custom LAVA tests

Zygmunt Krynicki zygmunt.krynicki at linaro.org
Tue Mar 27 13:37:21 UTC 2012

W dniu 27.03.2012 15:04, James Tunnicliffe pisze:
> On 27 March 2012 12:48, Zygmunt Krynicki <zygmunt.krynicki at linaro.org> wrote:
>> W dniu 27.03.2012 13:32, Paul Sokolovsky pisze:
>>> Hello,
>>> Yong Qin is working on the blueprint
>>> https://blueprints.launchpad.net/lava-android-test/+spec/modify-android-build
>>> to add arbitrary custom client-side scripts to Android Build. He
>>> submitted first implementation of it as
>>> https://code.launchpad.net/~liuyq0307/linaro-android-build-tools/run-custom/+merge/98825
>>> and documented as
>>> https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration .
>>> Unfortunately, I'm not thrilled with that implementation, more
>>> specifically, its "user interface" (i.e. any parts which user directly
>>> faces) by the following reasons:
>>> 1. The idea behind Android Build's build config was that they're short
>>> and easy for human to parse, essentially one glance-over would enough
>>> to get a good idea what is built here, even for outsider. Consequently,
>>> the configs should not be overloaded with details not related to
>>> building. If there's a need for integration with other systems, we have
>>> good pattern of externalizing such details and then just referring to
>>> them with a single variable in a build config.
>>> 2. The whole approach in
>>> https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration
>>> seems like trying to encode hierarchical structure in the shell syntax,
>>> which is not much supporting of that. The end result looks pretty much
>>> like representation of graph structure in raw assembler - it's
>>> spaghetti mix of data pieces and labels, requiring long time to wrap
>>> hand around to understand it, and cumbersome and error-prone to write.
>>> So, I would like to propose alternative syntax solving the issues
>>> above. I probably should start with saying that if the talk is about
>>> LAVA, then using native LAVA JSON request immediately comes to mind.
>>> Well, I guess human-writability wasn't a design goal for that, so I
>>> skip it. It still makes sense to stick to general-purpose hierarchical
>>> structure syntax though. Except that JSON has 2 problems: a) it doesn't
>>> support comments natively, so we'll need to pre-process it; b) error
>>> reporting/localization may be still no ideal.
>> Hi, just jumping into the conversation briefly to look at small
>> technical aspect. I have not really been tracking this command effort
>> and I don't understand what it's about.
>> On JSON: I think that comment 2 is inaccurate. We have very precise
>> syntax error reporting (down to line/column and text range on some
>> errors) and even better format reporting (the javascript expression that
>> pinpoints the part of json document that does not match the schema, same
>> for the schema actually).
>> Anyway, you have my full support for native json formats. I think that
>> if comments are an issue I can provide a parser that simply ignores
>> comments. We could then keep the human readable documents and strictly
>> machine readable, schema-backed data.
> There is always YAML - very much human readable, supports hierarchical
> structures, references and comments.

CC-ing the list again.

My long established stance is that YAML is too baroque to be used as a
data interchange format intended for humans to _write_.

Compare YAML spec: http://www.yaml.org/spec/1.2/spec.html
With JSON spec: http://www.json.org/

Make sure to scroll down on the YAML spec, that's _is_ what people need
to remember to understand what they write. The JSON spec is entirely
defined in that railroad graph that fills my screen.

IMHO YAML is filled so many feature that it is virtually impossible to
remember the full syntax. Last time I've checked there was still no
single python parser that supported everything the spec has to offer.
While JSON is strict to the point that it's sometimes annoying it does
not let you make a mistake by accident, there's like 5 syntax elements
to remember. YAML is full of, comparatively, exotic features that while
otherwise harmless can easily confuse users (seemingly random syntax has
meaning) and tools (oh, my parser does not support that feature).

The second argument against YAML is that JSON remains usable on the web
and you don't need two interfaces. Keeping everything in one format is a
good idea IMHO. This may be of no relevance here but if you look at the
entire LAVA ecosystem so far there's no YAML there, just JSON.

Lastly, we have also built nice validation for JSON so that not only
syntax but also the full structure can be checked automatically and
reliably and I'm not aware of any for YAML at this time. The fact that
YAML has support for loops and pointers makes the tools we have useless.

For the sake of human-readability _and_ writeability, a small subset of
YAML might be better than JSON but the full spec IMHO suffers just a
much as JSON, yet for opposite reasons.

My vote is -1.


Zygmunt Krynicki
Linaro Validation Team

More information about the linaro-validation mailing list