Hi, I've been playing around with this "lava" thing a bit :)
From the perspective of the QA team, I'm currently interested in a very
specific use case, but it's one that I think has some common things with what others want to do: We have test results for various {ubuntu, android} images. I'd like to see a current snapshot of the results on those images. I'd also like to see the history for previous results on the same type of image. This way, as we near release time, if we look at trying to figure out which build is likely to be the best one to use for release candidate testing, we can glance at the results and see where most of our automated tests passed.
So just looking at android as an example, because we already store data there about which image we are testing, which build number, which url. It's pretty simple to pull all bundles in a stream. It's also not too bad to look at a test_run in a bundle and get the fact that it has a certain attribute. It's not so easy to say "show me all the bundles that have this attribute". Am I missing something?
I think I brought this up before, but I ran across a project (heck if I can find it now) that had a python library providing an API to make it convenient for finding your test data. This is what the ORM *should* be providing, I know, but it's not always the most convenient for us as we've all struggled with this in the past.
This is nothing new, just wanted to present another use case that highlights something we're trying to do, but doesn't seem to be too easy at the moment, and looking for suggestions about how to proceed.
Thanks, Paul Larson
On Tue, 8 May 2012 16:54:49 -0700, Paul Larson paul.larson@linaro.org wrote:
Hi, I've been playing around with this "lava" thing a bit :)
Heh heh.
From the perspective of the QA team, I'm currently interested in a very specific use case, but it's one that I think has some common things with what others want to do: We have test results for various {ubuntu, android} images. I'd like to see a current snapshot of the results on those images. I'd also like to see the history for previous results on the same type of image.
FWIW, I've been thinking and talking about this as "horizontal" and "vertical" grouping of test results.
A vertical grouping would be something category based like "all android tests of panda builds from the -staging manifest" or "compile tests of the linus-tip tree" or...
A horizontal grouping would be something essentially time based -- "tests run yesterday" would be one example, although it could also be "tests on kernels built from a particular git commit" or "all tests on images built from a particular build of the developer rootfs".
(Unsurprisingly, the columns on the kernel ci view are example of vertical groupings and the rows horizontal groupings).
This way, as we near release time, if we look at trying to figure out which build is likely to be the best one to use for release candidate testing, we can glance at the results and see where most of our automated tests passed.
So just looking at android as an example, because we already store data there about which image we are testing, which build number, which url. It's pretty simple to pull all bundles in a stream. It's also not too bad to look at a test_run in a bundle and get the fact that it has a certain attribute. It's not so easy to say "show me all the bundles that have this attribute". Am I missing something?
No.
I think I brought this up before, but I ran across a project (heck if I can find it now) that had a python library providing an API to make it convenient for finding your test data. This is what the ORM *should* be providing, I know, but it's not always the most convenient for us as we've all struggled with this in the past.
I'd like to see the API of this library!
This is nothing new, just wanted to present another use case that highlights something we're trying to do, but doesn't seem to be too easy at the moment, and looking for suggestions about how to proceed.
Yeah. I'm not trying to overstate things, but I think this issue is the most important "soft" design issue facing us. I hope that there is plenty of talk about it at the connect! Let me brain dump a bit.
I guess I'm assuming a that the "CI loop" as currently demonstrated on the kernel ci page is going to become the core idea for building this kind of view.
Bundle streams are (in practice, at least) a vertical grouping by my definition, but the reason they fall short is that a test is tied to a bundle is tied to a stream whereas what we need are vertical groupings that are tied to a particular report.
So how could we specify the vertical and horizontal groupings? Currently, we use the only tool we really have: testrun attributes (we do have a few others, like bundle stream, the name of a result's test case and analyzer_assigned_date). For example:
"rootfs.date": "20120426",
might be a horizontal grouping and
"target.device_type": "snowball_sd",
might be a vertical one. I think tags might also become useful vertical groupings.
I think there might be some mileage in having a way of defining a report as: a bunch of 'bound' vertical groupings (for example target_device=='panda', build_type=='android'), one 'free' vertical grouping (for example the thing that says whether it's stable/tracking/...) and one 'free' horizontal tag (date of build or something -- build number wouldn't _quite_ work as it doesn't like up between the columns). The output would have a variable number of columns depending on how many values the free vertical grouping takes.
I've over simplified a bit here -- not thinking about how to format each "box" on the report e.g. in the way the kernel ci view shows compile result, boot result, lava-test results. Let's ignore that for now :)
One of the problems this approach has is that the data that feeds into the groupings needs to be specified ahead of time, it needs to go into the result bundle somehow. Maybe it would be nice to allow grouping values to be defined after the fact as derived data of other parts of the test run (this bit feels like how one defines a view in CouchDB somehow).
Anyway, apologies for the brain dump and the likely over-thinking! Any thoughts you guys have would be _most_ appreciated.
Cheers, mwh
On Thu, May 10, 2012 at 10:47 PM, Michael Hudson-Doyle < michael.hudson@linaro.org> wrote:
On Tue, 8 May 2012 16:54:49 -0700, Paul Larson paul.larson@linaro.org wrote:
Hi, I've been playing around with this "lava" thing a bit :)
Heh heh.
From the perspective of the QA team, I'm currently interested in a very specific use case, but it's one that I think has some common things with what others want to do: We have test results for various {ubuntu, android} images. I'd like to
see
a current snapshot of the results on those images. I'd also like to see the history for previous results on the same type of image.
FWIW, I've been thinking and talking about this as "horizontal" and "vertical" grouping of test results.
A vertical grouping would be something category based like "all android tests of panda builds from the -staging manifest" or "compile tests of the linus-tip tree" or...
A horizontal grouping would be something essentially time based -- "tests run yesterday" would be one example, although it could also be "tests on kernels built from a particular git commit" or "all tests on images built from a particular build of the developer rootfs".
(Unsurprisingly, the columns on the kernel ci view are example of vertical groupings and the rows horizontal groupings).
This way, as we near release time, if we look at trying to figure out which build is likely to be the best one to use for release candidate testing, we can glance at the results and see where most of our automated tests passed.
So just looking at android as an example, because we already store data there about which image we are testing, which build number, which url. It's pretty simple to pull all bundles in a stream. It's also not too
bad
to look at a test_run in a bundle and get the fact that it has a certain attribute. It's not so easy to say "show me all the bundles that have
this
attribute". Am I missing something?
No.
I think I brought this up before, but I ran across a project (heck if I
can
find it now) that had a python library providing an API to make it convenient for finding your test data. This is what the ORM *should* be providing, I know, but it's not always the most convenient for us as
we've
all struggled with this in the past.
I'd like to see the API of this library!
This is nothing new, just wanted to present another use case that highlights something we're trying to do, but doesn't seem to be too easy
at
the moment, and looking for suggestions about how to proceed.
Yeah. I'm not trying to overstate things, but I think this issue is the most important "soft" design issue facing us. I hope that there is plenty of talk about it at the connect! Let me brain dump a bit.
+1
I guess I'm assuming a that the "CI loop" as currently demonstrated on the kernel ci page is going to become the core idea for building this kind of view.
I *think* so, but I don't want to preclude other options. I think the ci loop model fits a lot of what many people want to achieve though, and is somewhat close to what I described.
Bundle streams are (in practice, at least) a vertical grouping by my definition, but the reason they fall short is that a test is tied to a bundle is tied to a stream whereas what we need are vertical groupings that are tied to a particular report.
They can be, but we don't currently use them that way. To do this, we would have to create a bundle stream for every vertical group: linaro-android/imx6-ics-gcc47-freescalelt-stable-open linaro-android/panda-ics-gcc47-tilt-tracking-blob-4430 linaro-android/panda-ics-gcc47-tilt-tracking-blob-4460 ... https://android-build.linaro.org/builds/%7Elinaro-android/panda-ics-gcc47-tilt-tracking-blob/The issue with this, and why we previously decided to stick to simply an android and an ubuntu stream, is that it makes it much more difficult if you ever want to see *all* the results, for example, all the results for a release together. So instead we opted for a singular stream, combined with metadata which would, in theory, help us find the right bundles. At the moment though, it only helps us find the test runs, not the whole bundles.
So how could we specify the vertical and horizontal groupings?
Currently, we use the only tool we really have: testrun attributes (we do have a few others, like bundle stream, the name of a result's test case and analyzer_assigned_date). For example:
"rootfs.date": "20120426",might be a horizontal grouping and
"target.device_type": "snowball_sd",might be a vertical one. I think tags might also become useful vertical groupings.
Tags are worth exploring, I'm not sure of the state at the moment. If they are usable right now, then maybe we should look at some examples and see about making the results use those instead of attributes, and how they could be used for generating views?
I think there might be some mileage in having a way of defining a report as: a bunch of 'bound' vertical groupings (for example target_device=='panda', build_type=='android'), one 'free' vertical grouping (for example the thing that says whether it's stable/tracking/...) and one 'free' horizontal tag (date of build or something -- build number wouldn't _quite_ work as it doesn't like up between the columns). The output would have a variable number of columns depending on how many values the free vertical grouping takes.
I've over simplified a bit here -- not thinking about how to format each "box" on the report e.g. in the way the kernel ci view shows compile result, boot result, lava-test results. Let's ignore that for now :)
One of the problems this approach has is that the data that feeds into the groupings needs to be specified ahead of time, it needs to go into the result bundle somehow. Maybe it would be nice to allow grouping values to be defined after the fact as derived data of other parts of the test run (this bit feels like how one defines a view in CouchDB somehow).
Anyway, apologies for the brain dump and the likely over-thinking! Any thoughts you guys have would be _most_ appreciated.
Comments added above, would love to hear from others on this. I think it should be a connect topic if it's not already, and maybe even some hacking on it.
linaro-validation@lists.linaro.org
-
Michael Hudson-Doyle -
Paul Larson