Hello,
As a heads-up for CBuild/LAVA integration project, http://ec2-54-243-13-36.compute-1.amazonaws.com/helpers/scheduler is now available, which has initial(?) web frontend integration with LAVA. That page (currently) shows job "gcc-linaro-4.6-2012.09.job", submitted against queue "lava-panda-mock", with a link to latest build in LAVA (extra feature, I couldn't find a way in CBuild frontend to get to native build results).
If you want to try it yourself, it's accessible to same TCWG people as http://cbuild.validation.linaro.org/ , plus me, Stevan, and Danilo. So, if you're not in list, please drop us a line. "Spawn" and "Bump up" actions support integration with LAVA currently. We're in the process of identifying what other features of web frontend would require LAVA integration, hints from TCWG are appreciated.
Now a bit about architecture. We're targetting to support both native CBuild and LAVA schedulers. This is per-queue, i.e. one queue can use native scheduler and another LAVA. LAVA is detected by the presence of LAVA job definition template file in a scheduler queue dir: http://bazaar.launchpad.net/~linaro-infrastructure/cbuild/cbuild-scheduler_c... lava-test-shell script with actual commands to run for build is also served by CBuild frontend based on a template in that dir.
Currently, sandbox defines 3 test queue for LAVA integration:
lava-panda - build something on panda (tested only with gcc, needs more work)
lava-panda-mock - doesn't really compiles stuff, just downloads snapshot of previous gcc build. This allows for quicker (~15min) turnaround during testing, if you test snadbox, please use this queue.
lava-qemu-mininal - for local development with even shorter turnaround, won't work on sandbox.
If you have any feedback on the architecture above, please share it. Otherwise, we're proceeding to support it in cbuild-tools.
Paul,
Before I start picking holes and asking questios: Thanks for doing this it looks like you are starting on the right track!
On Wednesday 12 Dec 2012 13:17:05 Paul Sokolovsky wrote:
Hello,
As a heads-up for CBuild/LAVA integration project, http://ec2-54-243-13-36.compute-1.amazonaws.com/helpers/scheduler is now available, which has initial(?) web frontend integration with LAVA. That page (currently) shows job "gcc-linaro-4.6-2012.09.job", submitted against queue "lava-panda-mock", with a link to latest build in LAVA (extra feature, I couldn't find a way in CBuild frontend to get to native build results).
So as soon as a board starts executing a job it should no longer appear in the 'Pending' list - just the Jobs list see: http://cbuild.validation.linaro.org/helpers/scheduler This is the moment the 'Taken' column gets filled in.
You know a job has finished because it appears in recent: http://cbuild.validation.linaro.org/helpers/recent These have a direct link to the build logs.
You can also get to the build logs via: http://cbuild.validation.linaro.org/helpers/buildlog
If you want to try it yourself, it's accessible to same TCWG people as http://cbuild.validation.linaro.org/ , plus me, Stevan, and Danilo. So, if you're not in list, please drop us a line. "Spawn" and "Bump up" actions support integration with LAVA currently. We're in the process of identifying what other features of web frontend would require LAVA integration, hints from TCWG are appreciated.
Spawn and Bump up seem a good start.
'Release' would be useful as well - which actually means: 'Respawn' (i.e. do it again)
Another one which would be useful (and which we currently do manually with "killall make" on toolchain64.lab) is: Kill all current jobs. This is useful in release week when we want to clear the way for the release builds.
Now a bit about architecture. We're targetting to support both native CBuild and LAVA schedulers. This is per-queue, i.e. one queue can use native scheduler and another LAVA. LAVA is detected by the presence of LAVA job definition template file in a scheduler queue dir: http://bazaar.launchpad.net/~linaro-infrastructure/cbuild/cbuild-scheduler_c build-lava/files/head:/queue/lava-panda-mock/ lava-test-shell script with actual commands to run for build is also served by CBuild frontend based on a template in that dir.
Currently, sandbox defines 3 test queue for LAVA integration:
lava-panda - build something on panda (tested only with gcc, needs more work)
lava-panda-mock - doesn't really compiles stuff, just downloads snapshot of previous gcc build. This allows for quicker (~15min) turnaround during testing, if you test snadbox, please use this queue.
lava-qemu-mininal - for local development with even shorter turnaround, won't work on sandbox.
If you have any feedback on the architecture above, please share it. Otherwise, we're proceeding to support it in cbuild-tools.
So I do have some Lava issues - which are probably mostly due to Lava terms not being what I would expect them to mean:
The following Link is to the 'test results bundle' https://validation.linaro.org/lava- server/dashboard/streams/anonymous/cbuild/bundles/9c87b6a32ef38e06d081f12f115c08e79ce89f8b/3df9cc54-60ec-484b-91bf- cc4d34e3a494/
This shows four test cases - which reflect the historic build steps shown under 'Details' in the historic Scheduler (gcc-configure, gcc-build, gcc-install, cbuild-global)
So currently these seem to reflect stages in the build process, rather than actual test results. Eventually you will run the 'gcc-testsuite' build stage which actually does the regression testing (all 100K+ of tests) and I don't expect that to succeed usually (in terms of make check returning 0), although it will succeed in running the testsuite.
What is the intention here? Are we going to have Lava be the high-level builder, and so its 'tests' are each of the build steps, and then the testsuite analysis is done elsewhere? This is the current setup (see http://cbuild.validation.linaro.org/helpers/testcompare/gcc- linaro-4.6-2012.09/logs/x86_64-precise-cbuild367-oort3-x86_64r1/gcc- testsuite.txt - except this is currently returning an Internal Server Error). Or is the intention for Lava to handle all of the gcc-testsuite results individually?
Thanks,
Matt
У сре, 12. 12 2012. у 12:18 +0000, Matthew Gretton-Dann пише:
What is the intention here? Are we going to have Lava be the high-level builder, and so its 'tests' are each of the build steps, and then the testsuite analysis is done elsewhere?
Yes, in this (first) phase, LAVA is only supposed to be used as a "native builder farm". Any LAVA specifics should, for now, be considered implementation details: i.e. a build happens just like before and cbuild collects the results and you look at them there.
LAVA does give us access to some lower level details there, but we are not using it for what it was originally designed to do.
The effort happening here doesn't tie in (directly) with Michael Hope's departure: it was planned a while ago to move from static board allocation to using the dynamic board allocation provided by LAVA. Next steps in that vision are to migrate cross-builds to jenkins (i.e. move all the builds to our shared infrastructure), and then integrate all the useful features of cbuild into our existing CI systems.
For now, we'll also simply run the gcc testsuite as part of the build, just like it was done so far (as far as I understand it). We want to get to the goal (of TCWG using the common CI/validation infrastructure) step by step.
Cheers, Danilo
On Wed, 12 Dec 2012 12:18:56 +0000 Matthew Gretton-Dann matthew.gretton-dann@linaro.org wrote:
Paul,
Before I start picking holes and asking questios: Thanks for doing this it looks like you are starting on the right track!
On Wednesday 12 Dec 2012 13:17:05 Paul Sokolovsky wrote:
Hello,
As a heads-up for CBuild/LAVA integration project, http://ec2-54-243-13-36.compute-1.amazonaws.com/helpers/scheduler is now available, which has initial(?) web frontend integration with LAVA. That page (currently) shows job "gcc-linaro-4.6-2012.09.job", submitted against queue "lava-panda-mock", with a link to latest build in LAVA (extra feature, I couldn't find a way in CBuild frontend to get to native build results).
So as soon as a board starts executing a job it should no longer appear in the 'Pending' list - just the Jobs list see: http://cbuild.validation.linaro.org/helpers/scheduler This is the moment the 'Taken' column gets filled in.
Thanks, we'll look into managing .lock file more consistently for LAVA scheduler case.
You know a job has finished because it appears in recent: http://cbuild.validation.linaro.org/helpers/recent These have a direct link to the build logs.
You can also get to the build logs via: http://cbuild.validation.linaro.org/helpers/buildlog
Ok, we'll look how to implement those as close as possible with LAVA.
One thing we don't fully understand is "build" concept as used in CBuild. For example, in Jenkins it's simple - there's "job" which is kind of class and "build" which is kind of instance of job, built against particular revision of sources and capturing build logs (so, for each job, it's possible to browse all recent builds and their results). But with CBuild, it seems to me that it tracks only latest build of a job. What happens with previous builds - are they just removed, or their build artifacts still available there in some distinct directory, just not referenced in web frontend?
If you want to try it yourself, it's accessible to same TCWG people as http://cbuild.validation.linaro.org/ , plus me, Stevan, and Danilo. So, if you're not in list, please drop us a line. "Spawn" and "Bump up" actions support integration with LAVA currently. We're in the process of identifying what other features of web frontend would require LAVA integration, hints from TCWG are appreciated.
Spawn and Bump up seem a good start.
'Release' would be useful as well - which actually means: 'Respawn' (i.e. do it again)
Ok, will look into that.
Another one which would be useful (and which we currently do manually with "killall make" on toolchain64.lab) is: Kill all current jobs. This is useful in release week when we want to clear the way for the release builds.
Well, that likely will need to go as separate feature request for LAVA first, I'm not sure if that supported.
Now a bit about architecture. We're targetting to support both native CBuild and LAVA schedulers. This is per-queue, i.e. one queue can use native scheduler and another LAVA. LAVA is detected by the presence of LAVA job definition template file in a scheduler queue dir: http://bazaar.launchpad.net/~linaro-infrastructure/cbuild/cbuild-scheduler_c build-lava/files/head:/queue/lava-panda-mock/ lava-test-shell script with actual commands to run for build is also served by CBuild frontend based on a template in that dir.
Currently, sandbox defines 3 test queue for LAVA integration:
lava-panda - build something on panda (tested only with gcc, needs more work)
lava-panda-mock - doesn't really compiles stuff, just downloads snapshot of previous gcc build. This allows for quicker (~15min) turnaround during testing, if you test snadbox, please use this queue.
lava-qemu-mininal - for local development with even shorter turnaround, won't work on sandbox.
If you have any feedback on the architecture above, please share it. Otherwise, we're proceeding to support it in cbuild-tools.
So I do have some Lava issues - which are probably mostly due to Lava terms not being what I would expect them to mean:
The following Link is to the 'test results bundle' https://validation.linaro.org/lava- server/dashboard/streams/anonymous/cbuild/bundles/9c87b6a32ef38e06d081f12f115c08e79ce89f8b/3df9cc54-60ec-484b-91bf- cc4d34e3a494/
Thanks for looking into that. I wanted add few sentences about LAVA result bundle browsing, but skipped that to not make mail too long, so I'd appreciate your exploring it.
This shows four test cases - which reflect the historic build steps shown under 'Details' in the historic Scheduler (gcc-configure, gcc-build, gcc-install, cbuild-global)
So currently these seem to reflect stages in the build process, rather than actual test results.
Yes. Using LAVA as a backend for this project was my concern from the very beginning. Fortunately, architecturely and feature-wise LAVA fits well, but there's another "organizational/representative" aspect of it - all the LAVA is build around "test" paradigm, which includes terminology, application wording, documentation, etc. At the CBuild/LAVA initiation session at the last Connect, I raised concern that it's both confusing and "looking hacky" and asked if we should discuss ways to deal with that. My concern wasn't shared at that time though. Well, I'm still sure it's valid, so thanks for sharing your experiences, hopefully this will catch attention from the other stakeholders. From my side, I submitted https://bugs.launchpad.net/lava-dashboard/+bug/1089384 as one small starting step to make it less confusing.
Eventually you will run the 'gcc-testsuite' build stage which actually does the regression testing (all 100K+ of tests) and I don't expect that to succeed usually (in terms of make check returning 0), although it will succeed in running the testsuite.
What is the intention here? Are we going to have Lava be the high-level builder, and so its 'tests' are each of the build steps, and then the testsuite analysis is done elsewhere? This is the current setup (see http://cbuild.validation.linaro.org/helpers/testcompare/gcc- linaro-4.6-2012.09/logs/x86_64-precise-cbuild367-oort3-x86_64r1/gcc- testsuite.txt - except this is currently returning an Internal Server Error). Or is the intention for Lava to handle all of the gcc-testsuite results individually?
My intention was to explore what result representation LAVA allows, and actually to have experience with that. And it's pretty flexible, so both of options you mentioned are possible (recording each case individually will of course require more work - either add commands to record each testcase results to existing makefiles/scripts, or to write parser for test output, or both).
Which exact to use is up to the stakeholders, and but that I mean not only TCWG, but also Validation, QA, Release Managers, Platform in general, which may be interested in detailed and standardly represented test details for toolchain. I see Infrastructure's role as ensuring the various options are possible, and providing focused documentation/examples how to achieve that. So, that's definitely something to discuss, but likely in 13.01, once basic integration is complete.
Thanks,
Matt
On Wednesday 12 Dec 2012 16:08:59 Paul Sokolovsky wrote:
On Wed, 12 Dec 2012 12:18:56 +0000 Matthew Gretton-Dann matthew.gretton-dann@linaro.org wrote:
On Wednesday 12 Dec 2012 13:17:05 Paul Sokolovsky wrote:
One thing we don't fully understand is "build" concept as used in CBuild. For example, in Jenkins it's simple - there's "job" which is kind of class and "build" which is kind of instance of job, built against particular revision of sources and capturing build logs (so, for each job, it's possible to browse all recent builds and their results). But with CBuild, it seems to me that it tracks only latest build of a job. What happens with previous builds - are they just removed, or their build artifacts still available there in some distinct directory, just not referenced in web frontend?
For CBuild a Job in the web-interface directly correlates to a .job file in a queue (say FOO.job). So there may be a FOO.job file in many queues.
From what you describe I think FOO.job relates to a Jenkins build most
directly.
The Jenkins 'job' concept is builtin to cbuild, and CBuild will take a FOO.job and will determine what to do based upon the name of the job (FOO) and some flags in FOO.job.
Respins overwrite the previous job's output by default - you have to add some magic to the FOO.job file to make the two outputs live side by side.
What is the intention here? Are we going to have Lava be the high-level builder, and so its 'tests' are each of the build steps, and then the testsuite analysis is done elsewhere? This is the current setup (see http://cbuild.validation.linaro.org/helpers/testcompare/gcc- linaro-4.6-2012.09/logs/x86_64-precise-cbuild367-oort3-x86_64r1/gcc- testsuite.txt - except this is currently returning an Internal Server Error). Or is the intention for Lava to handle all of the gcc-testsuite results individually?
My intention was to explore what result representation LAVA allows, and actually to have experience with that. And it's pretty flexible, so both of options you mentioned are possible (recording each case individually will of course require more work - either add commands to record each testcase results to existing makefiles/scripts, or to write parser for test output, or both).
Which exact to use is up to the stakeholders, and but that I mean not only TCWG, but also Validation, QA, Release Managers, Platform in general, which may be interested in detailed and standardly represented test details for toolchain. I see Infrastructure's role as ensuring the various options are possible, and providing focused documentation/examples how to achieve that. So, that's definitely something to discuss, but likely in 13.01, once basic integration is complete.
At the moment I do not have a preference - as long as it is clear what the current state is (which it is now).
At some point there will be work needed on results presentation layers - but whether that happens in LAVA or not is not important today.
Thanks,
Matt
linaro-validation@lists.linaro.org
-
Danilo Šegan -
Matthew Gretton-Dann -
Paul Sokolovsky