Adding Renato, since he's actively trying to work on this type of thing.
Dave
On 9 Jan 2013, at 23:43, Alexander Sack asac@linaro.org wrote:
On Wed, Jan 9, 2013 at 6:07 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Wed, 9 Jan 2013 13:22:40 +0100 Alexander Sack asac@linaro.org wrote:
Thanks for the status update. Sounds good in general. One question: you propose to submit stuff manual for one month to test etc. while cbuild is continuing in the old form for now as our production approach?
I definitely suggest to keep old form as production for this cycle, and then experience will tell if it will be ready for the next. It's not that LAVA integration may be not ready, it's that we replace adhoc, optimized native builders with more flexible LAVA, and that brings issues not directly related to integration per se, like that image lockup/thermal issue. It will require some real load to assess how it's viable/to bust all issues.
Could we somehow submit all jobs that cbuild puts onto the current non-lava-managed boards to LAVA instead of doing manual testing or does the step to use LAVA for job execution too intrusive to keep it running in parallel?
Yeah, that's the idea. Manual testing mentioned above was just for initial smoke test, and then to enable exactly what you propose. CBuild make that pretty easy - it has several levels of indirection for routing builds: there're "spawn queues" which correspond to a specific product, which fans out to several "scheduler queues" which correspond to specific board type, from which it goes to an available board of that type. So, we'll just add symlink(s) to spawn queue to include a LAVA queue, and all should work automagically (synthetically tested on a sandbox, but again, need to see how well it works with real CBuild load and build set).
cool. thanks a bunch. sounds very good then. Last question: do we create a staging UI/dashboard instance that consumes the results/logs from the LAVA produced results to ensure that the complete solution is equivalent as well?
On Wed, Jan 9, 2013 at 1:10 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
This milestone is last in series of 3 to add LAVA integration for CBuild. In anticipation of delays and waits for deployment, mid-December we made an optimistic plan to target to request deployment in the beginning of this milestone. We didn't finish as much as we wanted, but there're 2 main points: 1) we made all changes in a way to preserve original CBuild native builds; 2) the core functionality is done and proved working well on a sandbox.
So, we would like to follow original plan and set on deployment process for the changes, in anticipation that we may need to do incremental rollout (another point easy to forget is that TCWG has earlier deadline, so there's no "good" time for deployment, and starting it early is good way to avoid procrastination).
So, we would like to propose following plan:
- Infra stabilizes current changes (level of functionality: manual
scheduling of builds in LAVA). We already started this, as the changes spread around several bzr repos, we using Launchpad's great Merge Request functionality. It's Infra-only first stage review though, feel free to ignore it.
- We submit 2nd-stage MR for TCWG and LAVA teams to review. ETA is
beginning of the next week.
- Once review is passed (hopefully within a week or shorter), we
schedule a date for merging and deployment, subject to constraint of TCWG release schedule.
- We appoint team which will do deployment, Infra engineers have
access to cbuild.validation.linaro.org, so we can gladly work on that, making sure we can rollback to the currently running version easily.
We invite stakeholders to do manual builds in LAVA.
As remaining functionality is developed (UI tweaks, support for
advanced functionality requiring automated access to build logs), they are deployed in similar manner.
- Automated builds (dailies, MR CI builds, etc.) are configured to
run on LAVA in parallel to native CBuild.
This last point is what we can realistically and reasonably deliver by the end of this milestone, which will enable us to review functionality and stability of LAVA builds, work on resolving remaining infrastructural issues (like stability of build image), assess LAVA capacity, and overall prepare plans for migrating to LAVA as the main build platform for CBuild.
Please let me know how this plan looks.
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
-- Alexander Sack Director, Linaro Platform Engineering http://www.linaro.org | Open source software for ARM SoCs http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
linaro-validation mailing list linaro-validation@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-validation