Hello,
This milestone is last in series of 3 to add LAVA integration for CBuild. In anticipation of delays and waits for deployment, mid-December we made an optimistic plan to target to request deployment in the beginning of this milestone. We didn't finish as much as we wanted, but there're 2 main points: 1) we made all changes in a way to preserve original CBuild native builds; 2) the core functionality is done and proved working well on a sandbox.
So, we would like to follow original plan and set on deployment process for the changes, in anticipation that we may need to do incremental rollout (another point easy to forget is that TCWG has earlier deadline, so there's no "good" time for deployment, and starting it early is good way to avoid procrastination).
So, we would like to propose following plan:
1. Infra stabilizes current changes (level of functionality: manual scheduling of builds in LAVA). We already started this, as the changes spread around several bzr repos, we using Launchpad's great Merge Request functionality. It's Infra-only first stage review though, feel free to ignore it.
2. We submit 2nd-stage MR for TCWG and LAVA teams to review. ETA is beginning of the next week.
3. Once review is passed (hopefully within a week or shorter), we schedule a date for merging and deployment, subject to constraint of TCWG release schedule.
4. We appoint team which will do deployment, Infra engineers have access to cbuild.validation.linaro.org, so we can gladly work on that, making sure we can rollback to the currently running version easily.
5. We invite stakeholders to do manual builds in LAVA.
6. As remaining functionality is developed (UI tweaks, support for advanced functionality requiring automated access to build logs), they are deployed in similar manner.
7. Automated builds (dailies, MR CI builds, etc.) are configured to run on LAVA in parallel to native CBuild.
This last point is what we can realistically and reasonably deliver by the end of this milestone, which will enable us to review functionality and stability of LAVA builds, work on resolving remaining infrastructural issues (like stability of build image), assess LAVA capacity, and overall prepare plans for migrating to LAVA as the main build platform for CBuild.
Please let me know how this plan looks.
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
Thanks for the status update. Sounds good in general. One question: you propose to submit stuff manual for one month to test etc. while cbuild is continuing in the old form for now as our production approach?
Could we somehow submit all jobs that cbuild puts onto the current non-lava-managed boards to LAVA instead of doing manual testing or does the step to use LAVA for job execution too intrusive to keep it running in parallel?
On Wed, Jan 9, 2013 at 1:10 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
This milestone is last in series of 3 to add LAVA integration for CBuild. In anticipation of delays and waits for deployment, mid-December we made an optimistic plan to target to request deployment in the beginning of this milestone. We didn't finish as much as we wanted, but there're 2 main points: 1) we made all changes in a way to preserve original CBuild native builds; 2) the core functionality is done and proved working well on a sandbox.
So, we would like to follow original plan and set on deployment process for the changes, in anticipation that we may need to do incremental rollout (another point easy to forget is that TCWG has earlier deadline, so there's no "good" time for deployment, and starting it early is good way to avoid procrastination).
So, we would like to propose following plan:
- Infra stabilizes current changes (level of functionality: manual
scheduling of builds in LAVA). We already started this, as the changes spread around several bzr repos, we using Launchpad's great Merge Request functionality. It's Infra-only first stage review though, feel free to ignore it.
- We submit 2nd-stage MR for TCWG and LAVA teams to review. ETA is
beginning of the next week.
- Once review is passed (hopefully within a week or shorter), we
schedule a date for merging and deployment, subject to constraint of TCWG release schedule.
- We appoint team which will do deployment, Infra engineers have
access to cbuild.validation.linaro.org, so we can gladly work on that, making sure we can rollback to the currently running version easily.
We invite stakeholders to do manual builds in LAVA.
As remaining functionality is developed (UI tweaks, support for
advanced functionality requiring automated access to build logs), they are deployed in similar manner.
- Automated builds (dailies, MR CI builds, etc.) are configured to run
on LAVA in parallel to native CBuild.
This last point is what we can realistically and reasonably deliver by the end of this milestone, which will enable us to review functionality and stability of LAVA builds, work on resolving remaining infrastructural issues (like stability of build image), assess LAVA capacity, and overall prepare plans for migrating to LAVA as the main build platform for CBuild.
Please let me know how this plan looks.
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
On Wed, 9 Jan 2013 13:22:40 +0100 Alexander Sack asac@linaro.org wrote:
Thanks for the status update. Sounds good in general. One question: you propose to submit stuff manual for one month to test etc. while cbuild is continuing in the old form for now as our production approach?
I definitely suggest to keep old form as production for this cycle, and then experience will tell if it will be ready for the next. It's not that LAVA integration may be not ready, it's that we replace adhoc, optimized native builders with more flexible LAVA, and that brings issues not directly related to integration per se, like that image lockup/thermal issue. It will require some real load to assess how it's viable/to bust all issues.
Could we somehow submit all jobs that cbuild puts onto the current non-lava-managed boards to LAVA instead of doing manual testing or does the step to use LAVA for job execution too intrusive to keep it running in parallel?
Yeah, that's the idea. Manual testing mentioned above was just for initial smoke test, and then to enable exactly what you propose. CBuild make that pretty easy - it has several levels of indirection for routing builds: there're "spawn queues" which correspond to a specific product, which fans out to several "scheduler queues" which correspond to specific board type, from which it goes to an available board of that type. So, we'll just add symlink(s) to spawn queue to include a LAVA queue, and all should work automagically (synthetically tested on a sandbox, but again, need to see how well it works with real CBuild load and build set).
On Wed, Jan 9, 2013 at 1:10 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
This milestone is last in series of 3 to add LAVA integration for CBuild. In anticipation of delays and waits for deployment, mid-December we made an optimistic plan to target to request deployment in the beginning of this milestone. We didn't finish as much as we wanted, but there're 2 main points: 1) we made all changes in a way to preserve original CBuild native builds; 2) the core functionality is done and proved working well on a sandbox.
So, we would like to follow original plan and set on deployment process for the changes, in anticipation that we may need to do incremental rollout (another point easy to forget is that TCWG has earlier deadline, so there's no "good" time for deployment, and starting it early is good way to avoid procrastination).
So, we would like to propose following plan:
- Infra stabilizes current changes (level of functionality: manual
scheduling of builds in LAVA). We already started this, as the changes spread around several bzr repos, we using Launchpad's great Merge Request functionality. It's Infra-only first stage review though, feel free to ignore it.
- We submit 2nd-stage MR for TCWG and LAVA teams to review. ETA is
beginning of the next week.
- Once review is passed (hopefully within a week or shorter), we
schedule a date for merging and deployment, subject to constraint of TCWG release schedule.
- We appoint team which will do deployment, Infra engineers have
access to cbuild.validation.linaro.org, so we can gladly work on that, making sure we can rollback to the currently running version easily.
We invite stakeholders to do manual builds in LAVA.
As remaining functionality is developed (UI tweaks, support for
advanced functionality requiring automated access to build logs), they are deployed in similar manner.
- Automated builds (dailies, MR CI builds, etc.) are configured to
run on LAVA in parallel to native CBuild.
This last point is what we can realistically and reasonably deliver by the end of this milestone, which will enable us to review functionality and stability of LAVA builds, work on resolving remaining infrastructural issues (like stability of build image), assess LAVA capacity, and overall prepare plans for migrating to LAVA as the main build platform for CBuild.
Please let me know how this plan looks.
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
On Wed, Jan 9, 2013 at 6:07 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Wed, 9 Jan 2013 13:22:40 +0100 Alexander Sack asac@linaro.org wrote:
Thanks for the status update. Sounds good in general. One question: you propose to submit stuff manual for one month to test etc. while cbuild is continuing in the old form for now as our production approach?
I definitely suggest to keep old form as production for this cycle, and then experience will tell if it will be ready for the next. It's not that LAVA integration may be not ready, it's that we replace adhoc, optimized native builders with more flexible LAVA, and that brings issues not directly related to integration per se, like that image lockup/thermal issue. It will require some real load to assess how it's viable/to bust all issues.
Could we somehow submit all jobs that cbuild puts onto the current non-lava-managed boards to LAVA instead of doing manual testing or does the step to use LAVA for job execution too intrusive to keep it running in parallel?
Yeah, that's the idea. Manual testing mentioned above was just for initial smoke test, and then to enable exactly what you propose. CBuild make that pretty easy - it has several levels of indirection for routing builds: there're "spawn queues" which correspond to a specific product, which fans out to several "scheduler queues" which correspond to specific board type, from which it goes to an available board of that type. So, we'll just add symlink(s) to spawn queue to include a LAVA queue, and all should work automagically (synthetically tested on a sandbox, but again, need to see how well it works with real CBuild load and build set).
cool. thanks a bunch. sounds very good then. Last question: do we create a staging UI/dashboard instance that consumes the results/logs from the LAVA produced results to ensure that the complete solution is equivalent as well?
On Wed, Jan 9, 2013 at 1:10 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
This milestone is last in series of 3 to add LAVA integration for CBuild. In anticipation of delays and waits for deployment, mid-December we made an optimistic plan to target to request deployment in the beginning of this milestone. We didn't finish as much as we wanted, but there're 2 main points: 1) we made all changes in a way to preserve original CBuild native builds; 2) the core functionality is done and proved working well on a sandbox.
So, we would like to follow original plan and set on deployment process for the changes, in anticipation that we may need to do incremental rollout (another point easy to forget is that TCWG has earlier deadline, so there's no "good" time for deployment, and starting it early is good way to avoid procrastination).
So, we would like to propose following plan:
- Infra stabilizes current changes (level of functionality: manual
scheduling of builds in LAVA). We already started this, as the changes spread around several bzr repos, we using Launchpad's great Merge Request functionality. It's Infra-only first stage review though, feel free to ignore it.
- We submit 2nd-stage MR for TCWG and LAVA teams to review. ETA is
beginning of the next week.
- Once review is passed (hopefully within a week or shorter), we
schedule a date for merging and deployment, subject to constraint of TCWG release schedule.
- We appoint team which will do deployment, Infra engineers have
access to cbuild.validation.linaro.org, so we can gladly work on that, making sure we can rollback to the currently running version easily.
We invite stakeholders to do manual builds in LAVA.
As remaining functionality is developed (UI tweaks, support for
advanced functionality requiring automated access to build logs), they are deployed in similar manner.
- Automated builds (dailies, MR CI builds, etc.) are configured to
run on LAVA in parallel to native CBuild.
This last point is what we can realistically and reasonably deliver by the end of this milestone, which will enable us to review functionality and stability of LAVA builds, work on resolving remaining infrastructural issues (like stability of build image), assess LAVA capacity, and overall prepare plans for migrating to LAVA as the main build platform for CBuild.
Please let me know how this plan looks.
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
Adding Renato, since he's actively trying to work on this type of thing.
Dave
On 9 Jan 2013, at 23:43, Alexander Sack asac@linaro.org wrote:
On Wed, Jan 9, 2013 at 6:07 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Wed, 9 Jan 2013 13:22:40 +0100 Alexander Sack asac@linaro.org wrote:
Thanks for the status update. Sounds good in general. One question: you propose to submit stuff manual for one month to test etc. while cbuild is continuing in the old form for now as our production approach?
I definitely suggest to keep old form as production for this cycle, and then experience will tell if it will be ready for the next. It's not that LAVA integration may be not ready, it's that we replace adhoc, optimized native builders with more flexible LAVA, and that brings issues not directly related to integration per se, like that image lockup/thermal issue. It will require some real load to assess how it's viable/to bust all issues.
Could we somehow submit all jobs that cbuild puts onto the current non-lava-managed boards to LAVA instead of doing manual testing or does the step to use LAVA for job execution too intrusive to keep it running in parallel?
Yeah, that's the idea. Manual testing mentioned above was just for initial smoke test, and then to enable exactly what you propose. CBuild make that pretty easy - it has several levels of indirection for routing builds: there're "spawn queues" which correspond to a specific product, which fans out to several "scheduler queues" which correspond to specific board type, from which it goes to an available board of that type. So, we'll just add symlink(s) to spawn queue to include a LAVA queue, and all should work automagically (synthetically tested on a sandbox, but again, need to see how well it works with real CBuild load and build set).
cool. thanks a bunch. sounds very good then. Last question: do we create a staging UI/dashboard instance that consumes the results/logs from the LAVA produced results to ensure that the complete solution is equivalent as well?
On Wed, Jan 9, 2013 at 1:10 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
This milestone is last in series of 3 to add LAVA integration for CBuild. In anticipation of delays and waits for deployment, mid-December we made an optimistic plan to target to request deployment in the beginning of this milestone. We didn't finish as much as we wanted, but there're 2 main points: 1) we made all changes in a way to preserve original CBuild native builds; 2) the core functionality is done and proved working well on a sandbox.
So, we would like to follow original plan and set on deployment process for the changes, in anticipation that we may need to do incremental rollout (another point easy to forget is that TCWG has earlier deadline, so there's no "good" time for deployment, and starting it early is good way to avoid procrastination).
So, we would like to propose following plan:
- Infra stabilizes current changes (level of functionality: manual
scheduling of builds in LAVA). We already started this, as the changes spread around several bzr repos, we using Launchpad's great Merge Request functionality. It's Infra-only first stage review though, feel free to ignore it.
- We submit 2nd-stage MR for TCWG and LAVA teams to review. ETA is
beginning of the next week.
- Once review is passed (hopefully within a week or shorter), we
schedule a date for merging and deployment, subject to constraint of TCWG release schedule.
- We appoint team which will do deployment, Infra engineers have
access to cbuild.validation.linaro.org, so we can gladly work on that, making sure we can rollback to the currently running version easily.
We invite stakeholders to do manual builds in LAVA.
As remaining functionality is developed (UI tweaks, support for
advanced functionality requiring automated access to build logs), they are deployed in similar manner.
- Automated builds (dailies, MR CI builds, etc.) are configured to
run on LAVA in parallel to native CBuild.
This last point is what we can realistically and reasonably deliver by the end of this milestone, which will enable us to review functionality and stability of LAVA builds, work on resolving remaining infrastructural issues (like stability of build image), assess LAVA capacity, and overall prepare plans for migrating to LAVA as the main build platform for CBuild.
Please let me know how this plan looks.
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
-- Alexander Sack Director, Linaro Platform Engineering http://www.linaro.org | Open source software for ARM SoCs http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
linaro-validation mailing list linaro-validation@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-validation
Hello,
On Thu, 10 Jan 2013 00:43:42 +0100 Alexander Sack asac@linaro.org wrote:
[]
CBuild make that pretty easy - it has several levels of indirection for routing builds: there're "spawn queues" which correspond to a specific product, which fans out to several "scheduler queues" which correspond to specific board type, from which it goes to an available board of that type. So, we'll just add symlink(s) to spawn queue to include a LAVA queue, and all should work automagically (synthetically tested on a sandbox, but again, need to see how well it works with real CBuild load and build set).
cool. thanks a bunch. sounds very good then. Last question: do we create a staging UI/dashboard instance that consumes the results/logs from the LAVA produced results to ensure that the complete solution is equivalent as well?
I assume "staging UI/dashboard instance" of a CBuild web frontend itself, because as we decided at last Connect, "episode 1" will be centered around bringing LAVA as a backend to CBuild, and keep using its frontend as is.
So yes, that's exactly what's left on development front of things - to bring build logs from LAVA back to CBuild, so CBuild advanced functionality (build/benchmark diff viewer, etc.) worked. Stevan and Milo work on that as we speak, using http://ec2-54-243-13-36.compute-1.amazonaws.com/helpers/ as a staging server.
Paul,
This sounds good. In particular:
On 09/01/13 12:10, Paul Sokolovsky wrote:
- Once review is passed (hopefully within a week or shorter), we
schedule a date for merging and deployment, subject to constraint of TCWG release schedule.
Completing reviews so deployment can happen in the week beginning 21st Jan is good for us (as you suggest), as this is the week after release and gives two weeks to sort any issues out before we start merges again.
Thanks,
Matt
On Wed, Jan 9, 2013 at 1:49 PM, Matthew Gretton-Dann matthew.gretton-dann@linaro.org wrote:
Paul,
This sounds good. In particular:
On 09/01/13 12:10, Paul Sokolovsky wrote:
- Once review is passed (hopefully within a week or shorter), we
schedule a date for merging and deployment, subject to constraint of TCWG release schedule.
Completing reviews so deployment can happen in the week beginning 21st Jan is good for us (as you suggest), as this is the week after release and gives two weeks to sort any issues out before we start merges again.
And remember: ensure that we can easily go back to old setup in case things fall apart and put our release at risk :)
Thanks,
Matt
-- Matthew Gretton-Dann Toolchain Working Group, Linaro
On 01/09/2013 06:10 AM, Paul Sokolovsky wrote:
- Once review is passed (hopefully within a week or shorter), we
schedule a date for merging and deployment, subject to constraint of TCWG release schedule.
This seems like it could be the item that could cause the other stuff to slip. I don't think you have lava specific stuff, so its not really a big personal concern - just more curious how much code we are talking about here?
Hello Andy,
On Wed, 09 Jan 2013 09:56:02 -0600 Andy Doan andy.doan@linaro.org wrote:
On 01/09/2013 06:10 AM, Paul Sokolovsky wrote:
- Once review is passed (hopefully within a week or shorter), we
schedule a date for merging and deployment, subject to constraint of TCWG release schedule.
This seems like it could be the item that could cause the other stuff to slip. I don't think you have lava specific stuff, so its not really a big personal concern - just more curious how much code we are talking about here?
Currently, all code changes are localized to CBuild components (thanks for fixing all previously reported LAVA issues), but those are spread across 5 component repos, they're still not too big otherwise.
Paul Sokolovsky paul.sokolovsky@linaro.org writes:
Hello,
This milestone is last in series of 3 to add LAVA integration for CBuild. In anticipation of delays and waits for deployment, mid-December we made an optimistic plan to target to request deployment in the beginning of this milestone. We didn't finish as much as we wanted, but there're 2 main points: 1) we made all changes in a way to preserve original CBuild native builds; 2) the core functionality is done and proved working well on a sandbox.
So, we would like to follow original plan and set on deployment process for the changes, in anticipation that we may need to do incremental rollout (another point easy to forget is that TCWG has earlier deadline, so there's no "good" time for deployment, and starting it early is good way to avoid procrastination).
It doesn't sounds like much is being asked of the LAVA team here (feel free to correct this if I'm wrong) but I should point out that we do LAVA deployments pretty frequently, probably approximately weekly currently but more frequently is possible too -- we very definitely do not just stick to deploying once a month.
Cheers, mwh
Hello Michael,
On Thu, 10 Jan 2013 09:41:55 +1300 Michael Hudson-Doyle michael.hudson@linaro.org wrote:
Paul Sokolovsky paul.sokolovsky@linaro.org writes:
Hello,
This milestone is last in series of 3 to add LAVA integration for CBuild. In anticipation of delays and waits for deployment, mid-December we made an optimistic plan to target to request deployment in the beginning of this milestone. We didn't finish as much as we wanted, but there're 2 main points: 1) we made all changes in a way to preserve original CBuild native builds; 2) the core functionality is done and proved working well on a sandbox.
So, we would like to follow original plan and set on deployment process for the changes, in anticipation that we may need to do incremental rollout (another point easy to forget is that TCWG has earlier deadline, so there's no "good" time for deployment, and starting it early is good way to avoid procrastination).
It doesn't sounds like much is being asked of the LAVA team here (feel free to correct this if I'm wrong),
Indeed, it's mostly just reviewing LAVA API usage and lava_test_shell testdefs (I don't expect any big issues there, as it was done based on regular communication with you guys), and just being aware of the launch plan and that Infra will be doing changes on cbuild.validation.linaro.org VM.
but I should point out that we do LAVA deployments pretty frequently, probably approximately weekly currently but more frequently is possible too -- we very definitely do not just stick to deploying once a month.
Yes, thanks much for prompt responses to all issues we faced so far - we understand lava_test_shell et al. is WIP and may break, and it does from time to time for us (like yesterday's log attachment issue), but turnaround times a very good (yesterday I noticed that even https://bugs.launchpad.net/lava-dashboard/+bug/1085989 feature request was implemented already ;-) ).
Cheers, mwh
Adding Renato, since he's an interested party to this discussion.
Dave
On 9 Jan 2013, at 12:10, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
This milestone is last in series of 3 to add LAVA integration for CBuild. In anticipation of delays and waits for deployment, mid-December we made an optimistic plan to target to request deployment in the beginning of this milestone. We didn't finish as much as we wanted, but there're 2 main points: 1) we made all changes in a way to preserve original CBuild native builds; 2) the core functionality is done and proved working well on a sandbox.
So, we would like to follow original plan and set on deployment process for the changes, in anticipation that we may need to do incremental rollout (another point easy to forget is that TCWG has earlier deadline, so there's no "good" time for deployment, and starting it early is good way to avoid procrastination).
So, we would like to propose following plan:
- Infra stabilizes current changes (level of functionality: manual
scheduling of builds in LAVA). We already started this, as the changes spread around several bzr repos, we using Launchpad's great Merge Request functionality. It's Infra-only first stage review though, feel free to ignore it.
- We submit 2nd-stage MR for TCWG and LAVA teams to review. ETA is
beginning of the next week.
- Once review is passed (hopefully within a week or shorter), we
schedule a date for merging and deployment, subject to constraint of TCWG release schedule.
- We appoint team which will do deployment, Infra engineers have
access to cbuild.validation.linaro.org, so we can gladly work on that, making sure we can rollback to the currently running version easily.
We invite stakeholders to do manual builds in LAVA.
As remaining functionality is developed (UI tweaks, support for
advanced functionality requiring automated access to build logs), they are deployed in similar manner.
- Automated builds (dailies, MR CI builds, etc.) are configured to run
on LAVA in parallel to native CBuild.
This last point is what we can realistically and reasonably deliver by the end of this milestone, which will enable us to review functionality and stability of LAVA builds, work on resolving remaining infrastructural issues (like stability of build image), assess LAVA capacity, and overall prepare plans for migrating to LAVA as the main build platform for CBuild.
Please let me know how this plan looks.
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
linaro-validation mailing list linaro-validation@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-validation
linaro-validation@lists.linaro.org
-
Alexander Sack -
Andy Doan -
Dave Pigott -
Matthew Gretton-Dann -
Michael Hudson-Doyle -
Paul Sokolovsky