Hi there. The GCC build time is approaching 24 hours and our five Panda boards can't keep up. Sounds like a good time to LAVAise the toolchain build process a bit more...
Mike Hudson and I talked about doing a hybrid LAVA/cbuild as a first step where LAVA manages the boards and cbuild does the build. The idea is:
* There's a image with the standard toolchain kernel, rootfs, build environment, and build scripts * The LAVA scheduler manages access to the boards * The LAVA dispatcher spins up the board and then invokes the cbuild scripts to build and test GCC
This gives me a fixed environment and re-uses the existing build scripts. In the future these can be replaced with different LAVA components. There's a few details:
* cbuild self updates on start so the image can stay fixed * Full results are pushed by cbuild to 'control' * The dispatcher records the top level log and an overall pass/fail * No other bundles are generated
Most of this fits in with the existing LAVA features. There's a few additions like a 'run command' JSON action, passing the board name to the instance, and passing the job name as an argument. These seem OK.
I'd like some of the boards to be fitted with fast USB flash drives for temporary storage. SD cards are slow and unreliable especially when GCC is involved.
Thoughts?
-- Michael
On Tue, Mar 27, 2012 at 9:26 PM, Michael Hope michael.hope@linaro.orgwrote:
Hi there. The GCC build time is approaching 24 hours and our five Panda boards can't keep up. Sounds like a good time to LAVAise the toolchain build process a bit more...
Mike Hudson and I talked about doing a hybrid LAVA/cbuild as a first step where LAVA manages the boards and cbuild does the build. The idea is:
- There's a image with the standard toolchain kernel, rootfs, build
environment, and build scripts
- The LAVA scheduler manages access to the boards
- The LAVA dispatcher spins up the board and then invokes the cbuild
scripts to build and test GCC
This gives me a fixed environment and re-uses the existing build scripts. In the future these can be replaced with different LAVA components. There's a few details:
- cbuild self updates on start so the image can stay fixed
- Full results are pushed by cbuild to 'control'
- The dispatcher records the top level log and an overall pass/fail
- No other bundles are generated
Most of this fits in with the existing LAVA features. There's a few additions like a 'run command' JSON action, passing the board name to the instance, and passing the job name as an argument. These seem OK.
I'd like some of the boards to be fitted with fast USB flash drives for temporary storage. SD cards are slow and unreliable especially when GCC is involved.
It just so happens that some of the panda boards already have usb drives
attached, and have the tag usb-flash-drive so that jobs can be scheduled in such a way that you can basically say "just give me some panda with a usb flash drive". Right now, it's just a few, but the plan is to add the usb sticks to all boards soon. Dave, do we have them now and we're just planning to put this in as part of the move?
Thanks, Paul Larson
On 28 Mar 2012, at 04:06, Paul Larson wrote:
On Tue, Mar 27, 2012 at 9:26 PM, Michael Hope michael.hope@linaro.org wrote: Hi there. The GCC build time is approaching 24 hours and our five Panda boards can't keep up. Sounds like a good time to LAVAise the toolchain build process a bit more...
Mike Hudson and I talked about doing a hybrid LAVA/cbuild as a first step where LAVA manages the boards and cbuild does the build. The idea is:
- There's a image with the standard toolchain kernel, rootfs, build
environment, and build scripts
- The LAVA scheduler manages access to the boards
- The LAVA dispatcher spins up the board and then invokes the cbuild
scripts to build and test GCC
This gives me a fixed environment and re-uses the existing build scripts. In the future these can be replaced with different LAVA components. There's a few details:
- cbuild self updates on start so the image can stay fixed
- Full results are pushed by cbuild to 'control'
- The dispatcher records the top level log and an overall pass/fail
- No other bundles are generated
Most of this fits in with the existing LAVA features. There's a few additions like a 'run command' JSON action, passing the board name to the instance, and passing the job name as an argument. These seem OK.
I'd like some of the boards to be fitted with fast USB flash drives for temporary storage. SD cards are slow and unreliable especially when GCC is involved.
It just so happens that some of the panda boards already have usb drives attached, and have the tag usb-flash-drive so that jobs can be scheduled in such a way that you can basically say "just give me some panda with a usb flash drive". Right now, it's just a few, but the plan is to add the usb sticks to all boards soon. Dave, do we have them now and we're just planning to put this in as part of the move?
13 out of the 24 pandas have USB flash drives. I have some more and was planning on completing the deployment on the move next week.
Dave
On 28/03/12 03:26, Michael Hope wrote:
Hi there. The GCC build time is approaching 24 hours and our five Panda boards can't keep up. Sounds like a good time to LAVAise the toolchain build process a bit more...
As you know, I've been doing some experiments with this over the last few months. I was blocked by a LAVA bug for a while, but that's been fixed now.
Here's the latest test run (done by Le Chi Thu):
http://validation.linaro.org/lava-server/dashboard/permalink/bundle/d73af579...
The test fails due to a GCC build failure:
//usr/include/linux/errno.h:4:23: fatal error: asm/errno.h: No such file or directory
This surprised me, because I thought it was using the same rootfs you had on the ursa boards, but I've not had time to do anything about it yet.
Andrew
On Wed, Mar 28, 2012 at 10:03 AM, Andrew Stubbs andrew.stubbs@linaro.orgwrote:
On 28/03/12 03:26, Michael Hope wrote:
Hi there. The GCC build time is approaching 24 hours and our five Panda boards can't keep up. Sounds like a good time to LAVAise the toolchain build process a bit more...
As you know, I've been doing some experiments with this over the last few months. I was blocked by a LAVA bug for a while, but that's been fixed now.
Here's the latest test run (done by Le Chi Thu):
http://validation.linaro.org/**lava-server/dashboard/**permalink/bundle/** d73af579ed77957615bd3db2d9055d**82bb14299e/http://validation.linaro.org/lava-server/dashboard/permalink/bundle/d73af579ed77957615bd3db2d9055d82bb14299e/
The test fails due to a GCC build failure:
//usr/include/linux/errno.h:4:**23: fatal error: asm/errno.h: No such file or directory
This surprised me, because I thought it was using the same rootfs you had on the ursa boards, but I've not had time to do anything about it yet.
Looks like something that can likely be resolved by adding a dependency for the test. However, if you need, or if it would be more convenient to have a custom rootfs for this, that's certainly an option. Nothing says we necessarily have to run these tests on nano, developer, etc... but if it's interesting to make it possible for later running this as part of a platform release test, it might be better to make them generic so that they don't depend on a custom rootfs.
Thanks, Paul Larson
On 28/03/12 16:20, Paul Larson wrote:
On Wed, Mar 28, 2012 at 10:03 AM, Andrew Stubbs <andrew.stubbs@linaro.org mailto:andrew.stubbs@linaro.org> wrote: This surprised me, because I thought it was using the same rootfs you had on the ursa boards, but I've not had time to do anything about it yet.
Looks like something that can likely be resolved by adding a dependency for the test. However, if you need, or if it would be more convenient to have a custom rootfs for this, that's certainly an option. Nothing says we necessarily have to run these tests on nano, developer, etc... but if it's interesting to make it possible for later running this as part of a platform release test, it might be better to make them generic so that they don't depend on a custom rootfs.
I didn't!
The test image is composed of
http://releases.linaro.org/platform/linaro-n/hwpacks/11.09/hwpack_linaro-lt-...
and
http://people.linaro.org/~ams/natty-twg-rootfs.tar.gz
The latter is a repackaging of a rootfs supplied to me by Michael. I believe it's related to the ones in use on the boards in the lab (tcpanda*).
Andrew
On 29 March 2012 04:20, Paul Larson paul.larson@linaro.org wrote:
On Wed, Mar 28, 2012 at 10:03 AM, Andrew Stubbs andrew.stubbs@linaro.org wrote:
On 28/03/12 03:26, Michael Hope wrote:
Hi there. The GCC build time is approaching 24 hours and our five Panda boards can't keep up. Sounds like a good time to LAVAise the toolchain build process a bit more...
As you know, I've been doing some experiments with this over the last few months. I was blocked by a LAVA bug for a while, but that's been fixed now.
Here's the latest test run (done by Le Chi Thu):
http://validation.linaro.org/lava-server/dashboard/permalink/bundle/d73af579...
The test fails due to a GCC build failure:
//usr/include/linux/errno.h:4:23: fatal error: asm/errno.h: No such file or directory
This surprised me, because I thought it was using the same rootfs you had on the ursa boards, but I've not had time to do anything about it yet.
Looks like something that can likely be resolved by adding a dependency for the test. However, if you need, or if it would be more convenient to have a custom rootfs for this, that's certainly an option. Nothing says we necessarily have to run these tests on nano, developer, etc... but if it's interesting to make it possible for later running this as part of a platform release test, it might be better to make them generic so that they don't depend on a custom rootfs.
There's a switch coming up due to Precise and hard float and I'll normalise against the Linaro LEB and hwpacks then. developer is a good start but it'll need extra packages added. I'll script those up and spin a new image with them pre-installed rather than add speed and reliability issue of doing it at boot time.
-- Michael
-
Andrew Stubbs -
Dave Pigott -
Michael Hope -
Paul Larson