On 7 March 2012 23:47, Ken Werner ken.werner@linaro.org wrote:
On 03/06/2012 01:26 AM, Michael Hope wrote:
Hi Ken. In follow up to our 1-on-1 yesterday, here's what I'd like done next.
The goal is to use OE Core as a release test suite. The releases are tarballs so we can keep the current recipe format and punt bzr support for later. The first step is to be able to reliably build a release in the cloud or validation lab.
In all cases keep the other teams in mind. Much of this is related to Validation. Platform will be involved later. Ping them early.
Kernel: We're starting with GCC and need a kernel to supply headers and to boot some type of ARMv7 image. I don't want a linux-linaro recipe as people will use it and it's too early for that.
Find a kernel, preferably from OE Core, that is recent, ARMv7,>= 512 MB RAM, and works well with qemu-linaro. Prefer vexpress-a9, else OMAP?
Done. I've updated the meta-linaro layer to (re-)use the default OE-Core kernel sources - a yocto kernel - that gets built using a vexpress defconfig.
Talking: Say Hi to Validation re: EC2 and plans Say Hi to the ARM landing team re: vexpress upstream support Say Hi to Beth Flanagan re: Yocto's existing auto builders and any hints
Yep, it looks like there are several options: a) the Yocto autobuilder b) Jenkins c) LAVA d) something homegrown
I talked to fabo and zyga and agreed that - as a starting point - I'll provide a shell script that automates the process and we'll go from there. I've made contact with Elizabeth and briefly looked into the yocto autobuilder whis is based on buildbot. Obviously it's meant for regression testing the various yocto images. I think we want to keep oe-core+linaro-meta stable and exchange the toolchain which is kind of the other way round what they are doing. But I'll keep an eye on it.
(a) buildbot and (b) jenkins are both continuous integration tools. I guess that the main difference is roughly the infrastructure used (Yocto or Linaro).
LAVA is a testing framework. It doesn't build anything, it runs tests and generate reports/results. It can be extended (like all the tools mentioned) to meet your requirements but it has a cost (resources/time).
Cloud builds: Find out who is already doing OE builds in the cloud and how Run a build locally and time Push ~/downloads into the cloud, build, and time[1] Figure how much this build will cost in dollars
[1] c1.xlarge might be best. Builds are normally I/O bound and the cloud is I/O poor. Put /tmp and other chunks in a tmpfs? EC2 rounds up to the nearest hour as well.
If the cloud is too expensive then we'll get a machine installed.
S3 for storage: (only proceed if affordable) Use S3 for storing the input tarballs Use S3 either as a pre-mirror by serving over HTTP, or use s3cmd to sync down the tarballs before starting the build
Just to give you an overview. The build of the sato and qt4e images takes about two hours my machine (24x "E5649 @ 2.53GHz" with 32gb of RAM) and creates about 37GiB of object files, binaries, packages and images. This is excluding the size of the sources (and the time for fetching them in case they are not there). I haven't spent time on optimizing my build environment (tmpfs etc), so I guess there is room for improvement. I cannot even say whether it's I/O or CPU bound. Sometimes it appears to be the latter - when building Qt for example. But sometimes only one CPU gets used and it's waiting for a source being unpacked that other tasks depend on.
Scripting: Re-use existing scripts if feasible. Integrate with LAVA providing we can run exactly the same scripts on a laptop for debugging.
Script the bitbake, OE meta layer, and Linaro meta layer setup. Script the configuration including setting the release tarball URL and GCC preferred version. Script the build and result capture, especially the log, any ICEs, and the final sizes
Future: OE can grab a repository seed then update based on that. Check if the bzr backend supports this. If so, play with seeding to do tip builds.
Yeah, this is definitely useful for testing tip. I'd like to start with something smaller than the GCC though. For example the Linaro GDB or QEMU would be good. I hope these are the low hanging fruits, so I'll begin with them if I can squeeze some time in.
Let me know what you think then we'll spawn blueprints. Let's keep an eye on this as it's sounding expensive.
-- Michael
Regards, Ken
linaro-toolchain mailing list linaro-toolchain@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-toolchain