first of all, we are building dylan branch with oe-core + meta-linaro-toolchain, not using external toolchain, hence the cross post on OE and linaro lists.

we've been debugging some nasty build issues over the last few days on our Jenkins server. and i'd like to bring our findings up to the list... i suspect there could be a bug in OE gcc-cross, but i'd like to hear from you about that.

our bug appeared because we were 'wrongly' unsetting the CFLAGS from OE in one of our recipe, so we would loose the following from our compilation command line

The compilation error was that stdlib.h was not found. 

as a matter of fact, we build for several different machines (.conf file), *but* we share the same sstate and downloads for all machines. We don't do parallel builds, each machine is built sequentially in its own build folder (.../workspace/machines/<MACHINE>/build), and all build folder are deleted before each build (of course sstate and downloads aren't deleted).

What we noticed is that we end up pulling the compiler (gcc-cross) from sstate as expected. However we pull the same 'blob' from sstate for gcc-cross for *all* machines we build. It does seem that the compiler does not use $MACHINE for the sstate checksum. 

When the gcc-cross package was built and pushed in sstate, it was being built for a specific machine (let's say MACHINE-A), hence the compiler has the 'builtin' sysroot set to 'tmp/sysroots/MACHINE-A'. 

Now when building our image for MACHINE-B, we pull gcc-cross from sstate, and we get the default builtin sysroot for MACHINE-A!! That's why our build failed, because we tried to build MACHINE-B before MACHINE-A, so stdlib.h wasn't there yet on Jenkins..

To me the problem is that gcc-cross 'embedds' some $MACHINE data in its package, but it is not marked as 'machine specific, but arch specific.  So several machines will end up sharing the same gcc-cross package.

So, of course we shouldn't ignore the CFLAGS from OE, and the CFLAGS would set the right sysroot, and of course we fixed our software so that we don't ignore CFLAGS anymore... but that still looks like a bug to me.

cheers

nico