Paul,
you suggest to use different configurations for doing the "build the kernel" validation and the benchmarking? Why wouldn't we run in the same heat issues while benchmarking?
Can we somehow put the SoC into a very low power state for 2 minutes for cool down and simply wait before starting the build/test?
Anyway, I assume that the strict environment requirement where toolchain WG needs to sign off on the setup would mostly apply to benchmarking only and that we could probably choose any stable image for doing the build validation of the toolchain that is rock solid. Matt?
If so, it sounds sensible to just pick a recent release with thermal enabled for the build job and use the special configuration for the benchmarking parts - maybe with a cooling step as above.
On Mon, Apr 8, 2013 at 7:56 PM, Paul Sokolovsky paul.sokolovsky@linaro.orgwrote:
Hello,
On Sat, 6 Apr 2013 15:40:15 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
[]
Ok, so here're these 2 builds:
gcc-4.8~svn196132
panda-es02 https://validation.linaro.org/lava-server/scheduler/job/50993
panda-es05 https://validation.linaro.org/lava-server/scheduler/job/50994
#50993 went well.
#50994 midway in compilation started to get invalid data (grep for "is not valid in preprocessor expressions"), then got kernel fault, then got caught in reboot loop, apparently due to:
[ 6.631256] thermal_init_thermal_state: Getting initial temp for cpu domain [ 6.638702] thermal_request_temp [ 6.642150] omap_fatal_zone:FATAL ZONE (hot spot temp: 128490)
- all these behaviors were seen by me before (actually, previously, I
didn't see such explicit messages from kernel that it's a thermal faults).
So, CBuild/LAVA can do (successful) builds, but some builds fail due to thermal issues. Actually, let me load up all boards with the same build now, towards assessing thermal failure rate more scientifically.
Well, let's count:
https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild... OK
https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild... Failed, thermal (see above)
https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild... "../../../gcc-4.8~svn196132/gcc/gengtype.c:4106:39: error: cannot convert 'flisT*' to 'flist*' in assignment" flipped bit in file (so that before, including on local Panda), then other random failures.
https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild... "rsync: getaddrinfo: toolchain64 2000: Temporary failure in name resolution" - network flip
https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild... OK
https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild... Lot of "malloc: ../bash/jobs.c:743: assertion botched"
https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild... Reboot during configure, then reboot cycle.
So, out of 7 builds, only 2 were successful, the rest fail due to thermal issues (apparently, all but the one with network issues). We can try to consider why LAVA-based builds have such low yield rate comparing to native Cbuild builds (one explanation is that LAVA does pretty heavy lifting to install OS, etc., so when build starts, CPU is already pretty hot), but it's clear that using kernel with voltage/frequency scaling disabled simply doesn't work that well for *builds* (vs benchmarks).
So, we can consider preparing OS image which uses the same basic OS, but normal kernel, and use that for builds. Would TCWG be able to prepare such image? If not, we can add it to the list of tasks for (now combined) LAVA/Infra team, but with all the other tasks we have, it may take some time to get to.
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog