Hello,

On Sat, 6 Apr 2013 15:40:15 +0300
Paul Sokolovsky <Paul.Sokolovsky@linaro.org> wrote:

[]

> Ok, so here're these 2 builds:
>
> gcc-4.8~svn196132
>
> panda-es02
> https://validation.linaro.org/lava-server/scheduler/job/50993
>
> panda-es05
> https://validation.linaro.org/lava-server/scheduler/job/50994
>
> #50993 went well.
>
> #50994 midway in compilation started to get invalid data (grep for
> "is not valid in preprocessor expressions"), then got kernel fault,
> then got caught in reboot loop, apparently due to:
>
> [    6.631256] thermal_init_thermal_state: Getting initial temp for
> cpu domain [    6.638702] thermal_request_temp
> [    6.642150] omap_fatal_zone:FATAL ZONE (hot spot temp: 128490)
>
> - all these behaviors were seen by me before (actually, previously, I
> didn't see such explicit messages from kernel that it's a thermal
> faults).
>
> So, CBuild/LAVA can do (successful) builds, but some builds fail due
> to thermal issues. Actually, let me load up all boards with the same
> build now, towards assessing thermal failure rate more scientifically.

Well, let's count:

https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild/bundles/9dea0c78604ce5e65178ec8d71edac8d8a499c4e/
OK

https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild/bundles/cf346215160e778be13e031378b36328eeb315c3/
Failed, thermal (see above)

https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild/bundles/bce0a440e5cece819678e6fd7f276a43c2a44ecd/
"../../../gcc-4.8~svn196132/gcc/gengtype.c:4106:39: error: cannot
convert 'flisT*' to 'flist*' in assignment" flipped bit in file (so
that before, including on local Panda), then other random failures.

https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild/bundles/6c31d8e21dbc716255c7ec13702b5252434f9ced/
"rsync: getaddrinfo: toolchain64 2000: Temporary failure in name
resolution" - network flip

https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild/bundles/de94c85961e59dbaba41f8dec6c20f4740384241/
OK

https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild/bundles/c032912d494e80857b931ab8c2dafc4e4f83c762/
Lot of "malloc: ../bash/jobs.c:743: assertion botched"

https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild/bundles/d36badcd7310aa0db4a1efcd1554d11da06ef6bc/
Reboot during configure, then reboot cycle.

So, out of 7 builds, only 2 were successful, the rest fail due to
thermal issues (apparently, all but the one with network issues). We
can try to consider why LAVA-based builds have such low yield rate
comparing to native Cbuild builds (one explanation is that LAVA does
pretty heavy lifting to install OS, etc., so when build starts, CPU is
already pretty hot), but it's clear that using kernel with
voltage/frequency scaling disabled simply doesn't work that well for
*builds* (vs benchmarks).

So, we can consider preparing OS image which uses the same basic OS,
but normal kernel, and use that for builds. Would TCWG be able to
prepare such image? If not, we can add it to the list of tasks for (now
combined) LAVA/Infra team, but with all the other tasks we have, it
may take some time to get to.

--
Best Regards,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog