Re: next boot: 267 boots: 5 failed, 261 passed with 1 offline (next-20150408)

Hi Maxime,

On 12 April 2015 at 23:47, Maxime Ripard maxime.ripard@free-electrons.com wrote:

Hi,

On Wed, Apr 08, 2015 at 12:13:08PM -0700, Tyler Baker wrote:

...

On 8 April 2015 at 09:05, kernelci.org bot bot@kernelci.org wrote:

...

next boot: 267 boots: 5 failed, 261 passed with 1 offline (next-20150408)

Full Boot Summary: http://kernelci.org/boot/all/job/next/kernel/next-20150408/ Full Build Summary: http://kernelci.org/build/next/kernel/next-20150408/

Tree: next Branch: local/master Git Describe: next-20150408 Git Commit: a897436e0e233e84b664bb7f33c4e0d4d3e3bdad Git URL: git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git Tested: 68 unique boards, 20 SoC families, 25 builds out of 123

Boot Failures Detected: http://kernelci.org/boot/?next-20150408&fail

arm:

imx_v6_v7_defconfig:
    imx6q-sabrelite: 1 failed lab

multi_v7_defconfig+CONFIG_PROVE_LOCKING=y:
    sun7i-a20-bananapi: 1 failed lab
    sun7i-a20-cubieboard2: 1 failed lab
    sun7i-a20-cubietruck: 2 failed labs

Kevin and I bisected this failure with some interesting results.

There are two ways we have found to get these platforms booting again.

1. git checkout next && git revert a897436e0e233e84b664bb7f33c4e0d4d3e3bdad

Add linux-next specific files for 20150408

2. git checkout next && rm -f localversion-next

We thought this might be a kernel load address issue, so we appended the dtb and removed the ramdisk + modules from the equation, same boot failure. Using hexdiff to diff the two zImages (good and bad) yield quite a bit of changes. We also tested building the test kernels with gcc 4.7.3 and 4.9 same result.

Any ideas on what might be the issue here?

I gave this a shot this weekend, and unfortunately, I can't build a kernel that points out this issue.

Given that this seem to be very sensitive to various string length, I wonder whether the fact that I'm possibly using a different compiler, and building at a different path can "cover" it.

Are you using it using a VM/container? If so, if it's an option, I can always run that VM here and give it a try.

Sorry for not posting this earlier, we have been tracking our findings here if you would like to have a look.

https://github.com/kernelci/kernel-bugs/issues/15

Thanks, Maxime

-- Maxime Ripard, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com