The mainline kernel v6.10-rc6 boot failed across all arm64 devices and qemu-arm64 and no crash log found (earlycon) did not help.
But the defconfig builds boot PASS.
The boot problem is with defconfig + Extra Kconfigs and builds links provided in the bottom of this email.
The boot test history shows that, GOOD: v6.10-rc5 BAD: v6.10-rc6
We are bisecting this issue and get back to you with git bisect results.
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
Following is the list of build combinations that boot failed. Regressions found on qemu-arm64, Juno-r2, db410c, db845c, e850-96 and Rock-pi-4. - boot/gcc-13-lkftconfig-64k_page_size - boot/gcc-13-lkftconfig-kunit - boot/gcc-13-lkftconfig-rcutorture - boot/gcc-13-lkftconfig-16k_page_size - boot/gcc-13-lkftconfig-libgpiod - boot/gcc-13-lkftconfig-devicetree - boot/gcc-13-lkftconfig-debug-kmemleak - boot/gcc-13-lkftconfig-debug - boot/gcc-13-lkftconfig
Steps to reproduce boot failure on qemu-arm64: ---------- - https://tuxapi.tuxsuite.com/v1/groups/linaro/projects/lkft/tests/2idWuAb51mc...
metadata: ---- git_describe: v6.10-rc6 git_repo: https://gitlab.com/Linaro/lkft/mirrors/torvalds/linux-mainline git_sha: 22a40d14b572deb80c0648557f4bd502d7e83826 git_short_log: 22a40d14b572 ("Linux 6.10-rc6") arch: arm64
Links: ----- - https://storage.tuxsuite.com/public/linaro/lkft/builds/2idWrbdtwMqpD9wx7IPRG... - https://storage.tuxsuite.com/public/linaro/lkft/builds/2idWrYDposdlRsS4jwF91... - https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.10-rc6/tes... - https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.10-rc6/tes...
-- Linaro LKFT https://lkft.linaro.org
On Mon, Jul 01, 2024 at 05:28:29PM +0530, Naresh Kamboju wrote:
The mainline kernel v6.10-rc6 boot failed across all arm64 devices and qemu-arm64 and no crash log found (earlycon) did not help.
But the defconfig builds boot PASS.
The boot problem is with defconfig + Extra Kconfigs and builds links provided in the bottom of this email.
The boot test history shows that, GOOD: v6.10-rc5 BAD: v6.10-rc6
Other CI systems aren't showing this. I'm guessing you've updated the toolchain versions you're using and are running into:
https://sourceware.org/bugzilla/show_bug.cgi?id=31924#c9
and will see this with other kernels. It's probably worth testing toolchains before deploying them...
On Mon, Jul 01, 2024 at 01:12:07PM +0100, Mark Brown wrote:
On Mon, Jul 01, 2024 at 05:28:29PM +0530, Naresh Kamboju wrote:
The mainline kernel v6.10-rc6 boot failed across all arm64 devices and qemu-arm64 and no crash log found (earlycon) did not help.
But the defconfig builds boot PASS.
The boot problem is with defconfig + Extra Kconfigs and builds links provided in the bottom of this email.
The boot test history shows that, GOOD: v6.10-rc5 BAD: v6.10-rc6
Other CI systems aren't showing this. I'm guessing you've updated the toolchain versions you're using and are running into:
https://sourceware.org/bugzilla/show_bug.cgi?id=31924#c9
and will see this with other kernels. It's probably worth testing toolchains before deploying them...
Aha, that would explain why I'm failing to reproduce this with clang or my old version of GCC. I only see the problem with the binary provided by the CI.
Will
On Mon, 1 Jul 2024 at 17:42, Mark Brown broonie@kernel.org wrote:
On Mon, Jul 01, 2024 at 05:28:29PM +0530, Naresh Kamboju wrote:
The mainline kernel v6.10-rc6 boot failed across all arm64 devices and qemu-arm64 and no crash log found (earlycon) did not help.
But the defconfig builds boot PASS.
The boot problem is with defconfig + Extra Kconfigs and builds links provided in the bottom of this email.
The boot test history shows that, GOOD: v6.10-rc5 BAD: v6.10-rc6
Other CI systems aren't showing this. I'm guessing you've updated the toolchain versions you're using and are running into:
Our LKFT tests saw the same bug as #31924.
BAD: "ar": "GNU ar (GNU Binutils for Debian) 2.42.50.20240625", "as": "GNU assembler (GNU Binutils for Debian) 2.42.50.20240625", "ld": "GNU ld (GNU Binutils for Debian) 2.42.50.20240625", "gcc": "gcc (Debian 13.3.0-1) 13.3.0",
Good: "ar": "GNU ar (GNU Binutils for Debian) 2.42", "as": "GNU assembler (GNU Binutils for Debian) 2.42", "ld": "GNU ld (GNU Binutils for Debian) 2.42", "gcc": "gcc (Debian 13.2.0-25) 13.2.0",
We are able to reproduce this bug ^. The gcc-13 builds failed and gcc-8 builds and clang-18 builds PASS.
and will see this with other kernels. It's probably worth testing toolchains before deploying them...
While we talk, I get to see today's Linux next and also encounter the same problem. Thanks for the suggestion to test toolchains.
- Naresh
Hi Naresh,
This report is quite hard to follow, so just some minor comments which you can hopefully take on board to improve things in future:
On Mon, Jul 01, 2024 at 05:28:29PM +0530, Naresh Kamboju wrote:
The mainline kernel v6.10-rc6 boot failed across all arm64 devices and qemu-arm64 and no crash log found (earlycon) did not help.
But the defconfig builds boot PASS.
The boot problem is with defconfig + Extra Kconfigs and builds links provided in the bottom of this email.
If I go to the bottom of the email, I see this:
Links:
- https://storage.tuxsuite.com/public/linaro/lkft/builds/2idWrbdtwMqpD9wx7IPRG...
- https://storage.tuxsuite.com/public/linaro/lkft/builds/2idWrYDposdlRsS4jwF91...
- https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.10-rc6/tes...
- https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.10-rc6/tes...
but these are just random links without any context. The first two appear to be kernel build artifacts, but I can't tell:
- Are both "good" or "bad"? - What is the difference between them?
If I look at the first of the last two links, click through to the "qemu-arm64" target and follow the "job_url" link, then I end up at:
https://tuxapi.tuxsuite.com/v1/groups/linaro/projects/lkft/tests/2idWuAb51mc...
which has some empty logs and something from "TuxRun" which says:
{"error": "File not found"}
It's very hard to know what to do with this :/
Will
Hi Will,
On Mon, 1 Jul 2024 at 18:17, Will Deacon will@kernel.org wrote:
Hi Naresh,
This report is quite hard to follow, so just some minor comments which you can hopefully take on board to improve things in future:
On Mon, Jul 01, 2024 at 05:28:29PM +0530, Naresh Kamboju wrote:
The mainline kernel v6.10-rc6 boot failed across all arm64 devices and qemu-arm64 and no crash log found (earlycon) did not help.
But the defconfig builds boot PASS.
The boot problem is with defconfig + Extra Kconfigs and builds links provided in the bottom of this email.
If I go to the bottom of the email, I see this:
Links:
- https://storage.tuxsuite.com/public/linaro/lkft/builds/2idWrbdtwMqpD9wx7IPRG...
- https://storage.tuxsuite.com/public/linaro/lkft/builds/2idWrYDposdlRsS4jwF91...
- https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.10-rc6/tes...
- https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.10-rc6/tes...
but these are just random links without any context. The first two appear to be kernel build artifacts, but I can't tell:
- Are both "good" or "bad"?
Both are bad. which failed to boot from these build artifacts. Which is the location of storage of Build artifacts and build logs, Image, Vmlinux.
- What is the difference between them?
one build is pointing to Rcu Torture test build, - CONFIG_RCU_TORTURE_TEST=m ++Kconfigs
other one is pointing to libgpiod build, - CONFIG_GPIOLIB=y ++Kconfigs
If I look at the first of the last two links, click through to the "qemu-arm64" target and follow the "job_url" link, then I end up at:
https://tuxapi.tuxsuite.com/v1/groups/linaro/projects/lkft/tests/2idWuAb51mc...
which has some empty logs and something from "TuxRun" which says:
{"error": "File not found"}
Due to missing boot log this file is empty.
It's very hard to know what to do with this :/
Sorry for the trouble. I will add more detailed information next to each link in my next report.
Will
- Naresh
-
Mark Brown -
Naresh Kamboju -
Will Deacon