On Thu, 3 Apr 2025 at 06:48, Nathan Chancellor nathan@kernel.org wrote:

Hi Naresh,

On Wed, Apr 02, 2025 at 06:19:19PM +0530, Naresh Kamboju wrote:

...

Regressions on rk3399-rock-pi-4b, dragonboard-410c and dragonboard-845c the lto-thing, hardening and lto-full config boot failed with toolchain clang-nightly on the mainline master branch with no console output.

First seen on the v6.14-12245-g91e5bfe317d8 Good: v6.14-11270-g08733088b566 Bad: v6.14-12245-g91e5bfe317d8

Regressions found on rk3399-rock-pi-4b:

• boot/clang-nightly-lkftconfig-kselftest
• boot/clang-nightly-lkftconfig-lto-thing
• boot/clang-nightly-lkftconfig-hardening
• boot/clang-nightly-lkftconfig-lto-full

Regressions found on dragonboard-410c:

• boot/clang-nightly-lkftconfig-lto-thing
• boot/clang-nightly-lkftconfig-lto-full
• boot/clang-nightly-lkftconfig-hardening

Regressions found on dragonboard-845c:

• boot/clang-nightly-lkftconfig-hardening
• boot/clang-nightly-lkftconfig-lto-thing

Regression Analysis:

• New regression? Yes
• Reproducibility? Yes

Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output

Reported-by: Linux Kernel Functional Testing lkft@linaro.org

## Boot log Starting kernel ...

<No console output>

## Source

• Kernel version: 6.14.0
• Git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
• Git sha: 91e5bfe317d8f8471fbaa3e70cf66cae1314a516
• Git describe: v6.14-12245-g91e5bfe317d8
• Project details:

https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-12245-g...

• Architectures: arm64 (rk3399-rock-pi-4b, dragonboard-410c, dragonboard-845c)
• Toolchains: clang-nightly (Debian clang version 21.0.0 )

The version string for the toolchain seems to be slightly truncated but from the configuration it is:

Debian clang version 21.0.0 (++20250330105456+3b3d1a5c2614-1~exp1~20250330225508.1357)

which can be parsed with our parse-debian-clang.py [1] to get more information:

$ scripts/parse-debian-clang.py -p -v 'Debian clang version 21.0.0 (++20250330105456+3b3d1a5c2614-1~exp1~20250330225508.1357)' clang checkout date: 2025-03-30 10:54 UTC (3 days, 14:17:24.134272 ago) clang revision: 3b3d1a5c2614 clang revision link: https://github.com/llvm/llvm-project/commit/3b3d1a5c2614

Our CI is using a slightly newer version:

$ scripts/parse-debian-clang.py -p -v 'Debian clang version 21.0.0 (++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)' clang checkout date: 2025-04-01 11:25 UTC (1 day, 13:49:39.394836 ago) clang revision: 290d7b82cb5d clang revision link: https://github.com/llvm/llvm-project/commit/290d7b82cb5d

Can you see if it is reproducible with that revision?

I’ve re-run the tests to validate the boot behavior on the Rock Pi 4 board with different Clang nightly versions and the latest mainline kernel.

The combination using clang-nightly:20250319 successfully booted the Rock Pi 4 board. However, the combination using clang-nightly:20250401 failed to boot the same board.

"name": "clang", "version": "21.0.0", "version_full": "Debian clang version 21.0.0 (++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)"

Reference: - https://lkft.validation.linaro.org/scheduler/job/8196258 - https://lkft.validation.linaro.org/scheduler/job/8196275

- Naresh

For what it's worth, both of the arm64 boxes I have can boot a ThinLTO kernel compiled with a version of LLVM @ 749535ba2808e133682074f712ac6829335f8875, so it could be something that was broken for a little bit but Debian happened to sync before the fix was committed.

Cheers, Nathan

On Sat, 5 Apr 2025 at 05:58, Nathan Chancellor nathan@kernel.org wrote:

On Thu, Apr 03, 2025 at 04:17:44PM +0530, Naresh Kamboju wrote:

...

I’ve re-run the tests to validate the boot behavior on the Rock Pi 4 board with different Clang nightly versions and the latest mainline kernel.

The combination using clang-nightly:20250319 successfully booted the Rock Pi 4 board. However, the combination using clang-nightly:20250401 failed to boot the same board.

"name": "clang", "version": "21.0.0", "version_full": "Debian clang version 21.0.0 (++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)"

Reference:

• https://lkft.validation.linaro.org/scheduler/job/8196258
• https://lkft.validation.linaro.org/scheduler/job/8196275

Thank you for verifying. Do you test with clang-20 and if so, are tests at that same revision passing? I want to make sure this is an LLVM regression before trying to get into figuring out how to bisect with your setup. Another way to do that is to see if clang-nightly boots are failing on the stable branches, which would point to Linux not being at fault.

clang-nightly boot failed with stable-rc 6.12, 6.13 and 6.14 and mainline and next on following devices and virtual platforms. - dragonboard-410c - dragonboard-845c - rk3399-rock-pi-4b - Juno-r2 - e850-96 - FVP (Fast Models Fixed Virtual Platforms)

Boot Test links, - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12.... - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12.... - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12....

Lore link, - https://lore.kernel.org/all/CA+G9fYve7+nXJNoV48TksXoMeVjgJuP8Gs=+1br+Qur1DPW...

Cheers, Nathan