clang-nightly: vdso/compat_gettimeofday.h:152:15: error: instruction variant requires ARMv6 or later
Following build errors noticed on Linux next-20231204 tag with clang-nightly for arm and arm64.
## Test Regressions (compared to next-20231201) * arm64, build - clang-nightly-defconfig - clang-nightly-defconfig-40bc7ee5 - clang-nightly-lkftconfig - clang-nightly-lkftconfig-kselftest
* arm, build - clang-nightly-allnoconfig - clang-nightly-axm55xx_defconfig - clang-nightly-bcm2835_defconfig - clang-nightly-clps711x_defconfig - clang-nightly-defconfig - clang-nightly-exynos_defconfig - clang-nightly-imx_v6_v7_defconfig - clang-nightly-keystone_defconfig - clang-nightly-lkftconfig - clang-nightly-lkftconfig-kselftest - clang-nightly-omap2plus_defconfig - clang-nightly-pxa910_defconfig - clang-nightly-s3c6400_defconfig - clang-nightly-s5pv210_defconfig - clang-nightly-sama5_defconfig - clang-nightly-shmobile_defconfig - clang-nightly-tinyconfig - clang-nightly-u8500_defconfig - clang-nightly-vexpress_defconfig
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
Build log on arm64: --------- In file included from lib/vdso/gettimeofday.c:5: In file included from include/vdso/datapage.h:135: arch/arm64/include/asm/vdso/compat_gettimeofday.h:152:15: error: instruction variant requires ARMv6 or later 152 | asm volatile("mov %0, %1" : "=r"(ret) : "r"(_vdso_data)); | ^ <inline asm>:1:2: note: instantiated into assembly here 1 | mov r4, r1 | ^ In file included from <built-in>:3: lib/vdso/gettimeofday.c:139:3: error: invalid instruction 139 | smp_rmb(); | ^
Build log on arm: --------- In file included from arch/arm/vfp/vfpmodule.c:23: arch/arm/include/asm/cp15.h:101:2: error: instruction requires: data-barriers 101 | isb(); | ^
Links: arm64 - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20231204/tes...
arm - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20231204/tes... - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20231204/tes...
-- Linaro LKFT https://lkft.linaro.org
Hi Naresh,
On Mon, Dec 04, 2023 at 05:33:26PM +0530, Naresh Kamboju wrote:
Following build errors noticed on Linux next-20231204 tag with clang-nightly for arm and arm64.
## Test Regressions (compared to next-20231201)
arm64, build
- clang-nightly-defconfig
- clang-nightly-defconfig-40bc7ee5
- clang-nightly-lkftconfig
- clang-nightly-lkftconfig-kselftest
arm, build
- clang-nightly-allnoconfig
- clang-nightly-axm55xx_defconfig
- clang-nightly-bcm2835_defconfig
- clang-nightly-clps711x_defconfig
- clang-nightly-defconfig
- clang-nightly-exynos_defconfig
- clang-nightly-imx_v6_v7_defconfig
- clang-nightly-keystone_defconfig
- clang-nightly-lkftconfig
- clang-nightly-lkftconfig-kselftest
- clang-nightly-omap2plus_defconfig
- clang-nightly-pxa910_defconfig
- clang-nightly-s3c6400_defconfig
- clang-nightly-s5pv210_defconfig
- clang-nightly-sama5_defconfig
- clang-nightly-shmobile_defconfig
- clang-nightly-tinyconfig
- clang-nightly-u8500_defconfig
- clang-nightly-vexpress_defconfig
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
Build log on arm64:
In file included from lib/vdso/gettimeofday.c:5: In file included from include/vdso/datapage.h:135: arch/arm64/include/asm/vdso/compat_gettimeofday.h:152:15: error: instruction variant requires ARMv6 or later 152 | asm volatile("mov %0, %1" : "=r"(ret) : "r"(_vdso_data)); | ^ <inline asm>:1:2: note: instantiated into assembly here 1 | mov r4, r1 | ^ In file included from <built-in>:3: lib/vdso/gettimeofday.c:139:3: error: invalid instruction 139 | smp_rmb(); | ^
Build log on arm:
In file included from arch/arm/vfp/vfpmodule.c:23: arch/arm/include/asm/cp15.h:101:2: error: instruction requires: data-barriers 101 | isb(); | ^
This is caused by a change to Debian's LLVM that changes the internal defaults of the arm-linux-gnueabi and arm-linux-gnueabihf tuples:
https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/907baf024b9a5...
We use arm-linux-gnueabi for the kernel (see scripts/Makefile.clang) so now we have a hardcoded armv5te CPU, even if we are building for armv7 or such.
I am still investigating into what (if anything) can be done to resolve this on the kernel side. We could potentially revert commit ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target triple") but I am not sure that will save us from that change, as tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an armv7 CPU even though we may not be building for armv7.
Cheers, Nathan
On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
Hi Naresh,
On Mon, Dec 04, 2023 at 05:33:26PM +0530, Naresh Kamboju wrote:
Following build errors noticed on Linux next-20231204 tag with clang-nightly for arm and arm64.
## Test Regressions (compared to next-20231201)
arm64, build
- clang-nightly-defconfig
- clang-nightly-defconfig-40bc7ee5
- clang-nightly-lkftconfig
- clang-nightly-lkftconfig-kselftest
arm, build
- clang-nightly-allnoconfig
- clang-nightly-axm55xx_defconfig
- clang-nightly-bcm2835_defconfig
- clang-nightly-clps711x_defconfig
- clang-nightly-defconfig
- clang-nightly-exynos_defconfig
- clang-nightly-imx_v6_v7_defconfig
- clang-nightly-keystone_defconfig
- clang-nightly-lkftconfig
- clang-nightly-lkftconfig-kselftest
- clang-nightly-omap2plus_defconfig
- clang-nightly-pxa910_defconfig
- clang-nightly-s3c6400_defconfig
- clang-nightly-s5pv210_defconfig
- clang-nightly-sama5_defconfig
- clang-nightly-shmobile_defconfig
- clang-nightly-tinyconfig
- clang-nightly-u8500_defconfig
- clang-nightly-vexpress_defconfig
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
Build log on arm64:
In file included from lib/vdso/gettimeofday.c:5: In file included from include/vdso/datapage.h:135: arch/arm64/include/asm/vdso/compat_gettimeofday.h:152:15: error: instruction variant requires ARMv6 or later 152 | asm volatile("mov %0, %1" : "=r"(ret) : "r"(_vdso_data)); | ^ <inline asm>:1:2: note: instantiated into assembly here 1 | mov r4, r1 | ^
I have to wonder why Clang is complaining about "mov r4, r1" because that certainly should not require "ARMv6 or later". On the face of it, this to me looks like a bug in Clang.
On Mon, Dec 04, 2023 at 06:26:15PM +0000, Russell King (Oracle) wrote:
On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
Hi Naresh,
On Mon, Dec 04, 2023 at 05:33:26PM +0530, Naresh Kamboju wrote:
Following build errors noticed on Linux next-20231204 tag with clang-nightly for arm and arm64.
## Test Regressions (compared to next-20231201)
arm64, build
- clang-nightly-defconfig
- clang-nightly-defconfig-40bc7ee5
- clang-nightly-lkftconfig
- clang-nightly-lkftconfig-kselftest
arm, build
- clang-nightly-allnoconfig
- clang-nightly-axm55xx_defconfig
- clang-nightly-bcm2835_defconfig
- clang-nightly-clps711x_defconfig
- clang-nightly-defconfig
- clang-nightly-exynos_defconfig
- clang-nightly-imx_v6_v7_defconfig
- clang-nightly-keystone_defconfig
- clang-nightly-lkftconfig
- clang-nightly-lkftconfig-kselftest
- clang-nightly-omap2plus_defconfig
- clang-nightly-pxa910_defconfig
- clang-nightly-s3c6400_defconfig
- clang-nightly-s5pv210_defconfig
- clang-nightly-sama5_defconfig
- clang-nightly-shmobile_defconfig
- clang-nightly-tinyconfig
- clang-nightly-u8500_defconfig
- clang-nightly-vexpress_defconfig
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
Build log on arm64:
In file included from lib/vdso/gettimeofday.c:5: In file included from include/vdso/datapage.h:135: arch/arm64/include/asm/vdso/compat_gettimeofday.h:152:15: error: instruction variant requires ARMv6 or later 152 | asm volatile("mov %0, %1" : "=r"(ret) : "r"(_vdso_data)); | ^ <inline asm>:1:2: note: instantiated into assembly here 1 | mov r4, r1 | ^
I have to wonder why Clang is complaining about "mov r4, r1" because that certainly should not require "ARMv6 or later". On the face of it, this to me looks like a bug in Clang.
This is because the compat vDSO is compiled with '-mthumb' by default and Nick helpfully pointed out on IRC that prior to ARMv6, one of the operands to the mov had to be a HI register, which I do see in the ARM:
Encoding T1 ARMv6*, ARMv7 if <Rd> and <Rm> both from R0-R7 ARMv4T, ARMv5T*, ARMv6*, ARMv7 otherwise MOV<c> <Rd>, <Rm>
Cheers, Nathan
On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
Hi Naresh,
On Mon, Dec 04, 2023 at 05:33:26PM +0530, Naresh Kamboju wrote:
Following build errors noticed on Linux next-20231204 tag with clang-nightly for arm and arm64.
## Test Regressions (compared to next-20231201)
arm64, build
- clang-nightly-defconfig
- clang-nightly-defconfig-40bc7ee5
- clang-nightly-lkftconfig
- clang-nightly-lkftconfig-kselftest
arm, build
- clang-nightly-allnoconfig
- clang-nightly-axm55xx_defconfig
- clang-nightly-bcm2835_defconfig
- clang-nightly-clps711x_defconfig
- clang-nightly-defconfig
- clang-nightly-exynos_defconfig
- clang-nightly-imx_v6_v7_defconfig
- clang-nightly-keystone_defconfig
- clang-nightly-lkftconfig
- clang-nightly-lkftconfig-kselftest
- clang-nightly-omap2plus_defconfig
- clang-nightly-pxa910_defconfig
- clang-nightly-s3c6400_defconfig
- clang-nightly-s5pv210_defconfig
- clang-nightly-sama5_defconfig
- clang-nightly-shmobile_defconfig
- clang-nightly-tinyconfig
- clang-nightly-u8500_defconfig
- clang-nightly-vexpress_defconfig
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
Build log on arm64:
In file included from lib/vdso/gettimeofday.c:5: In file included from include/vdso/datapage.h:135: arch/arm64/include/asm/vdso/compat_gettimeofday.h:152:15: error: instruction variant requires ARMv6 or later 152 | asm volatile("mov %0, %1" : "=r"(ret) : "r"(_vdso_data)); | ^ <inline asm>:1:2: note: instantiated into assembly here 1 | mov r4, r1 | ^ In file included from <built-in>:3: lib/vdso/gettimeofday.c:139:3: error: invalid instruction 139 | smp_rmb(); | ^
Build log on arm:
In file included from arch/arm/vfp/vfpmodule.c:23: arch/arm/include/asm/cp15.h:101:2: error: instruction requires: data-barriers 101 | isb(); | ^
This is caused by a change to Debian's LLVM that changes the internal defaults of the arm-linux-gnueabi and arm-linux-gnueabihf tuples:
https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/907baf024b9a5...
We use arm-linux-gnueabi for the kernel (see scripts/Makefile.clang) so now we have a hardcoded armv5te CPU, even if we are building for armv7 or such.
I am still investigating into what (if anything) can be done to resolve this on the kernel side. We could potentially revert commit ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target triple") but I am not sure that will save us from that change, as tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an armv7 CPU even though we may not be building for armv7.
Okay, this is a pretty awful situation the more I look into it :(
The arm64 compat vDSO build is easy enough to fix because we require use of the integrated assembler, which means we can add '-mcpu=generic' (the default in LLVM for those files based on my debugging) to those files and be done with it:
diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile index 1f911a76c5af..5f5cb722cfc2 100644 --- a/arch/arm64/kernel/vdso32/Makefile +++ b/arch/arm64/kernel/vdso32/Makefile @@ -9,6 +9,10 @@ include $(srctree)/lib/vdso/Makefile ifeq ($(CONFIG_CC_IS_CLANG), y) CC_COMPAT ?= $(CC) CC_COMPAT += --target=arm-linux-gnueabi +# Some distributions (such as Debian) change the default CPU for the +# arm-linux-gnueabi target triple, which can break the build. Explicitly set +# the CPU to generic, which is the default for Linux in LLVM. +CC_COMPAT += -mcpu=generic else CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc endif
The failures for all the ARCH=arm configurations appear to be much more difficult to fix because the default CPU value changes based on the '-march' value, which basically means that we would have to hardcode LLVM's default CPU logic into the kernel's Makefile, which is just not maintainable in my opinion. Just doing a multi_v7_defconfig build of arch/arm/ shows the value returned from ARM::getARMCPUForArch() in llvm/lib/TargetParser/ARMTargetParser.cpp can vary between "arm7tdmi" or "generic". Supplying '-mcpu=generic' explicitly won't work with LLVM_IAS=0 because GNU as does not support it and clang just happily passes it along, even though it does not do that in the implicit default case.
Sylvestre, I strongly believe you should consider reverting that change or give us some compiler flag that allows us to fallback to upstream LLVM's default CPU selection logic. I think that hardcoding Debian's architecture defintions based on the target triple into the compiler could cause issues for other projects as well. For example, '--target=arm-linux-gnueabi -march=armv7-a' won't actually target ARMv7:
$ echo 'int main(void) { asm("dsb"); return 0; }' | \ clang --target=arm-linux-gnueabi -march=armv7-a \ -x c -c -o /dev/null -v - ... "/usr/bin/clang-17" -cc1 -triple armv7-unknown-linux-gnueabi ... ...
vs.
$ echo 'int main(void) { asm("dsb"); return 0; }' | \ clang --target=arm-linux-gnueabi -march=armv7-a \ -x c -c -o /dev/null -v - ... "<prefix>/bin/clang-18" -cc1 -triple armv5e-unknown-linux-gnueabi ... ...
Cheers, Nathan
Hello,
Le 04/12/2023 à 23:33, Nathan Chancellor a écrit :
On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
Hi Naresh,
On Mon, Dec 04, 2023 at 05:33:26PM +0530, Naresh Kamboju wrote:
Following build errors noticed on Linux next-20231204 tag with clang-nightly for arm and arm64.
## Test Regressions (compared to next-20231201)
arm64, build
- clang-nightly-defconfig
- clang-nightly-defconfig-40bc7ee5
- clang-nightly-lkftconfig
- clang-nightly-lkftconfig-kselftest
arm, build
- clang-nightly-allnoconfig
- clang-nightly-axm55xx_defconfig
- clang-nightly-bcm2835_defconfig
- clang-nightly-clps711x_defconfig
- clang-nightly-defconfig
- clang-nightly-exynos_defconfig
- clang-nightly-imx_v6_v7_defconfig
- clang-nightly-keystone_defconfig
- clang-nightly-lkftconfig
- clang-nightly-lkftconfig-kselftest
- clang-nightly-omap2plus_defconfig
- clang-nightly-pxa910_defconfig
- clang-nightly-s3c6400_defconfig
- clang-nightly-s5pv210_defconfig
- clang-nightly-sama5_defconfig
- clang-nightly-shmobile_defconfig
- clang-nightly-tinyconfig
- clang-nightly-u8500_defconfig
- clang-nightly-vexpress_defconfig
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
Build log on arm64:
In file included from lib/vdso/gettimeofday.c:5: In file included from include/vdso/datapage.h:135: arch/arm64/include/asm/vdso/compat_gettimeofday.h:152:15: error: instruction variant requires ARMv6 or later 152 | asm volatile("mov %0, %1" : "=r"(ret) : "r"(_vdso_data)); | ^ <inline asm>:1:2: note: instantiated into assembly here 1 | mov r4, r1 | ^ In file included from <built-in>:3: lib/vdso/gettimeofday.c:139:3: error: invalid instruction 139 | smp_rmb(); | ^
Build log on arm:
In file included from arch/arm/vfp/vfpmodule.c:23: arch/arm/include/asm/cp15.h:101:2: error: instruction requires: data-barriers 101 | isb(); | ^
This is caused by a change to Debian's LLVM that changes the internal defaults of the arm-linux-gnueabi and arm-linux-gnueabihf tuples:
https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/907baf024b9a5...
We use arm-linux-gnueabi for the kernel (see scripts/Makefile.clang) so now we have a hardcoded armv5te CPU, even if we are building for armv7 or such.
I am still investigating into what (if anything) can be done to resolve this on the kernel side. We could potentially revert commit ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target triple") but I am not sure that will save us from that change, as tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an armv7 CPU even though we may not be building for armv7.
Okay, this is a pretty awful situation the more I look into it :(
The arm64 compat vDSO build is easy enough to fix because we require use of the integrated assembler, which means we can add '-mcpu=generic' (the default in LLVM for those files based on my debugging) to those files and be done with it:
diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile index 1f911a76c5af..5f5cb722cfc2 100644 --- a/arch/arm64/kernel/vdso32/Makefile +++ b/arch/arm64/kernel/vdso32/Makefile @@ -9,6 +9,10 @@ include $(srctree)/lib/vdso/Makefile ifeq ($(CONFIG_CC_IS_CLANG), y) CC_COMPAT ?= $(CC) CC_COMPAT += --target=arm-linux-gnueabi +# Some distributions (such as Debian) change the default CPU for the +# arm-linux-gnueabi target triple, which can break the build. Explicitly set +# the CPU to generic, which is the default for Linux in LLVM. +CC_COMPAT += -mcpu=generic else CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc endif
The failures for all the ARCH=arm configurations appear to be much more difficult to fix because the default CPU value changes based on the '-march' value, which basically means that we would have to hardcode LLVM's default CPU logic into the kernel's Makefile, which is just not maintainable in my opinion. Just doing a multi_v7_defconfig build of arch/arm/ shows the value returned from ARM::getARMCPUForArch() in llvm/lib/TargetParser/ARMTargetParser.cpp can vary between "arm7tdmi" or "generic". Supplying '-mcpu=generic' explicitly won't work with LLVM_IAS=0 because GNU as does not support it and clang just happily passes it along, even though it does not do that in the implicit default case.
Sylvestre, I strongly believe you should consider reverting that change or give us some compiler flag that allows us to fallback to upstream LLVM's default CPU selection logic. I think that hardcoding Debian's architecture defintions based on the target triple into the compiler could cause issues for other projects as well. For example, '--target=arm-linux-gnueabi -march=armv7-a' won't actually target ARMv7:
$ echo 'int main(void) { asm("dsb"); return 0; }' | \ clang --target=arm-linux-gnueabi -march=armv7-a \ -x c -c -o /dev/null -v - ... "/usr/bin/clang-17" -cc1 -triple armv7-unknown-linux-gnueabi ... ...
vs.
$ echo 'int main(void) { asm("dsb"); return 0; }' | \ clang --target=arm-linux-gnueabi -march=armv7-a \ -x c -c -o /dev/null -v - ... "<prefix>/bin/clang-18" -cc1 -triple armv5e-unknown-linux-gnueabi ... ...
I guess it is this patch, right?
https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/97633b6d51ebc...
if so, do you want me to revert it?
Thanks S
On Mon, Dec 04, 2023 at 11:42:02PM +0100, Sylvestre Ledru wrote:
Hello,
Le 04/12/2023 à 23:33, Nathan Chancellor a écrit :
On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
Hi Naresh,
On Mon, Dec 04, 2023 at 05:33:26PM +0530, Naresh Kamboju wrote:
Following build errors noticed on Linux next-20231204 tag with clang-nightly for arm and arm64.
## Test Regressions (compared to next-20231201)
arm64, build
- clang-nightly-defconfig
- clang-nightly-defconfig-40bc7ee5
- clang-nightly-lkftconfig
- clang-nightly-lkftconfig-kselftest
arm, build
- clang-nightly-allnoconfig
- clang-nightly-axm55xx_defconfig
- clang-nightly-bcm2835_defconfig
- clang-nightly-clps711x_defconfig
- clang-nightly-defconfig
- clang-nightly-exynos_defconfig
- clang-nightly-imx_v6_v7_defconfig
- clang-nightly-keystone_defconfig
- clang-nightly-lkftconfig
- clang-nightly-lkftconfig-kselftest
- clang-nightly-omap2plus_defconfig
- clang-nightly-pxa910_defconfig
- clang-nightly-s3c6400_defconfig
- clang-nightly-s5pv210_defconfig
- clang-nightly-sama5_defconfig
- clang-nightly-shmobile_defconfig
- clang-nightly-tinyconfig
- clang-nightly-u8500_defconfig
- clang-nightly-vexpress_defconfig
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
Build log on arm64:
In file included from lib/vdso/gettimeofday.c:5: In file included from include/vdso/datapage.h:135: arch/arm64/include/asm/vdso/compat_gettimeofday.h:152:15: error: instruction variant requires ARMv6 or later 152 | asm volatile("mov %0, %1" : "=r"(ret) : "r"(_vdso_data)); | ^ <inline asm>:1:2: note: instantiated into assembly here 1 | mov r4, r1 | ^ In file included from <built-in>:3: lib/vdso/gettimeofday.c:139:3: error: invalid instruction 139 | smp_rmb(); | ^
Build log on arm:
In file included from arch/arm/vfp/vfpmodule.c:23: arch/arm/include/asm/cp15.h:101:2: error: instruction requires: data-barriers 101 | isb(); | ^
This is caused by a change to Debian's LLVM that changes the internal defaults of the arm-linux-gnueabi and arm-linux-gnueabihf tuples:
https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/907baf024b9a5...
We use arm-linux-gnueabi for the kernel (see scripts/Makefile.clang) so now we have a hardcoded armv5te CPU, even if we are building for armv7 or such.
I am still investigating into what (if anything) can be done to resolve this on the kernel side. We could potentially revert commit ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target triple") but I am not sure that will save us from that change, as tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an armv7 CPU even though we may not be building for armv7.
Okay, this is a pretty awful situation the more I look into it :(
The arm64 compat vDSO build is easy enough to fix because we require use of the integrated assembler, which means we can add '-mcpu=generic' (the default in LLVM for those files based on my debugging) to those files and be done with it:
diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile index 1f911a76c5af..5f5cb722cfc2 100644 --- a/arch/arm64/kernel/vdso32/Makefile +++ b/arch/arm64/kernel/vdso32/Makefile @@ -9,6 +9,10 @@ include $(srctree)/lib/vdso/Makefile ifeq ($(CONFIG_CC_IS_CLANG), y) CC_COMPAT ?= $(CC) CC_COMPAT += --target=arm-linux-gnueabi +# Some distributions (such as Debian) change the default CPU for the +# arm-linux-gnueabi target triple, which can break the build. Explicitly set +# the CPU to generic, which is the default for Linux in LLVM. +CC_COMPAT += -mcpu=generic else CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc endif
The failures for all the ARCH=arm configurations appear to be much more difficult to fix because the default CPU value changes based on the '-march' value, which basically means that we would have to hardcode LLVM's default CPU logic into the kernel's Makefile, which is just not maintainable in my opinion. Just doing a multi_v7_defconfig build of arch/arm/ shows the value returned from ARM::getARMCPUForArch() in llvm/lib/TargetParser/ARMTargetParser.cpp can vary between "arm7tdmi" or "generic". Supplying '-mcpu=generic' explicitly won't work with LLVM_IAS=0 because GNU as does not support it and clang just happily passes it along, even though it does not do that in the implicit default case.
Sylvestre, I strongly believe you should consider reverting that change or give us some compiler flag that allows us to fallback to upstream LLVM's default CPU selection logic. I think that hardcoding Debian's architecture defintions based on the target triple into the compiler could cause issues for other projects as well. For example, '--target=arm-linux-gnueabi -march=armv7-a' won't actually target ARMv7:
$ echo 'int main(void) { asm("dsb"); return 0; }' | \ clang --target=arm-linux-gnueabi -march=armv7-a \ -x c -c -o /dev/null -v - ... "/usr/bin/clang-17" -cc1 -triple armv7-unknown-linux-gnueabi ... ...
vs.
$ echo 'int main(void) { asm("dsb"); return 0; }' | \ clang --target=arm-linux-gnueabi -march=armv7-a \ -x c -c -o /dev/null -v - ... "<prefix>/bin/clang-18" -cc1 -triple armv5e-unknown-linux-gnueabi ... ...
I guess it is this patch, right?
https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/97633b6d51ebc...
Right, I should have made that clearer when bringing you in (I linked to the snapshot version of that change further up in the thread).
if so, do you want me to revert it?
Yes, I think it should be reverted.
Cheers, Nathan
On Mon, Dec 4, 2023, at 23:33, Nathan Chancellor wrote:
On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
I am still investigating into what (if anything) can be done to resolve this on the kernel side. We could potentially revert commit ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target triple") but I am not sure that will save us from that change, as tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an armv7 CPU even though we may not be building for armv7.
Okay, this is a pretty awful situation the more I look into it :(
The arm64 compat vDSO build is easy enough to fix because we require use of the integrated assembler, which means we can add '-mcpu=generic' (the default in LLVM for those files based on my debugging) to those files and be done with it:
diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile index 1f911a76c5af..5f5cb722cfc2 100644 --- a/arch/arm64/kernel/vdso32/Makefile +++ b/arch/arm64/kernel/vdso32/Makefile @@ -9,6 +9,10 @@ include $(srctree)/lib/vdso/Makefile ifeq ($(CONFIG_CC_IS_CLANG), y) CC_COMPAT ?= $(CC) CC_COMPAT += --target=arm-linux-gnueabi +# Some distributions (such as Debian) change the default CPU for the +# arm-linux-gnueabi target triple, which can break the build. Explicitly set +# the CPU to generic, which is the default for Linux in LLVM. +CC_COMPAT += -mcpu=generic else CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc endif
I'm still trying to follow what is actually going on. I see that we pass
VDSO_CAFLAGS += -march=armv8-a
which is meant to tell the compiler that we want it to use ARMv8 compatible instructions. Is the problem that clang ignores this flag, or do we not pass it correctly?
I would have expected -march=armv8-a to be better than -mcpu=generic here, as it allows the compiler to use a wider set of instructions that is still guaranteed to be available on everything it will run on.
Sylvestre, I strongly believe you should consider reverting that change or give us some compiler flag that allows us to fallback to upstream LLVM's default CPU selection logic. I think that hardcoding Debian's architecture defintions based on the target triple into the compiler could cause issues for other projects as well. For example, '--target=arm-linux-gnueabi -march=armv7-a' won't actually target ARMv7:
$ echo 'int main(void) { asm("dsb"); return 0; }' | \ clang --target=arm-linux-gnueabi -march=armv7-a \ -x c -c -o /dev/null -v - ... "/usr/bin/clang-17" -cc1 -triple armv7-unknown-linux-gnueabi ... ...
vs.
$ echo 'int main(void) { asm("dsb"); return 0; }' | \ clang --target=arm-linux-gnueabi -march=armv7-a \ -x c -c -o /dev/null -v - ... "<prefix>/bin/clang-18" -cc1 -triple armv5e-unknown-linux-gnueabi ... ...
Right, the kernel definitely relies on -march= taking precedence over the default CPU, the same way that we tell the compiler to pick a non-default endianess or ABI.
Arnd
On Tue, Dec 05, 2023 at 07:34:40AM +0100, Arnd Bergmann wrote:
On Mon, Dec 4, 2023, at 23:33, Nathan Chancellor wrote:
On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
I am still investigating into what (if anything) can be done to resolve this on the kernel side. We could potentially revert commit ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target triple") but I am not sure that will save us from that change, as tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an armv7 CPU even though we may not be building for armv7.
Okay, this is a pretty awful situation the more I look into it :(
The arm64 compat vDSO build is easy enough to fix because we require use of the integrated assembler, which means we can add '-mcpu=generic' (the default in LLVM for those files based on my debugging) to those files and be done with it:
diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile index 1f911a76c5af..5f5cb722cfc2 100644 --- a/arch/arm64/kernel/vdso32/Makefile +++ b/arch/arm64/kernel/vdso32/Makefile @@ -9,6 +9,10 @@ include $(srctree)/lib/vdso/Makefile ifeq ($(CONFIG_CC_IS_CLANG), y) CC_COMPAT ?= $(CC) CC_COMPAT += --target=arm-linux-gnueabi +# Some distributions (such as Debian) change the default CPU for the +# arm-linux-gnueabi target triple, which can break the build. Explicitly set +# the CPU to generic, which is the default for Linux in LLVM. +CC_COMPAT += -mcpu=generic else CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc endif
I'm still trying to follow what is actually going on. I see that we pass
VDSO_CAFLAGS += -march=armv8-a
which is meant to tell the compiler that we want it to use ARMv8 compatible instructions. Is the problem that clang ignores this flag, or do we not pass it correctly?
I would have expected -march=armv8-a to be better than -mcpu=generic here, as it allows the compiler to use a wider set of instructions that is still guaranteed to be available on everything it will run on.
I should have made it clearer in that message that adding '-mcpu=generic' was only to avoid the logic added by that Debian LLVM change, not because I believe the kernel is doing something incorrectly now. From what I could tell following through LLVM's code, '-march=' determines the default CPU, which is then used to further inform the full target triple and by overriding the CPU where that patch did, it was just blowing away the user's request. By providing an '-mcpu=' option explicitly, it would avoid the default selection logic and we would get what we asked for.
Sylvestre, I strongly believe you should consider reverting that change or give us some compiler flag that allows us to fallback to upstream LLVM's default CPU selection logic. I think that hardcoding Debian's architecture defintions based on the target triple into the compiler could cause issues for other projects as well. For example, '--target=arm-linux-gnueabi -march=armv7-a' won't actually target ARMv7:
$ echo 'int main(void) { asm("dsb"); return 0; }' | \ clang --target=arm-linux-gnueabi -march=armv7-a \ -x c -c -o /dev/null -v - ... "/usr/bin/clang-17" -cc1 -triple armv7-unknown-linux-gnueabi ... ...
vs.
$ echo 'int main(void) { asm("dsb"); return 0; }' | \ clang --target=arm-linux-gnueabi -march=armv7-a \ -x c -c -o /dev/null -v - ... "<prefix>/bin/clang-18" -cc1 -triple armv5e-unknown-linux-gnueabi ... ...
Right, the kernel definitely relies on -march= taking precedence over the default CPU, the same way that we tell the compiler to pick a non-default endianess or ABI.
Agreed, I have yet to test the new version of the patch but I see you and Ard have given input on it, so hopefully it does not have any problems like this.
Cheers, Nathan
Le 05/12/2023 à 16:04, Nathan Chancellor a écrit :
On Tue, Dec 05, 2023 at 07:34:40AM +0100, Arnd Bergmann wrote:
On Mon, Dec 4, 2023, at 23:33, Nathan Chancellor wrote:
On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
I am still investigating into what (if anything) can be done to resolve this on the kernel side. We could potentially revert commit ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target triple") but I am not sure that will save us from that change, as tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an armv7 CPU even though we may not be building for armv7.
Okay, this is a pretty awful situation the more I look into it :(
The arm64 compat vDSO build is easy enough to fix because we require use of the integrated assembler, which means we can add '-mcpu=generic' (the default in LLVM for those files based on my debugging) to those files and be done with it:
diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile index 1f911a76c5af..5f5cb722cfc2 100644 --- a/arch/arm64/kernel/vdso32/Makefile +++ b/arch/arm64/kernel/vdso32/Makefile @@ -9,6 +9,10 @@ include $(srctree)/lib/vdso/Makefile ifeq ($(CONFIG_CC_IS_CLANG), y) CC_COMPAT ?= $(CC) CC_COMPAT += --target=arm-linux-gnueabi +# Some distributions (such as Debian) change the default CPU for the +# arm-linux-gnueabi target triple, which can break the build. Explicitly set +# the CPU to generic, which is the default for Linux in LLVM. +CC_COMPAT += -mcpu=generic else CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc endif
I'm still trying to follow what is actually going on. I see that we pass
VDSO_CAFLAGS += -march=armv8-a
which is meant to tell the compiler that we want it to use ARMv8 compatible instructions. Is the problem that clang ignores this flag, or do we not pass it correctly?
I would have expected -march=armv8-a to be better than -mcpu=generic here, as it allows the compiler to use a wider set of instructions that is still guaranteed to be available on everything it will run on.
I should have made it clearer in that message that adding '-mcpu=generic' was only to avoid the logic added by that Debian LLVM change, not because I believe the kernel is doing something incorrectly now. From what I could tell following through LLVM's code, '-march=' determines the default CPU, which is then used to further inform the full target triple and by overriding the CPU where that patch did, it was just blowing away the user's request. By providing an '-mcpu=' option explicitly, it would avoid the default selection logic and we would get what we asked for.
Sylvestre, I strongly believe you should consider reverting that change or give us some compiler flag that allows us to fallback to upstream LLVM's default CPU selection logic. I think that hardcoding Debian's architecture defintions based on the target triple into the compiler could cause issues for other projects as well. For example, '--target=arm-linux-gnueabi -march=armv7-a' won't actually target ARMv7:
$ echo 'int main(void) { asm("dsb"); return 0; }' | \ clang --target=arm-linux-gnueabi -march=armv7-a \ -x c -c -o /dev/null -v - ... "/usr/bin/clang-17" -cc1 -triple armv7-unknown-linux-gnueabi ... ...
vs.
$ echo 'int main(void) { asm("dsb"); return 0; }' | \ clang --target=arm-linux-gnueabi -march=armv7-a \ -x c -c -o /dev/null -v - ... "<prefix>/bin/clang-18" -cc1 -triple armv5e-unknown-linux-gnueabi ... ...
Right, the kernel definitely relies on -march= taking precedence over the default CPU, the same way that we tell the compiler to pick a non-default endianess or ABI.
Agreed, I have yet to test the new version of the patch but I see you and Ard have given input on it, so hopefully it does not have any problems like this.
Matthias, as cc, pushed a potential fix for debian/ubuntu packages! https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/01a06b481e5a2...
Cheers, Sylvestre
On Tue, Dec 05, 2023 at 04:13:29PM +0100, Sylvestre Ledru wrote:
Le 05/12/2023 à 16:04, Nathan Chancellor a écrit :
On Tue, Dec 05, 2023 at 07:34:40AM +0100, Arnd Bergmann wrote:
On Mon, Dec 4, 2023, at 23:33, Nathan Chancellor wrote:
On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
I am still investigating into what (if anything) can be done to resolve this on the kernel side. We could potentially revert commit ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target triple") but I am not sure that will save us from that change, as tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an armv7 CPU even though we may not be building for armv7.
Okay, this is a pretty awful situation the more I look into it :(
The arm64 compat vDSO build is easy enough to fix because we require use of the integrated assembler, which means we can add '-mcpu=generic' (the default in LLVM for those files based on my debugging) to those files and be done with it:
diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile index 1f911a76c5af..5f5cb722cfc2 100644 --- a/arch/arm64/kernel/vdso32/Makefile +++ b/arch/arm64/kernel/vdso32/Makefile @@ -9,6 +9,10 @@ include $(srctree)/lib/vdso/Makefile ifeq ($(CONFIG_CC_IS_CLANG), y) CC_COMPAT ?= $(CC) CC_COMPAT += --target=arm-linux-gnueabi +# Some distributions (such as Debian) change the default CPU for the +# arm-linux-gnueabi target triple, which can break the build. Explicitly set +# the CPU to generic, which is the default for Linux in LLVM. +CC_COMPAT += -mcpu=generic else CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc endif
I'm still trying to follow what is actually going on. I see that we pass
VDSO_CAFLAGS += -march=armv8-a
which is meant to tell the compiler that we want it to use ARMv8 compatible instructions. Is the problem that clang ignores this flag, or do we not pass it correctly?
I would have expected -march=armv8-a to be better than -mcpu=generic here, as it allows the compiler to use a wider set of instructions that is still guaranteed to be available on everything it will run on.
I should have made it clearer in that message that adding '-mcpu=generic' was only to avoid the logic added by that Debian LLVM change, not because I believe the kernel is doing something incorrectly now. From what I could tell following through LLVM's code, '-march=' determines the default CPU, which is then used to further inform the full target triple and by overriding the CPU where that patch did, it was just blowing away the user's request. By providing an '-mcpu=' option explicitly, it would avoid the default selection logic and we would get what we asked for.
Sylvestre, I strongly believe you should consider reverting that change or give us some compiler flag that allows us to fallback to upstream LLVM's default CPU selection logic. I think that hardcoding Debian's architecture defintions based on the target triple into the compiler could cause issues for other projects as well. For example, '--target=arm-linux-gnueabi -march=armv7-a' won't actually target ARMv7:
$ echo 'int main(void) { asm("dsb"); return 0; }' | \ clang --target=arm-linux-gnueabi -march=armv7-a \ -x c -c -o /dev/null -v - ... "/usr/bin/clang-17" -cc1 -triple armv7-unknown-linux-gnueabi ... ...
vs.
$ echo 'int main(void) { asm("dsb"); return 0; }' | \ clang --target=arm-linux-gnueabi -march=armv7-a \ -x c -c -o /dev/null -v - ... "<prefix>/bin/clang-18" -cc1 -triple armv5e-unknown-linux-gnueabi ... ...
Right, the kernel definitely relies on -march= taking precedence over the default CPU, the same way that we tell the compiler to pick a non-default endianess or ABI.
Agreed, I have yet to test the new version of the patch but I see you and Ard have given input on it, so hopefully it does not have any problems like this.
Matthias, as cc, pushed a potential fix for debian/ubuntu packages! https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/01a06b481e5a2...
Thanks, that version survives my basic testing of both ARCH=arm and ARCH=arm64 defconfig. I'll holler if our full matrix explodes later (we have another regression in LLVM right now so we are not testing the snapshots daily at the moment).
Cheers, Nathan
-
Arnd Bergmann -
Naresh Kamboju -
Nathan Chancellor -
Russell King (Oracle) -
Sylvestre Ledru