On Tue, 20 Oct 2020 at 22:16, Nick Desaulniers ndesaulniers@google.com wrote:

On Tue, Oct 20, 2020 at 10:57 AM Will Deacon will@kernel.org wrote:

...

On Fri, 16 Oct 2020 10:53:39 -0700, Nick Desaulniers wrote:

...

With CONFIG_EXPERT=y, CONFIG_KASAN=y, CONFIG_RANDOMIZE_BASE=n, CONFIG_RELOCATABLE=n, we observe the following failure when trying to link the kernel image with LD=ld.lld:

error: section: .exit.data is not contiguous with other relro sections

ld.lld defaults to -z relro while ld.bfd defaults to -z norelro. This was previously fixed, but only for CONFIG_RELOCATABLE=y.

Applied to arm64 (for-next/core), thanks!

[1/1] arm64: link with -z norelro regardless of CONFIG_RELOCATABLE https://git.kernel.org/arm64/c/3b92fa7485eb

IF we wanted to go further and remove `-z norelro`, or even enable `-z relro` for aarch64, then we would have to detangle some KASAN/GCOV generated section discard spaghetti.

Why on earth would we want that?

Fangrui did some more digging and found that .fini_array.* sections were relro (read only after relocations, IIUC), so adding them to EXIT_DATA (include/asm-generic/vmlinux.lds.h) was causing them to get included in .exit.data (arch/arm64/kernel/vmlinux.lds.S) making that relro. There's some history here with commits:

• e41f501d39126 ("vmlinux.lds: account for destructor sections")
• 8dcf86caa1e3da ("vmlinux.lds.h: Fix incomplete .text.exit discards")
• d812db78288d7 ("vmlinux.lds.h: Avoid KASAN and KCSAN's unwanted sections")

It seems the following works for quite a few different configs/toolchains I played with, but the big IF is whether enabling `-z relro` is worthwhile? If the kernel does respect that mapping, then I assume that's a yes, but I haven't checked yet whether relro is respected within the kernel (`grep -rn RELRO` turns up nothing interesting). I also haven't checked yet whether all supported versions of GNU ld.bfd support -z relro (guessing not, since a quick test warns: `aarch64-linux-gnu-ld: warning: -z relro ignored` for v2.34.90.20200706, may be holding it wrong).

RELRO just moves statically initialized const pointers into a separate section so we can place it in a way that allows us to easily map it r/w during load, and switch it over to r/o once the relocations have been applied.

On AArch64, we don't even use -fpic to build the kernel, and load time relocations may appear everywhere in .text, .rodata etc etc, which is absolutely fine given that we apply the relocations way before we finalize the kernel mappings. This means that, in our case, statically initialized const pointers will be mapped r/o already, and we don't need RELRO.

In general, we should ensure that the 'relocatable bare metal' case doesn't get snowed under, as toolchain development is [understandably] very focused on hosted binaries that use shared libraries, where things like CoW footprint, ELF symbol preemption, text relocations and RELRO sections actually matter. For bare metal, it is quite the opposite: text relocations are fine, there is no CoW so minimizing the footprint of the .so pages that are modified due to relocations is unnecessary, and symbols cannot be preempted either. So many of the shared library tricks actually make things worse for us, because we have to work around them while they have no benefit for us.

I have suggested this before, but perhaps we should have a -mcmodel=kernel (like x86 does) that takes these things into account? As a start, it could imply -cmodel=small (which we rely on today), but with guarantees that the generated code is position independent, but without GOT indirections, and that the resulting object code can be linked with -pie (so that we have access to the load time relocations in the bare metal binary itself). This is something we rely on today, and happens to work in practice, but this could easily break in the future.

(Fangrui also filed https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97507 in regards to GCOV+GCC)

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index cd14444bf600..64578c998e53 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -744,7 +744,6 @@

#define EXIT_DATA \ *(.exit.data .exit.data.*) \

•   *(.fini_array .fini_array.*)                                    \
    *(.dtors .dtors.*)                                              \
    MEM_DISCARD(exit.data*)                                         \
    MEM_DISCARD(exit.rodata*)
  

@@ -995,6 +994,7 @@ #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KCSAN) # ifdef CONFIG_CONSTRUCTORS # define SANITIZER_DISCARDS \

•   *(.fini_array .fini_array.*)                                    \
    *(.eh_frame)
  

# else # define SANITIZER_DISCARDS \ @@ -1005,8 +1005,16 @@ # define SANITIZER_DISCARDS #endif

+#if defined(CONFIG_GCOV_KERNEL) && defined(CONFIG_CC_IS_GCC) +# define GCOV_DISCARDS \

•   *(.fini_array .fini_array.*)
  

+#else +# define GCOV_DISCARDS +#endif

#define COMMON_DISCARDS \ SANITIZER_DISCARDS \

•   GCOV_DISCARDS                                                   \
    *(.discard)                                                     \
    *(.discard.*)                                                   \
    *(.modinfo)                                                     \
  

-- Thanks, ~Nick Desaulniers

With newer GNU binutils, linking with BFD produces warnings for vmlinux: aarch64-linux-gnu-ld: warning: -z norelro ignored

BFD can produce this warning when the target emulation mode does not support RELRO relocation types, and -z relro or -z norelro is passed.

Alan Modra clarifies: The default linker emulation for an aarch64-linux ld.bfd is -maarch64linux, the default for an aarch64-elf linker is -maarch64elf. They are not equivalent. If you choose -maarch64elf you get an emulation that doesn't support -z relro.

The ARCH=arm64 kernel prefers -maarch64elf, but may fall back to -maarch64linux based on the toolchain configuration.

LLD will always create RELRO relocation types regardless of target emulation.

To avoid the above warning when linking with BFD, pass -z norelro only when linking with LLD or with -maarch64linux.

Cc: Alan Modra amodra@gmail.com Cc: Ard Biesheuvel ardb@kernel.org Cc: Fāng-ruì Sòng maskray@google.com Fixes: 3b92fa7485eb ("arm64: link with -z norelro regardless of CONFIG_RELOCATABLE") Reported-by: kernelci.org bot bot@kernelci.org Reported-by: Quentin Perret qperret@google.com Signed-off-by: Nick Desaulniers ndesaulniers@google.com --- arch/arm64/Makefile | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 6be9b3750250..90309208bb28 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -10,7 +10,7 @@ # # Copyright (C) 1995-2001 by Russell King

-LDFLAGS_vmlinux :=--no-undefined -X -z norelro +LDFLAGS_vmlinux :=--no-undefined -X

ifeq ($(CONFIG_RELOCATABLE), y) # Pass --no-apply-dynamic-relocs to restore pre-binutils-2.27 behaviour @@ -115,16 +115,20 @@ KBUILD_CPPFLAGS += -mbig-endian CHECKFLAGS += -D__AARCH64EB__ # Prefer the baremetal ELF build target, but not all toolchains include # it so fall back to the standard linux version if needed. -KBUILD_LDFLAGS += -EB $(call ld-option, -maarch64elfb, -maarch64linuxb) +KBUILD_LDFLAGS += -EB $(call ld-option, -maarch64elfb, -maarch64linuxb -z norelro) UTS_MACHINE := aarch64_be else KBUILD_CPPFLAGS += -mlittle-endian CHECKFLAGS += -D__AARCH64EL__ # Same as above, prefer ELF but fall back to linux target if needed. -KBUILD_LDFLAGS += -EL $(call ld-option, -maarch64elf, -maarch64linux) +KBUILD_LDFLAGS += -EL $(call ld-option, -maarch64elf, -maarch64linux -z norelro) UTS_MACHINE := aarch64 endif

+ifeq ($(CONFIG_LD_IS_LLD), y) +KBUILD_LDFLAGS += -z norelro +endif + CHECKFLAGS += -D__aarch64__

ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_REGS),y)

On Thu, Dec 17, 2020 at 04:24:32PM -0800, 'Nick Desaulniers' via Clang Built Linux wrote:

With GNU binutils 2.35+, linking with BFD produces warnings for vmlinux: aarch64-linux-gnu-ld: warning: -z norelro ignored

BFD can produce this warning when the target emulation mode does not support RELRO program headers, and -z relro or -z norelro is passed.

Alan Modra clarifies: The default linker emulation for an aarch64-linux ld.bfd is -maarch64linux, the default for an aarch64-elf linker is -maarch64elf. They are not equivalent. If you choose -maarch64elf you get an emulation that doesn't support -z relro.

The ARCH=arm64 kernel prefers -maarch64elf, but may fall back to -maarch64linux based on the toolchain configuration.

LLD will always create RELRO program header regardless of target emulation.

To avoid the above warning when linking with BFD, pass -z norelro only when linking with LLD or with -maarch64linux.

Cc: Alan Modra amodra@gmail.com Cc: Fāng-ruì Sòng maskray@google.com Fixes: 3b92fa7485eb ("arm64: link with -z norelro regardless of CONFIG_RELOCATABLE") Reported-by: kernelci.org bot bot@kernelci.org Reported-by: Quentin Perret qperret@google.com Acked-by: Ard Biesheuvel ardb@kernel.org Signed-off-by: Nick Desaulniers ndesaulniers@google.com

Reviewed-by: Nathan Chancellor natechancellor@gmail.com

---

Changes V1 -> V2:

• s/relocation types/program headers/
• s/newer GNU binutils/GNU binutils 2.35+/
• Pick up Ard's Ack.

Note: maintainers may want to pick up the following tag:

Fixes: 3bbd3db86470 ("arm64: relocatable: fix inconsistencies in linker script and options")

or drop the existing fixes tag (this patch is more so in response to change to BFD to warn than fix a kernel regression, IMO, but I don't care). Either way, it would be good to fix this for the newly minted v5.10.y.

Should probably have

Cc: stable@vger.kernel.org

then.

I'll probably be offline for the next two weeks for the holidays, so no promises on quick replies. Happy holidays+new year!

arch/arm64/Makefile | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 6be9b3750250..90309208bb28 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -10,7 +10,7 @@ # # Copyright (C) 1995-2001 by Russell King -LDFLAGS_vmlinux :=--no-undefined -X -z norelro +LDFLAGS_vmlinux :=--no-undefined -X ifeq ($(CONFIG_RELOCATABLE), y) # Pass --no-apply-dynamic-relocs to restore pre-binutils-2.27 behaviour @@ -115,16 +115,20 @@ KBUILD_CPPFLAGS += -mbig-endian CHECKFLAGS += -D__AARCH64EB__ # Prefer the baremetal ELF build target, but not all toolchains include # it so fall back to the standard linux version if needed. -KBUILD_LDFLAGS += -EB $(call ld-option, -maarch64elfb, -maarch64linuxb) +KBUILD_LDFLAGS += -EB $(call ld-option, -maarch64elfb, -maarch64linuxb -z norelro) UTS_MACHINE := aarch64_be else KBUILD_CPPFLAGS += -mlittle-endian CHECKFLAGS += -D__AARCH64EL__ # Same as above, prefer ELF but fall back to linux target if needed. -KBUILD_LDFLAGS += -EL $(call ld-option, -maarch64elf, -maarch64linux) +KBUILD_LDFLAGS += -EL $(call ld-option, -maarch64elf, -maarch64linux -z norelro) UTS_MACHINE := aarch64 endif +ifeq ($(CONFIG_LD_IS_LLD), y) +KBUILD_LDFLAGS += -z norelro +endif

CHECKFLAGS += -D__aarch64__ ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_REGS),y) -- 2.29.2.684.gfbc64c5ab5-goog

On Thu, 17 Dec 2020 at 01:41, Nick Desaulniers ndesaulniers@google.com wrote:

With newer GNU binutils, linking with BFD produces warnings for vmlinux: aarch64-linux-gnu-ld: warning: -z norelro ignored

BFD can produce this warning when the target emulation mode does not support RELRO relocation types, and -z relro or -z norelro is passed.

RELRO is not a relocation type, it is a type of program header which we might simply ignore, if it weren't for the fact that it can only be emitted if the layout of the sections adheres to certain rules (and ours doesn't), and we get an error otherwise.

It amounts to implicit __ro_after_init annotations for statically initialized const pointers, but given that we don't compile with -fpie, those const pointers reside in .rodata already, so RELRO adds no value for us.

Alan Modra clarifies: The default linker emulation for an aarch64-linux ld.bfd is -maarch64linux, the default for an aarch64-elf linker is -maarch64elf. They are not equivalent. If you choose -maarch64elf you get an emulation that doesn't support -z relro.

The ARCH=arm64 kernel prefers -maarch64elf, but may fall back to -maarch64linux based on the toolchain configuration.

LLD will always create RELRO relocation types regardless of target emulation.

RELRO program header

To avoid the above warning when linking with BFD, pass -z norelro only when linking with LLD or with -maarch64linux.

Cc: Alan Modra amodra@gmail.com Cc: Ard Biesheuvel ardb@kernel.org Cc: Fāng-ruì Sòng maskray@google.com Fixes: 3b92fa7485eb ("arm64: link with -z norelro regardless of CONFIG_RELOCATABLE") Reported-by: kernelci.org bot bot@kernelci.org Reported-by: Quentin Perret qperret@google.com Signed-off-by: Nick Desaulniers ndesaulniers@google.com

With mentions of 'RELRO relocation types' fixed:

Acked-by: Ard Biesheuvel ardb@kernel.org

---

arch/arm64/Makefile | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 6be9b3750250..90309208bb28 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -10,7 +10,7 @@ # # Copyright (C) 1995-2001 by Russell King

-LDFLAGS_vmlinux :=--no-undefined -X -z norelro +LDFLAGS_vmlinux :=--no-undefined -X

ifeq ($(CONFIG_RELOCATABLE), y) # Pass --no-apply-dynamic-relocs to restore pre-binutils-2.27 behaviour @@ -115,16 +115,20 @@ KBUILD_CPPFLAGS += -mbig-endian CHECKFLAGS += -D__AARCH64EB__ # Prefer the baremetal ELF build target, but not all toolchains include # it so fall back to the standard linux version if needed. -KBUILD_LDFLAGS += -EB $(call ld-option, -maarch64elfb, -maarch64linuxb) +KBUILD_LDFLAGS += -EB $(call ld-option, -maarch64elfb, -maarch64linuxb -z norelro) UTS_MACHINE := aarch64_be else KBUILD_CPPFLAGS += -mlittle-endian CHECKFLAGS += -D__AARCH64EL__ # Same as above, prefer ELF but fall back to linux target if needed. -KBUILD_LDFLAGS += -EL $(call ld-option, -maarch64elf, -maarch64linux) +KBUILD_LDFLAGS += -EL $(call ld-option, -maarch64elf, -maarch64linux -z norelro) UTS_MACHINE := aarch64 endif

+ifeq ($(CONFIG_LD_IS_LLD), y) +KBUILD_LDFLAGS += -z norelro +endif

CHECKFLAGS += -D__aarch64__

ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_REGS),y)

2.29.2.684.gfbc64c5ab5-goog