When upreving llvm I realised that kexec stopped working on my test platform.
The reason seems to be that due to PGO there are multiple .text sections on the purgatory, and kexec does not supports that.
Signed-off-by: Ricardo Ribalda ribalda@chromium.org --- Changes in v7: - Fix $SUBJECT of riscv patch - Rename PGO as Profile-guided optimization - Link to v6: https://lore.kernel.org/r/20230321-kexec_clang16-v6-0-a2255e81ab45@chromium....
Changes in v6: - Replace linker script with Makefile rule. Thanks Nick - Link to v5: https://lore.kernel.org/r/20230321-kexec_clang16-v5-0-5563bf7c4173@chromium....
Changes in v5: - Add warning when multiple text sections are found. Thanks Simon! - Add Fixes tag. - Link to v4: https://lore.kernel.org/r/20230321-kexec_clang16-v4-0-1340518f98e9@chromium....
Changes in v4: - Add Cc: stable - Add linker script for x86 - Add a warning when the kernel image has overlapping sections. - Link to v3: https://lore.kernel.org/r/20230321-kexec_clang16-v3-0-5f016c8d0e87@chromium....
Changes in v3: - Fix initial value. Thanks Ross! - Link to v2: https://lore.kernel.org/r/20230321-kexec_clang16-v2-0-d10e5d517869@chromium....
Changes in v2: - Fix if condition. Thanks Steven!. - Update Philipp email. Thanks Baoquan. - Link to v1: https://lore.kernel.org/r/20230321-kexec_clang16-v1-0-a768fc2c7c4d@chromium....
--- Ricardo Ribalda (4): kexec: Support purgatories with .text.hot sections x86/purgatory: Remove PGO flags powerpc/purgatory: Remove PGO flags riscv/purgatory: Remove PGO flags
arch/powerpc/purgatory/Makefile | 5 +++++ arch/riscv/purgatory/Makefile | 5 +++++ arch/x86/purgatory/Makefile | 5 +++++ kernel/kexec_file.c | 14 +++++++++++++- 4 files changed, 28 insertions(+), 1 deletion(-) --- base-commit: 58390c8ce1bddb6c623f62e7ed36383e7fa5c02f change-id: 20230321-kexec_clang16-4510c23d129c
Best regards,
Clang16 links the purgatory text in two sections when PGO is in use:
[ 1] .text PROGBITS 0000000000000000 00000040 00000000000011a1 0000000000000000 AX 0 0 16 [ 2] .rela.text RELA 0000000000000000 00003498 0000000000000648 0000000000000018 I 24 1 8 ... [17] .text.hot. PROGBITS 0000000000000000 00003220 000000000000020b 0000000000000000 AX 0 0 1 [18] .rela.text.hot. RELA 0000000000000000 00004428 0000000000000078 0000000000000018 I 24 17 8
And both of them have their range [sh_addr ... sh_addr+sh_size] on the area pointed by `e_entry`.
This causes that image->start is calculated twice, once for .text and another time for .text.hot. The second calculation leaves image->start in a random location.
Because of this, the system crashes immediately after:
kexec_core: Starting new kernel
Cc: stable@vger.kernel.org Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory") Reviewed-by: Ross Zwisler zwisler@google.com Reviewed-by: Steven Rostedt (Google) rostedt@goodmis.org Reviewed-by: Philipp Rudo prudo@redhat.com Signed-off-by: Ricardo Ribalda ribalda@chromium.org --- kernel/kexec_file.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c index f989f5f1933b..69ee4a29136f 100644 --- a/kernel/kexec_file.c +++ b/kernel/kexec_file.c @@ -901,10 +901,22 @@ static int kexec_purgatory_setup_sechdrs(struct purgatory_info *pi, }
offset = ALIGN(offset, align); + + /* + * Check if the segment contains the entry point, if so, + * calculate the value of image->start based on it. + * If the compiler has produced more than one .text section + * (Eg: .text.hot), they are generally after the main .text + * section, and they shall not be used to calculate + * image->start. So do not re-calculate image->start if it + * is not set to the initial value, and warn the user so they + * have a chance to fix their purgatory's linker script. + */ if (sechdrs[i].sh_flags & SHF_EXECINSTR && pi->ehdr->e_entry >= sechdrs[i].sh_addr && pi->ehdr->e_entry < (sechdrs[i].sh_addr - + sechdrs[i].sh_size)) { + + sechdrs[i].sh_size) && + !WARN_ON(kbuf->image->start != pi->ehdr->e_entry)) { kbuf->image->start -= sechdrs[i].sh_addr; kbuf->image->start += kbuf->mem + offset; }
If profile-guided optimization is enabled, the purgatory ends up with multiple .text sections. This is not supported by kexec and crashes the system.
Cc: stable@vger.kernel.org Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory") Signed-off-by: Ricardo Ribalda ribalda@chromium.org --- arch/x86/purgatory/Makefile | 5 +++++ 1 file changed, 5 insertions(+)
diff --git a/arch/x86/purgatory/Makefile b/arch/x86/purgatory/Makefile index 82fec66d46d2..42abd6af1198 100644 --- a/arch/x86/purgatory/Makefile +++ b/arch/x86/purgatory/Makefile @@ -14,6 +14,11 @@ $(obj)/sha256.o: $(srctree)/lib/crypto/sha256.c FORCE
CFLAGS_sha256.o := -D__DISABLE_EXPORTS
+# When profile-guided optimization is enabled, llvm emits two different +# overlapping text sections, which is not supported by kexec. Remove profile +# optimization flags. +KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=% -fprofile-use=%,$(KBUILD_CFLAGS)) + # When linking purgatory.ro with -r unresolved symbols are not checked, # also link a purgatory.chk binary without -r to check for unresolved symbols. PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
If profile-guided optimization is enabled, the purgatory ends up with multiple .text sections. This is not supported by kexec and crashes the system.
Cc: stable@vger.kernel.org Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory") Signed-off-by: Ricardo Ribalda ribalda@chromium.org --- arch/powerpc/purgatory/Makefile | 5 +++++ 1 file changed, 5 insertions(+)
diff --git a/arch/powerpc/purgatory/Makefile b/arch/powerpc/purgatory/Makefile index 6f5e2727963c..78473d69cd2b 100644 --- a/arch/powerpc/purgatory/Makefile +++ b/arch/powerpc/purgatory/Makefile @@ -5,6 +5,11 @@ KCSAN_SANITIZE := n
targets += trampoline_$(BITS).o purgatory.ro
+# When profile-guided optimization is enabled, llvm emits two different +# overlapping text sections, which is not supported by kexec. Remove profile +# optimization flags. +KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=% -fprofile-use=%,$(KBUILD_CFLAGS)) + LDFLAGS_purgatory.ro := -e purgatory_start -r --no-undefined
$(obj)/purgatory.ro: $(obj)/trampoline_$(BITS).o FORCE
Ricardo Ribalda ribalda@chromium.org writes:
If profile-guided optimization is enabled, the purgatory ends up with multiple .text sections. This is not supported by kexec and crashes the system.
Cc: stable@vger.kernel.org Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory") Signed-off-by: Ricardo Ribalda ribalda@chromium.org
arch/powerpc/purgatory/Makefile | 5 +++++ 1 file changed, 5 insertions(+)
Acked-by: Michael Ellerman mpe@ellerman.id.au (powerpc)
cheers
If profile-guided optimization is enabled, the purgatory ends up with multiple .text sections. This is not supported by kexec and crashes the system.
Cc: stable@vger.kernel.org Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory") Acked-by: Palmer Dabbelt palmer@rivosinc.com Signed-off-by: Ricardo Ribalda ribalda@chromium.org --- arch/riscv/purgatory/Makefile | 5 +++++ 1 file changed, 5 insertions(+)
diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile index 5730797a6b40..bd2e27f82532 100644 --- a/arch/riscv/purgatory/Makefile +++ b/arch/riscv/purgatory/Makefile @@ -35,6 +35,11 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS CFLAGS_string.o := -D__DISABLE_EXPORTS CFLAGS_ctype.o := -D__DISABLE_EXPORTS
+# When profile-guided optimization is enabled, llvm emits two different +# overlapping text sections, which is not supported by kexec. Remove profile +# optimization flags. +KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=% -fprofile-use=%,$(KBUILD_CFLAGS)) + # When linking purgatory.ro with -r unresolved symbols are not checked, # also link a purgatory.chk binary without -r to check for unresolved symbols. PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
Hi Ricardo and folks,
On Fri, May 19, 2023 at 7:48 AM Ricardo Ribalda ribalda@chromium.org wrote:
When upreving llvm I realised that kexec stopped working on my test platform.
The reason seems to be that due to PGO there are multiple .text sections on the purgatory, and kexec does not supports that.
Signed-off-by: Ricardo Ribalda ribalda@chromium.org
We are seeing WARNINGs like the following while kexec'ing a PGO and LTO enabled kernel:
WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919 kexec_load_purgatory+0x37f/0x390
AFAICT, the warning was added by this set, and it was triggered when we have many .text sections in purgatory.ro. The kexec was actually successful. So I wonder whether we really need the WARNING here. If we disable LTO (PGO is still enabled), we don't see the WARNING any more.
I also tested an older kernel (5.19 based), where we also see many .text sections with LTO. It kexec()'ed fine. (It doesn't have the WARN_ON() in kexec_purgatory_setup_sechdrs).
Please help us fix this properly (as I really don't know much about kexec).
Thanks in advance, Song
Here is readelf -S output on purgatory.ro.
With LTO:
readelf -W -S purgatory.ro There are 48 section headers, starting at offset 0x4a10:
Section Headers: [Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0 [ 1] .text PROGBITS 0000000000000000 000040 0000d0 00 AX 0 0 16 [ 2] .data PROGBITS 0000000000000000 001000 001000 00 WA 0 0 4096 [ 3] .rela.text RELA 0000000000000000 003788 000228 18 I 45 1 8 [ 4] .rodata PROGBITS 0000000000000000 002000 0000e0 00 A 0 0 16 [ 5] .rela.rodata RELA 0000000000000000 0039b0 000030 18 I 45 4 8 [ 6] .bss NOBITS 0000000000000000 0020e0 001000 00 WA 0 0 4096 [ 7] .text.purgatory PROGBITS 0000000000000000 0020e0 0000df 00 AX 0 0 16 [ 8] .rela.text.purgatory RELA 0000000000000000 0039e0 000060 18 I 45 7 8 [ 9] .text.warn PROGBITS 0000000000000000 0021c0 000001 00 AX 0 0 16 [10] .kexec-purgatory PROGBITS 0000000000000000 0021d0 000120 00 WA 0 0 16 [11] .comment PROGBITS 0000000000000000 003a40 000046 01 MS 0 0 1 [12] .llvm_addrsig LOOS+0xfff4c03 0000000000000000 003a86 000005 00 E 0 0 1 [13] .text.sha256_update PROGBITS 0000000000000000 0022f0 0008eb 00 AX 0 0 16 [14] .rela.text.sha256_update RELA 0000000000000000 003a90 000060 18 I 45 13 8 [15] .text.sha224_update PROGBITS 0000000000000000 002be0 00000c 00 AX 0 0 16 [16] .rela.text.sha224_update RELA 0000000000000000 003af0 000018 18 I 45 15 8 [17] .text.sha256_final PROGBITS 0000000000000000 002bf0 0000cd 00 AX 0 0 16 [18] .rela.text.sha256_final RELA 0000000000000000 003b08 000030 18 I 45 17 8 [19] .text.sha224_final PROGBITS 0000000000000000 002cc0 0000bd 00 AX 0 0 16 [20] .rela.text.sha224_final RELA 0000000000000000 003b38 000030 18 I 45 19 8 [21] .text.sha256 PROGBITS 0000000000000000 002d80 00011d 00 AX 0 0 16 [22] .rela.text.sha256 RELA 0000000000000000 003b68 000030 18 I 45 21 8 [23] .modinfo PROGBITS 0000000000000000 002e9d 000039 00 A 0 0 1 [24] .rodata.SHA256_K PROGBITS 0000000000000000 002ee0 000100 00 A 0 0 16 [25] .rodata.__sha256_final.padding PROGBITS 0000000000000000 002fe0 000040 00 A 0 0 16 [26] .text.memcmp PROGBITS 0000000000000000 003020 00000b 00 AX 0 0 16 [27] .text.bcmp PROGBITS 0000000000000000 003030 00000b 00 AX 0 0 16 [28] .text.strcmp PROGBITS 0000000000000000 003040 000041 00 AX 0 0 16 [29] .text.strncmp PROGBITS 0000000000000000 003090 00003a 00 AX 0 0 16 [30] .text.strnlen PROGBITS 0000000000000000 0030d0 000039 00 AX 0 0 16 [31] .text.atou PROGBITS 0000000000000000 003110 000035 00 AX 0 0 16 [32] .text.simple_strtoull PROGBITS 0000000000000000 003150 0000b6 00 AX 0 0 16 [33] .text.simple_strtol PROGBITS 0000000000000000 003210 0001b6 00 AX 0 0 16 [34] .text.strlen PROGBITS 0000000000000000 0033d0 00001c 00 AX 0 0 16 [35] .text.strstr PROGBITS 0000000000000000 0033f0 00005f 00 AX 0 0 16 [36] .text.strchr PROGBITS 0000000000000000 003450 000022 00 AX 0 0 16 [37] .text.kstrtoull PROGBITS 0000000000000000 003480 000142 00 AX 0 0 16 [38] .text.boot_kstrtoul PROGBITS 0000000000000000 0035d0 00000c 00 AX 0 0 16 [39] .rela.text.boot_kstrtoul RELA 0000000000000000 003b98 000018 18 I 45 38 8 [40] .text.memset PROGBITS 0000000000000000 0035e0 00001f 00 AX 0 0 16 [41] .text.memmove PROGBITS 0000000000000000 003600 0000a6 00 AX 0 0 16 [42] .text.memcpy PROGBITS 0000000000000000 0036b0 0000a6 00 AX 0 0 16 [43] .rodata.str1.1 PROGBITS 0000000000000000 003756 000032 01 AMS 0 0 1 [44] .note.GNU-stack PROGBITS 0000000000000000 003bb0 000000 00 0 0 1 [45] .symtab SYMTAB 0000000000000000 003bb0 000948 18 47 68 8 [46] .shstrtab STRTAB 0000000000000000 0044f8 0002cd 00 0 0 1 [47] .strtab STRTAB 0000000000000000 0047c5 000248 00 0 0 1 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings), I (info), L (link order), O (extra OS processing required), G (group), T (TLS), C (compressed), x (unknown), o (OS specific), E (exclude), l (large), p (processor specific)
Without LTO:
readelf -W -S purgatory.ro There are 16 section headers, starting at offset 0x4130:
Section Headers: [Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0 [ 1] .text PROGBITS 0000000000000000 000040 00131b 00 AX 0 0 16 [ 2] .rela.text RELA 0000000000000000 003290 000480 18 I 13 1 8 [ 3] .kexec-purgatory PROGBITS 0000000000000000 001360 000120 00 WA 0 0 16 [ 4] .comment PROGBITS 0000000000000000 003710 000046 01 MS 0 0 1 [ 5] .llvm_addrsig LOOS+0xfff4c03 0000000000000000 003756 000005 00 E 0 0 1 [ 6] .data PROGBITS 0000000000000000 002000 001000 00 WA 0 0 4096 [ 7] .rodata PROGBITS 0000000000000000 003000 000220 00 A 0 0 16 [ 8] .rela.rodata RELA 0000000000000000 003760 000030 18 I 13 7 8 [ 9] .bss NOBITS 0000000000000000 003220 001000 00 WA 0 0 4096 [10] .modinfo PROGBITS 0000000000000000 003220 000039 00 A 0 0 1 [11] .rodata.str1.1 PROGBITS 0000000000000000 003259 000032 01 AMS 0 0 1 [12] .note.GNU-stack PROGBITS 0000000000000000 003790 000000 00 0 0 1 [13] .symtab SYMTAB 0000000000000000 003790 0006d8 18 15 43 8 [14] .shstrtab STRTAB 0000000000000000 003e68 00009c 00 0 0 1 [15] .strtab STRTAB 0000000000000000 003f04 00022b 00 0 0 1 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings), I (info), L (link order), O (extra OS processing required), G (group), T (TLS), C (compressed), x (unknown), o (OS specific), E (exclude), l (large), p (processor specific)
[...]
Ricardo Ribalda (4): kexec: Support purgatories with .text.hot sections x86/purgatory: Remove PGO flags powerpc/purgatory: Remove PGO flags riscv/purgatory: Remove PGO flags
arch/powerpc/purgatory/Makefile | 5 +++++ arch/riscv/purgatory/Makefile | 5 +++++ arch/x86/purgatory/Makefile | 5 +++++ kernel/kexec_file.c | 14 +++++++++++++- 4 files changed, 28 insertions(+), 1 deletion(-)
base-commit: 58390c8ce1bddb6c623f62e7ed36383e7fa5c02f change-id: 20230321-kexec_clang16-4510c23d129c
Best regards,
Ricardo Ribalda Delgado ribalda@chromium.org
Hi Song
On Fri, 8 Sept 2023 at 01:08, Song Liu song@kernel.org wrote:
Hi Ricardo and folks,
On Fri, May 19, 2023 at 7:48 AM Ricardo Ribalda ribalda@chromium.org wrote:
When upreving llvm I realised that kexec stopped working on my test platform.
The reason seems to be that due to PGO there are multiple .text sections on the purgatory, and kexec does not supports that.
Signed-off-by: Ricardo Ribalda ribalda@chromium.org
We are seeing WARNINGs like the following while kexec'ing a PGO and LTO enabled kernel:
WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919 kexec_load_purgatory+0x37f/0x390
AFAICT, the warning was added by this set, and it was triggered when we have many .text sections in purgatory.ro. The kexec was actually successful. So I wonder whether we really need the WARNING here. If we disable LTO (PGO is still enabled), we don't see the WARNING any more.
I also tested an older kernel (5.19 based), where we also see many .text sections with LTO. It kexec()'ed fine. (It doesn't have the WARN_ON() in kexec_purgatory_setup_sechdrs).
You have been "lucky" that the code has chosen the correct start address, you need to modify the linker script of your kernel to disable PGO. You need to backport a patch like this: https://lore.kernel.org/lkml/CAPhsuW5_qAvV0N3o+hOiAnb1=buJ1pLzqYW9D+Bwft6hxJ...
(assuming x86)
Regards
Please help us fix this properly (as I really don't know much about kexec).
Thanks in advance, Song
Here is readelf -S output on purgatory.ro.
With LTO:
readelf -W -S purgatory.ro There are 48 section headers, starting at offset 0x4a10:
Section Headers: [Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0 [ 1] .text PROGBITS 0000000000000000 000040 0000d0 00 AX 0 0 16 [ 2] .data PROGBITS 0000000000000000 001000 001000 00 WA 0 0 4096 [ 3] .rela.text RELA 0000000000000000 003788 000228 18 I 45 1 8 [ 4] .rodata PROGBITS 0000000000000000 002000 0000e0 00 A 0 0 16 [ 5] .rela.rodata RELA 0000000000000000 0039b0 000030 18 I 45 4 8 [ 6] .bss NOBITS 0000000000000000 0020e0 001000 00 WA 0 0 4096 [ 7] .text.purgatory PROGBITS 0000000000000000 0020e0 0000df 00 AX 0 0 16 [ 8] .rela.text.purgatory RELA 0000000000000000 0039e0 000060 18 I 45 7 8 [ 9] .text.warn PROGBITS 0000000000000000 0021c0 000001 00 AX 0 0 16 [10] .kexec-purgatory PROGBITS 0000000000000000 0021d0 000120 00 WA 0 0 16 [11] .comment PROGBITS 0000000000000000 003a40 000046 01 MS 0 0 1 [12] .llvm_addrsig LOOS+0xfff4c03 0000000000000000 003a86 000005 00 E 0 0 1 [13] .text.sha256_update PROGBITS 0000000000000000 0022f0 0008eb 00 AX 0 0 16 [14] .rela.text.sha256_update RELA 0000000000000000 003a90 000060 18 I 45 13 8 [15] .text.sha224_update PROGBITS 0000000000000000 002be0 00000c 00 AX 0 0 16 [16] .rela.text.sha224_update RELA 0000000000000000 003af0 000018 18 I 45 15 8 [17] .text.sha256_final PROGBITS 0000000000000000 002bf0 0000cd 00 AX 0 0 16 [18] .rela.text.sha256_final RELA 0000000000000000 003b08 000030 18 I 45 17 8 [19] .text.sha224_final PROGBITS 0000000000000000 002cc0 0000bd 00 AX 0 0 16 [20] .rela.text.sha224_final RELA 0000000000000000 003b38 000030 18 I 45 19 8 [21] .text.sha256 PROGBITS 0000000000000000 002d80 00011d 00 AX 0 0 16 [22] .rela.text.sha256 RELA 0000000000000000 003b68 000030 18 I 45 21 8 [23] .modinfo PROGBITS 0000000000000000 002e9d 000039 00 A 0 0 1 [24] .rodata.SHA256_K PROGBITS 0000000000000000 002ee0 000100 00 A 0 0 16 [25] .rodata.__sha256_final.padding PROGBITS 0000000000000000 002fe0 000040 00 A 0 0 16 [26] .text.memcmp PROGBITS 0000000000000000 003020 00000b 00 AX 0 0 16 [27] .text.bcmp PROGBITS 0000000000000000 003030 00000b 00 AX 0 0 16 [28] .text.strcmp PROGBITS 0000000000000000 003040 000041 00 AX 0 0 16 [29] .text.strncmp PROGBITS 0000000000000000 003090 00003a 00 AX 0 0 16 [30] .text.strnlen PROGBITS 0000000000000000 0030d0 000039 00 AX 0 0 16 [31] .text.atou PROGBITS 0000000000000000 003110 000035 00 AX 0 0 16 [32] .text.simple_strtoull PROGBITS 0000000000000000 003150 0000b6 00 AX 0 0 16 [33] .text.simple_strtol PROGBITS 0000000000000000 003210 0001b6 00 AX 0 0 16 [34] .text.strlen PROGBITS 0000000000000000 0033d0 00001c 00 AX 0 0 16 [35] .text.strstr PROGBITS 0000000000000000 0033f0 00005f 00 AX 0 0 16 [36] .text.strchr PROGBITS 0000000000000000 003450 000022 00 AX 0 0 16 [37] .text.kstrtoull PROGBITS 0000000000000000 003480 000142 00 AX 0 0 16 [38] .text.boot_kstrtoul PROGBITS 0000000000000000 0035d0 00000c 00 AX 0 0 16 [39] .rela.text.boot_kstrtoul RELA 0000000000000000 003b98 000018 18 I 45 38 8 [40] .text.memset PROGBITS 0000000000000000 0035e0 00001f 00 AX 0 0 16 [41] .text.memmove PROGBITS 0000000000000000 003600 0000a6 00 AX 0 0 16 [42] .text.memcpy PROGBITS 0000000000000000 0036b0 0000a6 00 AX 0 0 16 [43] .rodata.str1.1 PROGBITS 0000000000000000 003756 000032 01 AMS 0 0 1 [44] .note.GNU-stack PROGBITS 0000000000000000 003bb0 000000 00 0 0 1 [45] .symtab SYMTAB 0000000000000000 003bb0 000948 18 47 68 8 [46] .shstrtab STRTAB 0000000000000000 0044f8 0002cd 00 0 0 1 [47] .strtab STRTAB 0000000000000000 0047c5 000248 00 0 0 1 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings), I (info), L (link order), O (extra OS processing required), G (group), T (TLS), C (compressed), x (unknown), o (OS specific), E (exclude), l (large), p (processor specific)
Without LTO:
readelf -W -S purgatory.ro There are 16 section headers, starting at offset 0x4130:
Section Headers: [Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0 [ 1] .text PROGBITS 0000000000000000 000040 00131b 00 AX 0 0 16 [ 2] .rela.text RELA 0000000000000000 003290 000480 18 I 13 1 8 [ 3] .kexec-purgatory PROGBITS 0000000000000000 001360 000120 00 WA 0 0 16 [ 4] .comment PROGBITS 0000000000000000 003710 000046 01 MS 0 0 1 [ 5] .llvm_addrsig LOOS+0xfff4c03 0000000000000000 003756 000005 00 E 0 0 1 [ 6] .data PROGBITS 0000000000000000 002000 001000 00 WA 0 0 4096 [ 7] .rodata PROGBITS 0000000000000000 003000 000220 00 A 0 0 16 [ 8] .rela.rodata RELA 0000000000000000 003760 000030 18 I 13 7 8 [ 9] .bss NOBITS 0000000000000000 003220 001000 00 WA 0 0 4096 [10] .modinfo PROGBITS 0000000000000000 003220 000039 00 A 0 0 1 [11] .rodata.str1.1 PROGBITS 0000000000000000 003259 000032 01 AMS 0 0 1 [12] .note.GNU-stack PROGBITS 0000000000000000 003790 000000 00 0 0 1 [13] .symtab SYMTAB 0000000000000000 003790 0006d8 18 15 43 8 [14] .shstrtab STRTAB 0000000000000000 003e68 00009c 00 0 0 1 [15] .strtab STRTAB 0000000000000000 003f04 00022b 00 0 0 1 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings), I (info), L (link order), O (extra OS processing required), G (group), T (TLS), C (compressed), x (unknown), o (OS specific), E (exclude), l (large), p (processor specific)
[...]
Ricardo Ribalda (4): kexec: Support purgatories with .text.hot sections x86/purgatory: Remove PGO flags powerpc/purgatory: Remove PGO flags riscv/purgatory: Remove PGO flags
arch/powerpc/purgatory/Makefile | 5 +++++ arch/riscv/purgatory/Makefile | 5 +++++ arch/x86/purgatory/Makefile | 5 +++++ kernel/kexec_file.c | 14 +++++++++++++- 4 files changed, 28 insertions(+), 1 deletion(-)
base-commit: 58390c8ce1bddb6c623f62e7ed36383e7fa5c02f change-id: 20230321-kexec_clang16-4510c23d129c
Best regards,
Ricardo Ribalda Delgado ribalda@chromium.org
Hi Ricardo,
Thanks for your kind reply.
On Fri, Sep 8, 2023 at 2:18 PM Ricardo Ribalda ribalda@chromium.org wrote:
Hi Song
On Fri, 8 Sept 2023 at 01:08, Song Liu song@kernel.org wrote:
Hi Ricardo and folks,
On Fri, May 19, 2023 at 7:48 AM Ricardo Ribalda ribalda@chromium.org wrote:
When upreving llvm I realised that kexec stopped working on my test platform.
The reason seems to be that due to PGO there are multiple .text sections on the purgatory, and kexec does not supports that.
Signed-off-by: Ricardo Ribalda ribalda@chromium.org
We are seeing WARNINGs like the following while kexec'ing a PGO and LTO enabled kernel:
WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919 kexec_load_purgatory+0x37f/0x390
AFAICT, the warning was added by this set, and it was triggered when we have many .text sections in purgatory.ro. The kexec was actually successful. So I wonder whether we really need the WARNING here. If we disable LTO (PGO is still enabled), we don't see the WARNING any more.
I also tested an older kernel (5.19 based), where we also see many .text sections with LTO. It kexec()'ed fine. (It doesn't have the WARN_ON() in kexec_purgatory_setup_sechdrs).
You have been "lucky" that the code has chosen the correct start address, you need to modify the linker script of your kernel to disable PGO. You need to backport a patch like this: https://lore.kernel.org/lkml/CAPhsuW5_qAvV0N3o+hOiAnb1=buJ1pLzqYW9D+Bwft6hxJ...
We already have this commit in our branch. AFAICT, the issue was triggered by LTO. So something like the following seems fixes it (I haven't finished the end-to-end test yet). Does this change make sense to you?
Thanks again, Song
diff --git i/arch/x86/purgatory/Makefile w/arch/x86/purgatory/Makefile index 8f71aaa04cc2..dc306fa7197d 100644 --- i/arch/x86/purgatory/Makefile +++ w/arch/x86/purgatory/Makefile @@ -19,6 +19,10 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS # optimization flags. KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=% -fprofile-use=%,$(KBUILD_CFLAGS))
+# When LTO is enabled, llvm emits many text sections, which is not supported +# by kexec. Remove -flto=* flags. +KBUILD_CFLAGS := $(filter-out -flto=%,$(KBUILD_CFLAGS)) + # When linking purgatory.ro with -r unresolved symbols are not checked, # also link a purgatory.chk binary without -r to check for unresolved symbols. PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
Hi Song
On Fri, 8 Sept 2023 at 23:48, Song Liu song@kernel.org wrote:
Hi Ricardo,
Thanks for your kind reply.
On Fri, Sep 8, 2023 at 2:18 PM Ricardo Ribalda ribalda@chromium.org wrote:
Hi Song
On Fri, 8 Sept 2023 at 01:08, Song Liu song@kernel.org wrote:
Hi Ricardo and folks,
On Fri, May 19, 2023 at 7:48 AM Ricardo Ribalda ribalda@chromium.org wrote:
When upreving llvm I realised that kexec stopped working on my test platform.
The reason seems to be that due to PGO there are multiple .text sections on the purgatory, and kexec does not supports that.
Signed-off-by: Ricardo Ribalda ribalda@chromium.org
We are seeing WARNINGs like the following while kexec'ing a PGO and LTO enabled kernel:
WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919 kexec_load_purgatory+0x37f/0x390
AFAICT, the warning was added by this set, and it was triggered when we have many .text sections in purgatory.ro. The kexec was actually successful. So I wonder whether we really need the WARNING here. If we disable LTO (PGO is still enabled), we don't see the WARNING any more.
I also tested an older kernel (5.19 based), where we also see many .text sections with LTO. It kexec()'ed fine. (It doesn't have the WARN_ON() in kexec_purgatory_setup_sechdrs).
You have been "lucky" that the code has chosen the correct start address, you need to modify the linker script of your kernel to disable PGO. You need to backport a patch like this: https://lore.kernel.org/lkml/CAPhsuW5_qAvV0N3o+hOiAnb1=buJ1pLzqYW9D+Bwft6hxJ...
We already have this commit in our branch. AFAICT, the issue was triggered by LTO. So something like the following seems fixes it (I haven't finished the end-to-end test yet). Does this change make sense to you?
if the end-to-end works, please send it as a patch to the mailing list.
Thanks! :)
Thanks again, Song
diff --git i/arch/x86/purgatory/Makefile w/arch/x86/purgatory/Makefile index 8f71aaa04cc2..dc306fa7197d 100644 --- i/arch/x86/purgatory/Makefile +++ w/arch/x86/purgatory/Makefile @@ -19,6 +19,10 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS # optimization flags. KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=% -fprofile-use=%,$(KBUILD_CFLAGS))
+# When LTO is enabled, llvm emits many text sections, which is not supported +# by kexec. Remove -flto=* flags. +KBUILD_CFLAGS := $(filter-out -flto=%,$(KBUILD_CFLAGS))
# When linking purgatory.ro with -r unresolved symbols are not checked, # also link a purgatory.chk binary without -r to check for unresolved symbols. PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
On Fri, Sep 8, 2023 at 2:52 PM Ricardo Ribalda ribalda@chromium.org wrote:
Hi Song
On Fri, 8 Sept 2023 at 23:48, Song Liu song@kernel.org wrote:
Hi Ricardo,
Thanks for your kind reply.
On Fri, Sep 8, 2023 at 2:18 PM Ricardo Ribalda ribalda@chromium.org wrote:
Hi Song
On Fri, 8 Sept 2023 at 01:08, Song Liu song@kernel.org wrote:
Hi Ricardo and folks,
On Fri, May 19, 2023 at 7:48 AM Ricardo Ribalda ribalda@chromium.org wrote:
When upreving llvm I realised that kexec stopped working on my test platform.
The reason seems to be that due to PGO there are multiple .text sections on the purgatory, and kexec does not supports that.
Signed-off-by: Ricardo Ribalda ribalda@chromium.org
We are seeing WARNINGs like the following while kexec'ing a PGO and LTO enabled kernel:
WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919 kexec_load_purgatory+0x37f/0x390
AFAICT, the warning was added by this set, and it was triggered when we have many .text sections in purgatory.ro. The kexec was actually successful. So I wonder whether we really need the WARNING here. If we disable LTO (PGO is still enabled), we don't see the WARNING any more.
I also tested an older kernel (5.19 based), where we also see many .text sections with LTO. It kexec()'ed fine. (It doesn't have the WARN_ON() in kexec_purgatory_setup_sechdrs).
You have been "lucky" that the code has chosen the correct start address, you need to modify the linker script of your kernel to disable PGO. You need to backport a patch like this: https://lore.kernel.org/lkml/CAPhsuW5_qAvV0N3o+hOiAnb1=buJ1pLzqYW9D+Bwft6hxJ...
We already have this commit in our branch. AFAICT, the issue was triggered by LTO. So something like the following seems fixes it (I haven't finished the end-to-end test yet). Does this change make sense to you?
if the end-to-end works, please send it as a patch to the mailing list.
Thanks! :)
OK, it works (AFAICT). Sending the patch.
Thanks, Song
linux-stable-mirror@lists.linaro.org
-
Michael Ellerman -
Ricardo Ribalda -
Song Liu