A recent LLVM 11 commit [1] made LLD stop implicitly coalescing some temporary LLVM sections, namely .{data,bss}..compoundliteral.XXX:
[30] .data..compoundli PROGBITS ffffffff9ac9a000 19e9a000 000000000000cea0 0000000000000000 WA 0 0 32 [31] .rela.data..compo RELA 0000000000000000 40965440 0000000000001d88 0000000000000018 I 2238 30 8 [32] .data..compoundli PROGBITS ffffffff9aca6ea0 19ea6ea0 00000000000033c0 0000000000000000 WA 0 0 32 [33] .rela.data..compo RELA 0000000000000000 409671c8 0000000000000948 0000000000000018 I 2238 32 8 [...] [2213] .bss..compoundlit NOBITS ffffffffa3000000 1d85c000 00000000000000a0 0000000000000000 WA 0 0 32 [2214] .bss..compoundlit NOBITS ffffffffa30000a0 1d85c000 0000000000000040 0000000000000000 WA 0 0 32 [...]
While these extra sections don't typically cause any breakage, they do inflate the vmlinux size due to the overhead of storing metadata for thousands of extra sections.
It's also worth noting that for some reason, some downstream Android kernels can't boot at all if these sections aren't coalesced.
This issue isn't limited to any specific architecture; it affects arm64 and x86 if CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is forced on.
Example on x86 allyesconfig: Before: 2241 sections, 1170972 KiB After: 56 sections, 1171169 KiB
[1] https://github.com/llvm/llvm-project/commit/9e33c096476ab5e02ab1c8442cc3cb4e...
Link: https://github.com/ClangBuiltLinux/linux/issues/958 Cc: stable@vger.kernel.org # v4.4+ Suggested-by: Fangrui Song maskray@google.com Signed-off-by: Danny Lin danny@kdrag0n.dev --- include/asm-generic/vmlinux.lds.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index db600ef218d7..18968cba87c7 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -94,10 +94,10 @@ */ #ifdef CONFIG_LD_DEAD_CODE_DATA_ELIMINATION #define TEXT_MAIN .text .text.[0-9a-zA-Z_]* -#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..LPBX* +#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..LPBX* .data..compoundliteral* #define SDATA_MAIN .sdata .sdata.[0-9a-zA-Z_]* #define RODATA_MAIN .rodata .rodata.[0-9a-zA-Z_]* -#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* +#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* .bss..compoundliteral* #define SBSS_MAIN .sbss .sbss.[0-9a-zA-Z_]* #else #define TEXT_MAIN .text
On Thu, Jul 02, 2020 at 01:54:00AM -0700, Danny Lin wrote:
A recent LLVM 11 commit [1] made LLD stop implicitly coalescing some temporary LLVM sections, namely .{data,bss}..compoundliteral.XXX:
[30] .data..compoundli PROGBITS ffffffff9ac9a000 19e9a000 000000000000cea0 0000000000000000 WA 0 0 32 [31] .rela.data..compo RELA 0000000000000000 40965440 0000000000001d88 0000000000000018 I 2238 30 8 [32] .data..compoundli PROGBITS ffffffff9aca6ea0 19ea6ea0 00000000000033c0 0000000000000000 WA 0 0 32 [33] .rela.data..compo RELA 0000000000000000 409671c8 0000000000000948 0000000000000018 I 2238 32 8 [...] [2213] .bss..compoundlit NOBITS ffffffffa3000000 1d85c000 00000000000000a0 0000000000000000 WA 0 0 32 [2214] .bss..compoundlit NOBITS ffffffffa30000a0 1d85c000 0000000000000040 0000000000000000 WA 0 0 32 [...]
While these extra sections don't typically cause any breakage, they do inflate the vmlinux size due to the overhead of storing metadata for thousands of extra sections.
It's also worth noting that for some reason, some downstream Android kernels can't boot at all if these sections aren't coalesced.
This issue isn't limited to any specific architecture; it affects arm64 and x86 if CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is forced on.
Example on x86 allyesconfig: Before: 2241 sections, 1170972 KiB After: 56 sections, 1171169 KiB
[1] https://github.com/llvm/llvm-project/commit/9e33c096476ab5e02ab1c8442cc3cb4e...
Link: https://github.com/ClangBuiltLinux/linux/issues/958 Cc: stable@vger.kernel.org # v4.4+ Suggested-by: Fangrui Song maskray@google.com Signed-off-by: Danny Lin danny@kdrag0n.dev
include/asm-generic/vmlinux.lds.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index db600ef218d7..18968cba87c7 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -94,10 +94,10 @@ */ #ifdef CONFIG_LD_DEAD_CODE_DATA_ELIMINATION #define TEXT_MAIN .text .text.[0-9a-zA-Z_]* -#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..LPBX* +#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..LPBX* .data..compoundliteral* #define SDATA_MAIN .sdata .sdata.[0-9a-zA-Z_]* #define RODATA_MAIN .rodata .rodata.[0-9a-zA-Z_]* -#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* +#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* .bss..compoundliteral*
Are there .data.. and .bss.. sections we do NOT want to collect? i.e. why not include .data..* and .bss..* ?
On Thu, Jul 02, 2020 at 08:54:53AM -0700, Kees Cook wrote:
On Thu, Jul 02, 2020 at 01:54:00AM -0700, Danny Lin wrote:
A recent LLVM 11 commit [1] made LLD stop implicitly coalescing some temporary LLVM sections, namely .{data,bss}..compoundliteral.XXX:
[30] .data..compoundli PROGBITS ffffffff9ac9a000 19e9a000 000000000000cea0 0000000000000000 WA 0 0 32 [31] .rela.data..compo RELA 0000000000000000 40965440 0000000000001d88 0000000000000018 I 2238 30 8 [32] .data..compoundli PROGBITS ffffffff9aca6ea0 19ea6ea0 00000000000033c0 0000000000000000 WA 0 0 32 [33] .rela.data..compo RELA 0000000000000000 409671c8 0000000000000948 0000000000000018 I 2238 32 8 [...] [2213] .bss..compoundlit NOBITS ffffffffa3000000 1d85c000 00000000000000a0 0000000000000000 WA 0 0 32 [2214] .bss..compoundlit NOBITS ffffffffa30000a0 1d85c000 0000000000000040 0000000000000000 WA 0 0 32 [...]
While these extra sections don't typically cause any breakage, they do inflate the vmlinux size due to the overhead of storing metadata for thousands of extra sections.
It's also worth noting that for some reason, some downstream Android kernels can't boot at all if these sections aren't coalesced.
This issue isn't limited to any specific architecture; it affects arm64 and x86 if CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is forced on.
It might be worth noting that this happens explicitly because of -fdata-sections, which is currently only used with CONFIG_LD_DEAD_CODE_DATA_ELIMINATION but there are other features such as LTO that will enable this and make this relevant in the future.
https://android-review.googlesource.com/c/kernel/common/+/1329278/6#message-...
It is also worth noting that those commits add .bss..L* and .data..L* and rodata variants. Do you know if those are relevant here?
Example on x86 allyesconfig: Before: 2241 sections, 1170972 KiB After: 56 sections, 1171169 KiB
Am I reading this right that coalescing those sections increases the image size? Kind of interesting.
[1] https://github.com/llvm/llvm-project/commit/9e33c096476ab5e02ab1c8442cc3cb4e...
Link: https://github.com/ClangBuiltLinux/linux/issues/958 Cc: stable@vger.kernel.org # v4.4+ Suggested-by: Fangrui Song maskray@google.com Signed-off-by: Danny Lin danny@kdrag0n.dev
Reviewed-by: Nathan Chancellor natechancellor@gmail.com
include/asm-generic/vmlinux.lds.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index db600ef218d7..18968cba87c7 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -94,10 +94,10 @@ */ #ifdef CONFIG_LD_DEAD_CODE_DATA_ELIMINATION #define TEXT_MAIN .text .text.[0-9a-zA-Z_]* -#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..LPBX* +#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..LPBX* .data..compoundliteral*
I am fairly certain this will fix a PowerPC warning that we had recently so good!
https://lore.kernel.org/lkml/202006180904.TVUXCf6H%25lkp@intel.com/
Unfortunately, I forgot to reply to that thread...
#define SDATA_MAIN .sdata .sdata.[0-9a-zA-Z_]* #define RODATA_MAIN .rodata .rodata.[0-9a-zA-Z_]* -#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* +#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* .bss..compoundliteral*
Are there .data.. and .bss.. sections we do NOT want to collect? i.e. why not include .data..* and .bss..* ?
At one point Android was doing that for modules but stopped:
https://android-review.googlesource.com/c/kernel/common/+/1266787
I wonder if that is a problem for the main kernel image.
Cheers, Nathan
Jul 2, 2020 9:04:25 AM Nathan Chancellor natechancellor@gmail.com:
On Thu, Jul 02, 2020 at 08:54:53AM -0700, Kees Cook wrote:
On Thu, Jul 02, 2020 at 01:54:00AM -0700, Danny Lin wrote:
A recent LLVM 11 commit [1] made LLD stop implicitly coalescing some temporary LLVM sections, namely .{data,bss}..compoundliteral.XXX:
[30] .data..compoundli PROGBITS ffffffff9ac9a000 19e9a000 000000000000cea0 0000000000000000 WA 0 0 32 [31] .rela.data..compo RELA 0000000000000000 40965440 0000000000001d88 0000000000000018 I 2238 30 8 [32] .data..compoundli PROGBITS ffffffff9aca6ea0 19ea6ea0 00000000000033c0 0000000000000000 WA 0 0 32 [33] .rela.data..compo RELA 0000000000000000 409671c8 0000000000000948 0000000000000018 I 2238 32 8 [...] [2213] .bss..compoundlit NOBITS ffffffffa3000000 1d85c000 00000000000000a0 0000000000000000 WA 0 0 32 [2214] .bss..compoundlit NOBITS ffffffffa30000a0 1d85c000 0000000000000040 0000000000000000 WA 0 0 32 [...]
While these extra sections don't typically cause any breakage, they do inflate the vmlinux size due to the overhead of storing metadata for thousands of extra sections.
It's also worth noting that for some reason, some downstream Android kernels can't boot at all if these sections aren't coalesced.
This issue isn't limited to any specific architecture; it affects arm64 and x86 if CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is forced on.
It might be worth noting that this happens explicitly because of -fdata-sections, which is currently only used with CONFIG_LD_DEAD_CODE_DATA_ELIMINATION but there are other features such as LTO that will enable this and make this relevant in the future.
https://android-review.googlesource.com/c/kernel/common/+/1329278/6#message-...
It is also worth noting that those commits add .bss..L* and .data..L* and rodata variants. Do you know if those are relevant here?
As far as I can tell, those sections are exclusive to LTO which isn't in mainline yet. I don't see any sections like that in my DCE-only vmlinux.
Example on x86 allyesconfig: Before: 2241 sections, 1170972 KiB After: 56 sections, 1171169 KiB
Am I reading this right that coalescing those sections increases the image size? Kind of interesting.
Oops, I accidentally swapped the numbers in the commit message. Coalescing the sections makes the image smaller as expected.
[1] https://github.com/llvm/llvm-project/commit/9e33c096476ab5e02ab1c8442cc3cb4e...
Link: https://github.com/ClangBuiltLinux/linux/issues/958 Cc: stable@vger.kernel.org # v4.4+ Suggested-by: Fangrui Song maskray@google.com Signed-off-by: Danny Lin danny@kdrag0n.dev
Reviewed-by: Nathan Chancellor natechancellor@gmail.com
include/asm-generic/vmlinux.lds.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index db600ef218d7..18968cba87c7 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -94,10 +94,10 @@ */ #ifdef CONFIG_LD_DEAD_CODE_DATA_ELIMINATION #define TEXT_MAIN .text .text.[0-9a-zA-Z_]* -#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..LPBX* +#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..LPBX* .data..compoundliteral*
I am fairly certain this will fix a PowerPC warning that we had recently so good!
https://lore.kernel.org/lkml/202006180904.TVUXCf6H%25lkp@intel.com/
Unfortunately, I forgot to reply to that thread...
#define SDATA_MAIN .sdata .sdata.[0-9a-zA-Z_]* #define RODATA_MAIN .rodata .rodata.[0-9a-zA-Z_]* -#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* +#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* .bss..compoundliteral*
Are there .data.. and .bss.. sections we do NOT want to collect? i.e. why not include .data..* and .bss..* ?
At one point Android was doing that for modules but stopped:
https://android-review.googlesource.com/c/kernel/common/+/1266787
I wonder if that is a problem for the main kernel image.
A comment above the code in question explicitly states that not all .data..* sections should be coalesced. There's a .data..percpu section in my x86 vmlinux which should probably remain separate.
Cheers, Nathan
A recent LLVM 11 commit [1] made LLD stop implicitly coalescing some temporary LLVM sections, namely .{data,bss}..compoundliteral.XXX:
[30] .data..compoundli PROGBITS ffffffff9ac9a000 19e9a000 000000000000cea0 0000000000000000 WA 0 0 32 [31] .rela.data..compo RELA 0000000000000000 40965440 0000000000001d88 0000000000000018 I 2238 30 8 [32] .data..compoundli PROGBITS ffffffff9aca6ea0 19ea6ea0 00000000000033c0 0000000000000000 WA 0 0 32 [33] .rela.data..compo RELA 0000000000000000 409671c8 0000000000000948 0000000000000018 I 2238 32 8 [...] [2213] .bss..compoundlit NOBITS ffffffffa3000000 1d85c000 00000000000000a0 0000000000000000 WA 0 0 32 [2214] .bss..compoundlit NOBITS ffffffffa30000a0 1d85c000 0000000000000040 0000000000000000 WA 0 0 32 [...]
.{data,bss}..L<symbol name> sections are also created in some cases. While there aren't any in this example, they should also be coalesced to be safe in case some config or future LLVM change makes it start creating more of those sections in the future. For example, enabling global merging causes ..L_MergedGlobals sections to be created, but it's likely that other changes will result in such sections as well.
While these extra sections don't typically cause any breakage, they do inflate the vmlinux size due to the overhead of storing metadata for thousands of extra sections.
It's also worth noting that for some reason, some downstream Android kernels can't boot at all if these sections aren't coalesced.
This issue isn't limited to any specific architecture; it affects arm64 and x86 if CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is forced on.
Example on x86 allyesconfig: Before: 2241 sections, 1171169 KiB After: 56 sections, 1170972 KiB
[1] https://github.com/llvm/llvm-project/commit/9e33c096476ab5e02ab1c8442cc3cb4e...
Link: https://github.com/ClangBuiltLinux/linux/issues/958 Cc: stable@vger.kernel.org # v4.4+ Suggested-by: Fangrui Song maskray@google.com Signed-off-by: Danny Lin danny@kdrag0n.dev Reviewed-by: Nathan Chancellor natechancellor@gmail.com --- v2: - Fixed swapped example sizes - Added .{data,bss}..L* sections, since it looks like they're emitted in some cases even when LTO is disabled
include/asm-generic/vmlinux.lds.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index db600ef218d7..737ecf782229 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -94,10 +94,11 @@ */ #ifdef CONFIG_LD_DEAD_CODE_DATA_ELIMINATION #define TEXT_MAIN .text .text.[0-9a-zA-Z_]* -#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..LPBX* +#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..LPBX* \ + .data..compoundliteral* .data..L* #define SDATA_MAIN .sdata .sdata.[0-9a-zA-Z_]* #define RODATA_MAIN .rodata .rodata.[0-9a-zA-Z_]* -#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* +#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* .bss..compoundliteral* .bss..L* #define SBSS_MAIN .sbss .sbss.[0-9a-zA-Z_]* #else #define TEXT_MAIN .text
Hi
[This is an automated email]
This commit has been processed because it contains a -stable tag. The stable tag indicates that it's relevant for the following trees: 4.4+
The bot has tested the following trees: v5.7.7, v5.4.50, v4.19.131, v4.14.187, v4.9.229, v4.4.229.
v5.7.7: Build OK! v5.4.50: Build OK! v4.19.131: Build OK! v4.14.187: Failed to apply! Possible dependencies: 266ff2a8f51f0 ("kbuild: Fix asm-generic/vmlinux.lds.h for LD_DEAD_CODE_DATA_ELIMINATION") 4bebdc7a85aa4 ("bpf: add helper bpf_perf_prog_read_value") 52c8ee5bad8f3 ("vmlinux.lds.h: Fix linker warnings about orphan .LPBX sections") 540adea3809f6 ("error-injection: Separate error-injection from kprobe") 663faf9f7beea ("error-injection: Add injectable error types") 908432ca84fc2 ("bpf: add helper bpf_perf_event_read_value for perf event array map") 97562633bcbac ("bpf: perf event change needed for subsequent bpf helpers") 9802d86585db9 ("bpf: add a bpf_override_function helper") a621438500533 ("vmlinux.lds.h: remove no-op macro VMLINUX_SYMBOL()") cd86d1fd21025 ("bpf: Adding helper function bpf_getsockops") dd0bb688eaa24 ("bpf: add a bpf_override_function helper") de8f3a83b0a0f ("bpf: add meta pointer for direct access") f3edacbd697f9 ("bpf: Revert bpf_overrid_function() helper changes.")
v4.9.229: Failed to apply! Possible dependencies: 17bedab272314 ("bpf: xdp: Allow head adjustment in XDP prog") 1a1d74d378b13 ("nfp: use AND instead of modulo to get ring indexes") 23a4e389bdc71 ("nfp: create separate define for max number of vectors") 266ff2a8f51f0 ("kbuild: Fix asm-generic/vmlinux.lds.h for LD_DEAD_CODE_DATA_ELIMINATION") 416db5c1e4488 ("nfp: remove support for nfp3200") 4b89b7f7aad57 ("kbuild: keep data tables through dead code elimination") 52c8ee5bad8f3 ("vmlinux.lds.h: Fix linker warnings about orphan .LPBX sections") 540adea3809f6 ("error-injection: Separate error-injection from kprobe") 663faf9f7beea ("error-injection: Add injectable error types") 67f8b1dcb9ee7 ("net/mlx4_en: Refactor the XDP forwarding rings scheme") 68453c7a89733 ("nfp: centralize runtime reconfiguration logic") 7ff5c83a1deb0 ("nfp: simplify nfp_net_poll()") 9802d86585db9 ("bpf: add a bpf_override_function helper") a4b562bb8ebd4 ("nfp: use unsigned int for vector/ring counts") a621438500533 ("vmlinux.lds.h: remove no-op macro VMLINUX_SYMBOL()") bf187ea01b077 ("nfp: centralize the buffer size calculation") cbeaf7aa733a1 ("nfp: bring back support for different ring counts") ccc109b8ed24c ("net/mlx4_en: Add TX_XDP for CQ types") dd0bb688eaa24 ("bpf: add a bpf_override_function helper") e390b55d5aefe ("bpf: make bpf_xdp_adjust_head support mandatory") ecd63a0217d5f ("nfp: add XDP support in the driver") f3edacbd697f9 ("bpf: Revert bpf_overrid_function() helper changes.")
v4.4.229: Failed to apply! Possible dependencies: 0f4c4af06eec5 ("kbuild: -ffunction-sections fix for archs with conflicting sections") 266ff2a8f51f0 ("kbuild: Fix asm-generic/vmlinux.lds.h for LD_DEAD_CODE_DATA_ELIMINATION") 52c8ee5bad8f3 ("vmlinux.lds.h: Fix linker warnings about orphan .LPBX sections") a5967db9af51a ("kbuild: allow architectures to use thin archives instead of ld -r") b67067f1176df ("kbuild: allow archs to select link dead code/data elimination") cb87481ee89db ("kbuild: linker script do not match C names unless LD_DEAD_CODE_DATA_ELIMINATION is configured") f235541699bcf ("export.h: allow for per-symbol configurable EXPORT_SYMBOL()")
NOTE: The patch will not be queued to stable trees until it is upstream.
How should we proceed with this patch?
linux-stable-mirror@lists.linaro.org
-
Danny Lin -
Kees Cook -
Nathan Chancellor -
Sasha Levin