Hi,
please pick up the following patches for 5.4.
Those are build time optimizations for kernel/gen_kheaders.sh, and - by removing bashisms - dropping the dependency to /bin/bash.
In addition, this enables build time improvements across the tree by optionally allowing to use alternative implementations for various compression tools, e.g. GZIP=pigz.
The documentation-only change is not strictly necessary, but keeps kernel/gen_kheaders.sh in sync with mainline.
Cheers, Matthias
Cc: Denis Efremov efremov@linux.com Cc: Masahiro Yamada yamada.masahiro@socionext.com Cc: Greg Kroah-Hartman gregkh@linuxfoundation.org
Denis Efremov (1): kbuild: add variables for compression tools
Masahiro Yamada (5): kheaders: remove unneeded 'cat' command piped to 'head' / 'tail' kheaders: optimize md5sum calculation for in-tree builds kheaders: optimize header copy for in-tree builds kheaders: remove the last bashism to allow sh to run it kheaders: explain why include/config/autoconf.h is excluded from md5sum
Makefile | 25 +++++++++++- arch/arm/boot/deflate_xip_data.sh | 2 +- arch/ia64/Makefile | 2 +- arch/m68k/Makefile | 8 ++-- arch/parisc/Makefile | 2 +- kernel/Makefile | 2 +- kernel/gen_kheaders.sh | 66 ++++++++++++++++++------------- scripts/Makefile.lib | 12 +++--- scripts/Makefile.package | 8 ++-- scripts/package/buildtar | 6 +-- scripts/xz_wrap.sh | 2 +- 11 files changed, 83 insertions(+), 52 deletions(-)
From: Masahiro Yamada yamada.masahiro@socionext.com
The 'head' and 'tail' commands can take a file path directly. So, you do not need to run 'cat'.
cat kernel/kheaders.md5 | head -1
... is equivalent to:
head -1 kernel/kheaders.md5
and the latter saves forking one process.
While I was here, I replaced 'head -1' with 'head -n 1'.
I also replaced '==' with '=' since we do not have a good reason to use the bashism.
Signed-off-by: Masahiro Yamada yamada.masahiro@socionext.com (cherry picked from commit 9a066357184485784f782719093ff804d05b85db) Signed-off-by: Matthias Maennich maennich@google.com --- kernel/gen_kheaders.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh index 5a0fc0b0403a..b8054b0d5010 100755 --- a/kernel/gen_kheaders.sh +++ b/kernel/gen_kheaders.sh @@ -41,10 +41,10 @@ obj_files_md5="$(find $dir_list -name "*.h" | this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)" if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi if [ -f kernel/kheaders.md5 ] && - [ "$(cat kernel/kheaders.md5|head -1)" == "$src_files_md5" ] && - [ "$(cat kernel/kheaders.md5|head -2|tail -1)" == "$obj_files_md5" ] && - [ "$(cat kernel/kheaders.md5|head -3|tail -1)" == "$this_file_md5" ] && - [ "$(cat kernel/kheaders.md5|tail -1)" == "$tarfile_md5" ]; then + [ "$(head -n 1 kernel/kheaders.md5)" = "$src_files_md5" ] && + [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$obj_files_md5" ] && + [ "$(head -n 3 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] && + [ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then exit fi
From: Masahiro Yamada yamada.masahiro@socionext.com
This script computes md5sum of headers in srctree and in objtree. However, when we are building in-tree, we know the srctree and the objtree are the same. That is, we end up with the same computation twice. In fact, the first two lines of kernel/kheaders.md5 are always the same for in-tree builds.
Unify the two md5sum calculations.
For in-tree builds ($building_out_of_srctree is empty), we check only two directories, "include", and "arch/$SRCARCH/include".
For out-of-tree builds ($building_out_of_srctree is 1), we check 4 directories, "$srctree/include", "$srctree/arch/$SRCARCH/include", "include", and "arch/$SRCARCH/include" since we know they are all different.
Signed-off-by: Masahiro Yamada yamada.masahiro@socionext.com (cherry picked from commit 0e11773e76098729552b750ccff79374d1e62002) Signed-off-by: Matthias Maennich maennich@google.com --- kernel/gen_kheaders.sh | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)
diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh index b8054b0d5010..6ff86e62787f 100755 --- a/kernel/gen_kheaders.sh +++ b/kernel/gen_kheaders.sh @@ -21,29 +21,30 @@ arch/$SRCARCH/include/ # Uncomment it for debugging. # if [ ! -f /tmp/iter ]; then iter=1; echo 1 > /tmp/iter; # else iter=$(($(cat /tmp/iter) + 1)); echo $iter > /tmp/iter; fi -# find $src_file_list -name "*.h" | xargs ls -l > /tmp/src-ls-$iter -# find $obj_file_list -name "*.h" | xargs ls -l > /tmp/obj-ls-$iter +# find $all_dirs -name "*.h" | xargs ls -l > /tmp/ls-$iter + +all_dirs= +if [ "$building_out_of_srctree" ]; then + for d in $dir_list; do + all_dirs="$all_dirs $srctree/$d" + done +fi +all_dirs="$all_dirs $dir_list"
# include/generated/compile.h is ignored because it is touched even when none # of the source files changed. This causes pointless regeneration, so let us # ignore them for md5 calculation. -pushd $srctree > /dev/null -src_files_md5="$(find $dir_list -name "*.h" | - grep -v "include/generated/compile.h" | - grep -v "include/generated/autoconf.h" | - xargs ls -l | md5sum | cut -d ' ' -f1)" -popd > /dev/null -obj_files_md5="$(find $dir_list -name "*.h" | - grep -v "include/generated/compile.h" | - grep -v "include/generated/autoconf.h" | +headers_md5="$(find $all_dirs -name "*.h" | + grep -v "include/generated/compile.h" | + grep -v "include/generated/autoconf.h" | xargs ls -l | md5sum | cut -d ' ' -f1)" + # Any changes to this script will also cause a rebuild of the archive. this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)" if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi if [ -f kernel/kheaders.md5 ] && - [ "$(head -n 1 kernel/kheaders.md5)" = "$src_files_md5" ] && - [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$obj_files_md5" ] && - [ "$(head -n 3 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] && + [ "$(head -n 1 kernel/kheaders.md5)" = "$headers_md5" ] && + [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] && [ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then exit fi @@ -79,8 +80,7 @@ find $cpio_dir -printf "./%P\n" | LC_ALL=C sort | \ --owner=0 --group=0 --numeric-owner --no-recursion \ -Jcf $tarfile -C $cpio_dir/ -T - > /dev/null
-echo "$src_files_md5" > kernel/kheaders.md5 -echo "$obj_files_md5" >> kernel/kheaders.md5 +echo $headers_md5 > kernel/kheaders.md5 echo "$this_file_md5" >> kernel/kheaders.md5 echo "$(md5sum $tarfile | cut -d ' ' -f1)" >> kernel/kheaders.md5
From: Masahiro Yamada yamada.masahiro@socionext.com
This script copies headers by the cpio command twice; first from srctree, and then from objtree. However, when we building in-tree, we know the srctree and the objtree are the same. That is, all the headers copied by the first cpio are overwritten by the second one.
Skip the first cpio when we are building in-tree.
Signed-off-by: Masahiro Yamada yamada.masahiro@socionext.com (cherry picked from commit ea79e5168be644fdaf7d4e6a73eceaf07b3da76a) Signed-off-by: Matthias Maennich maennich@google.com --- kernel/gen_kheaders.sh | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)
diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh index 6ff86e62787f..0f7752dd93a6 100755 --- a/kernel/gen_kheaders.sh +++ b/kernel/gen_kheaders.sh @@ -56,14 +56,16 @@ fi rm -rf $cpio_dir mkdir $cpio_dir
-pushd $srctree > /dev/null -for f in $dir_list; - do find "$f" -name "*.h"; -done | cpio --quiet -pd $cpio_dir -popd > /dev/null +if [ "$building_out_of_srctree" ]; then + pushd $srctree > /dev/null + for f in $dir_list + do find "$f" -name "*.h"; + done | cpio --quiet -pd $cpio_dir + popd > /dev/null +fi
-# The second CPIO can complain if files already exist which can -# happen with out of tree builds. Just silence CPIO for now. +# The second CPIO can complain if files already exist which can happen with out +# of tree builds having stale headers in srctree. Just silence CPIO for now. for f in $dir_list; do find "$f" -name "*.h"; done | cpio --quiet -pd $cpio_dir >/dev/null 2>&1
From: Masahiro Yamada yamada.masahiro@socionext.com
'pushd' ... 'popd' is the last bash-specific code in this script. One way to avoid it is to run the code in a sub-shell.
With that addressed, you can run this script with sh.
I replaced $(BASH) with $(CONFIG_SHELL), and I changed the hashbang to #!/bin/sh.
Signed-off-by: Masahiro Yamada yamada.masahiro@socionext.com (cherry picked from commit 1463f74f492eea7191f0178e01f3d38371a48210) Signed-off-by: Matthias Maennich maennich@google.com --- kernel/Makefile | 2 +- kernel/gen_kheaders.sh | 13 +++++++------ 2 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/kernel/Makefile b/kernel/Makefile index daad787fb795..42557f251fea 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -128,7 +128,7 @@ $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
quiet_cmd_genikh = CHK $(obj)/kheaders_data.tar.xz - cmd_genikh = $(BASH) $(srctree)/kernel/gen_kheaders.sh $@ + cmd_genikh = $(CONFIG_SHELL) $(srctree)/kernel/gen_kheaders.sh $@ $(obj)/kheaders_data.tar.xz: FORCE $(call cmd,genikh)
diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh index 0f7752dd93a6..dc5744b93f8c 100755 --- a/kernel/gen_kheaders.sh +++ b/kernel/gen_kheaders.sh @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh # SPDX-License-Identifier: GPL-2.0
# This script generates an archive consisting of kernel headers @@ -57,11 +57,12 @@ rm -rf $cpio_dir mkdir $cpio_dir
if [ "$building_out_of_srctree" ]; then - pushd $srctree > /dev/null - for f in $dir_list - do find "$f" -name "*.h"; - done | cpio --quiet -pd $cpio_dir - popd > /dev/null + ( + cd $srctree + for f in $dir_list + do find "$f" -name "*.h"; + done | cpio --quiet -pd $cpio_dir + ) fi
# The second CPIO can complain if files already exist which can happen with out
From: Masahiro Yamada yamada.masahiro@socionext.com
This comment block explains why include/generated/compile.h is omitted, but nothing about include/generated/autoconf.h, which might be more difficult to understand. Add more comments.
Signed-off-by: Masahiro Yamada yamada.masahiro@socionext.com (cherry picked from commit f276031b4e2f4c961ed6d8a42f0f0124ccac2e09) Signed-off-by: Matthias Maennich maennich@google.com --- kernel/gen_kheaders.sh | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh index dc5744b93f8c..e13ca842eb7e 100755 --- a/kernel/gen_kheaders.sh +++ b/kernel/gen_kheaders.sh @@ -32,8 +32,15 @@ fi all_dirs="$all_dirs $dir_list"
# include/generated/compile.h is ignored because it is touched even when none -# of the source files changed. This causes pointless regeneration, so let us -# ignore them for md5 calculation. +# of the source files changed. +# +# When Kconfig regenerates include/generated/autoconf.h, its timestamp is +# updated, but the contents might be still the same. When any CONFIG option is +# changed, Kconfig touches the corresponding timestamp file include/config/*.h. +# Hence, the md5sum detects the configuration change anyway. We do not need to +# check include/generated/autoconf.h explicitly. +# +# Ignore them for md5 calculation to avoid pointless regeneration. headers_md5="$(find $all_dirs -name "*.h" | grep -v "include/generated/compile.h" | grep -v "include/generated/autoconf.h" |
From: Denis Efremov efremov@linux.com
Allow user to use alternative implementations of compression tools, such as pigz, pbzip2, pxz. For example, multi-threaded tools to speed up the build: $ make GZIP=pigz BZIP2=pbzip2
Variables _GZIP, _BZIP2, _LZOP are used internally because original env vars are reserved by the tools. The use of GZIP in gzip tool is obsolete since 2015. However, alternative implementations (e.g., pigz) still rely on it. BZIP2, BZIP, LZOP vars are not obsolescent.
The credit goes to @grsecurity.
As a sidenote, for multi-threaded lzma, xz compression one can use: $ export XZ_OPT="--threads=0"
Signed-off-by: Denis Efremov efremov@linux.com Signed-off-by: Masahiro Yamada masahiroy@kernel.org (cherry picked from commit 8dfb61dcbaceb19a5ded5e9c9dcf8d05acc32294) Signed-off-by: Matthias Maennich maennich@google.com Change-Id: Idaa5daac69c820c78e32b4630f3d72739e74be12 --- Makefile | 25 +++++++++++++++++++++++-- arch/arm/boot/deflate_xip_data.sh | 2 +- arch/ia64/Makefile | 2 +- arch/m68k/Makefile | 8 ++++---- arch/parisc/Makefile | 2 +- kernel/gen_kheaders.sh | 2 +- scripts/Makefile.lib | 12 ++++++------ scripts/Makefile.package | 8 ++++---- scripts/package/buildtar | 6 +++--- scripts/xz_wrap.sh | 2 +- 10 files changed, 45 insertions(+), 24 deletions(-)
diff --git a/Makefile b/Makefile index 7c001e21e28e..3521ba6097a7 100644 --- a/Makefile +++ b/Makefile @@ -426,6 +426,26 @@ PYTHON2 = python2 PYTHON3 = python3 CHECK = sparse BASH = bash +GZIP = gzip +BZIP2 = bzip2 +LZOP = lzop +LZMA = lzma +LZ4 = lz4c +XZ = xz + +# GZIP, BZIP2, LZOP env vars are used by the tools. Support them as the command +# line interface, but use _GZIP, _BZIP2, _LZOP internally. +_GZIP := $(GZIP) +_BZIP2 := $(BZIP2) +_LZOP := $(LZOP) + +# Reset GZIP, BZIP2, LZOP in this Makefile +override GZIP= +override BZIP2= +override LZOP= + +# Reset GZIP, BZIP2, LZOP in recursive invocations +MAKEOVERRIDES += GZIP= BZIP2= LZOP=
CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF) @@ -474,6 +494,7 @@ CLANG_FLAGS := export ARCH SRCARCH CONFIG_SHELL BASH HOSTCC KBUILD_HOSTCFLAGS CROSS_COMPILE AS LD CC export CPP AR NM STRIP OBJCOPY OBJDUMP OBJSIZE PAHOLE LEX YACC AWK INSTALLKERNEL export PERL PYTHON PYTHON2 PYTHON3 CHECK CHECKFLAGS MAKE UTS_MACHINE HOSTCXX +export _GZIP _BZIP2 _LZOP LZMA LZ4 XZ export KBUILD_HOSTCXXFLAGS KBUILD_HOSTLDFLAGS KBUILD_HOSTLDLIBS LDFLAGS_MODULE
export KBUILD_CPPFLAGS NOSTDINC_FLAGS LINUXINCLUDE OBJCOPYFLAGS KBUILD_LDFLAGS @@ -981,10 +1002,10 @@ export mod_strip_cmd mod_compress_cmd = true ifdef CONFIG_MODULE_COMPRESS ifdef CONFIG_MODULE_COMPRESS_GZIP - mod_compress_cmd = gzip -n -f + mod_compress_cmd = $(_GZIP) -n -f endif # CONFIG_MODULE_COMPRESS_GZIP ifdef CONFIG_MODULE_COMPRESS_XZ - mod_compress_cmd = xz -f + mod_compress_cmd = $(XZ) -f endif # CONFIG_MODULE_COMPRESS_XZ endif # CONFIG_MODULE_COMPRESS export mod_compress_cmd diff --git a/arch/arm/boot/deflate_xip_data.sh b/arch/arm/boot/deflate_xip_data.sh index 40937248cebe..739f0464321e 100755 --- a/arch/arm/boot/deflate_xip_data.sh +++ b/arch/arm/boot/deflate_xip_data.sh @@ -56,7 +56,7 @@ trap 'rm -f "$XIPIMAGE.tmp"; exit 1' 1 2 3 # substitute the data section by a compressed version $DD if="$XIPIMAGE" count=$data_start iflag=count_bytes of="$XIPIMAGE.tmp" $DD if="$XIPIMAGE" skip=$data_start iflag=skip_bytes | -gzip -9 >> "$XIPIMAGE.tmp" +$_GZIP -9 >> "$XIPIMAGE.tmp"
# replace kernel binary mv -f "$XIPIMAGE.tmp" "$XIPIMAGE" diff --git a/arch/ia64/Makefile b/arch/ia64/Makefile index 32240000dc0c..f817f3d5e758 100644 --- a/arch/ia64/Makefile +++ b/arch/ia64/Makefile @@ -40,7 +40,7 @@ $(error Sorry, you need a newer version of the assember, one that is built from endif
quiet_cmd_gzip = GZIP $@ -cmd_gzip = cat $(real-prereqs) | gzip -n -f -9 > $@ +cmd_gzip = cat $(real-prereqs) | $(_GZIP) -n -f -9 > $@
quiet_cmd_objcopy = OBJCOPY $@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@ diff --git a/arch/m68k/Makefile b/arch/m68k/Makefile index 5d9288384096..ce6db5e5a5a3 100644 --- a/arch/m68k/Makefile +++ b/arch/m68k/Makefile @@ -135,10 +135,10 @@ vmlinux.gz: vmlinux ifndef CONFIG_KGDB cp vmlinux vmlinux.tmp $(STRIP) vmlinux.tmp - gzip -9c vmlinux.tmp >vmlinux.gz + $(_GZIP) -9c vmlinux.tmp >vmlinux.gz rm vmlinux.tmp else - gzip -9c vmlinux >vmlinux.gz + $(_GZIP) -9c vmlinux >vmlinux.gz endif
bzImage: vmlinux.bz2 @@ -148,10 +148,10 @@ vmlinux.bz2: vmlinux ifndef CONFIG_KGDB cp vmlinux vmlinux.tmp $(STRIP) vmlinux.tmp - bzip2 -1c vmlinux.tmp >vmlinux.bz2 + $(_BZIP2) -1c vmlinux.tmp >vmlinux.bz2 rm vmlinux.tmp else - bzip2 -1c vmlinux >vmlinux.bz2 + $(_BZIP2) -1c vmlinux >vmlinux.bz2 endif
archclean: diff --git a/arch/parisc/Makefile b/arch/parisc/Makefile index 36b834f1c933..ad4ba8b5d1e4 100644 --- a/arch/parisc/Makefile +++ b/arch/parisc/Makefile @@ -156,7 +156,7 @@ vmlinuz: bzImage $(OBJCOPY) $(boot)/bzImage $@ else vmlinuz: vmlinux - @gzip -cf -9 $< > $@ + @$(_GZIP) -cf -9 $< > $@ endif
install: diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh index e13ca842eb7e..c1510f0ab3ea 100755 --- a/kernel/gen_kheaders.sh +++ b/kernel/gen_kheaders.sh @@ -88,7 +88,7 @@ find $cpio_dir -type f -print0 | find $cpio_dir -printf "./%P\n" | LC_ALL=C sort | \ tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \ --owner=0 --group=0 --numeric-owner --no-recursion \ - -Jcf $tarfile -C $cpio_dir/ -T - > /dev/null + -I $XZ -cf $tarfile -C $cpio_dir/ -T - > /dev/null
echo $headers_md5 > kernel/kheaders.md5 echo "$this_file_md5" >> kernel/kheaders.md5 diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib index 342618a2bccb..75d3684aceec 100644 --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -230,7 +230,7 @@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@ # ---------------------------------------------------------------------------
quiet_cmd_gzip = GZIP $@ - cmd_gzip = cat $(real-prereqs) | gzip -n -f -9 > $@ + cmd_gzip = cat $(real-prereqs) | $(_GZIP) -n -f -9 > $@
# DTC # --------------------------------------------------------------------------- @@ -322,19 +322,19 @@ printf "%08x\n" $$dec_size | \ )
quiet_cmd_bzip2 = BZIP2 $@ - cmd_bzip2 = { cat $(real-prereqs) | bzip2 -9; $(size_append); } > $@ + cmd_bzip2 = { cat $(real-prereqs) | $(_BZIP2) -9; $(size_append); } > $@
# Lzma # ---------------------------------------------------------------------------
quiet_cmd_lzma = LZMA $@ - cmd_lzma = { cat $(real-prereqs) | lzma -9; $(size_append); } > $@ + cmd_lzma = { cat $(real-prereqs) | $(LZMA) -9; $(size_append); } > $@
quiet_cmd_lzo = LZO $@ - cmd_lzo = { cat $(real-prereqs) | lzop -9; $(size_append); } > $@ + cmd_lzo = { cat $(real-prereqs) | $(_LZOP) -9; $(size_append); } > $@
quiet_cmd_lz4 = LZ4 $@ - cmd_lz4 = { cat $(real-prereqs) | lz4c -l -c1 stdin stdout; \ + cmd_lz4 = { cat $(real-prereqs) | $(LZ4) -l -c1 stdin stdout; \ $(size_append); } > $@
# U-Boot mkimage @@ -381,7 +381,7 @@ quiet_cmd_xzkern = XZKERN $@ $(size_append); } > $@
quiet_cmd_xzmisc = XZMISC $@ - cmd_xzmisc = cat $(real-prereqs) | xz --check=crc32 --lzma2=dict=1MiB > $@ + cmd_xzmisc = cat $(real-prereqs) | $(XZ) --check=crc32 --lzma2=dict=1MiB > $@
# ASM offsets # --------------------------------------------------------------------------- diff --git a/scripts/Makefile.package b/scripts/Makefile.package index 56eadcc48d46..4ea20e6b7e2b 100644 --- a/scripts/Makefile.package +++ b/scripts/Makefile.package @@ -45,7 +45,7 @@ if test "$(objtree)" != "$(srctree)"; then \ false; \ fi ; \ $(srctree)/scripts/setlocalversion --save-scmversion; \ -tar -cz $(RCS_TAR_IGNORE) -f $(2).tar.gz \ +tar -I $(_GZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \ --transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3); \ rm -f $(objtree)/.scmversion
@@ -127,9 +127,9 @@ util/PERF-VERSION-GEN $(CURDIR)/$(perf-tar)/); \ tar rf $(perf-tar).tar $(perf-tar)/HEAD $(perf-tar)/PERF-VERSION-FILE; \ rm -r $(perf-tar); \ $(if $(findstring tar-src,$@),, \ -$(if $(findstring bz2,$@),bzip2, \ -$(if $(findstring gz,$@),gzip, \ -$(if $(findstring xz,$@),xz, \ +$(if $(findstring bz2,$@),$(_BZIP2), \ +$(if $(findstring gz,$@),$(_GZIP), \ +$(if $(findstring xz,$@),$(XZ), \ $(error unknown target $@)))) \ -f -9 $(perf-tar).tar)
diff --git a/scripts/package/buildtar b/scripts/package/buildtar index 2f66c81e4021..4ae3698cd84a 100755 --- a/scripts/package/buildtar +++ b/scripts/package/buildtar @@ -28,15 +28,15 @@ case "${1}" in opts= ;; targz-pkg) - opts=--gzip + opts="-I ${_GZIP}" tarball=${tarball}.gz ;; tarbz2-pkg) - opts=--bzip2 + opts="-I ${_BZIP2}" tarball=${tarball}.bz2 ;; tarxz-pkg) - opts=--xz + opts="-I ${XZ}" tarball=${tarball}.xz ;; *) diff --git a/scripts/xz_wrap.sh b/scripts/xz_wrap.sh index 7a2d372f4885..76e9cbcfbeab 100755 --- a/scripts/xz_wrap.sh +++ b/scripts/xz_wrap.sh @@ -20,4 +20,4 @@ case $SRCARCH in sparc) BCJ=--sparc ;; esac
-exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB +exec $XZ --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB
Hi,
This patch introduces build error, see fix: e4a42c82e943 kbuild: fix broken builds because of GZIP,BZIP2,LZOP variables
Thanks, Denis
On 8/26/20 7:28 PM, Matthias Maennich wrote:
From: Denis Efremov efremov@linux.com
Allow user to use alternative implementations of compression tools, such as pigz, pbzip2, pxz. For example, multi-threaded tools to speed up the build: $ make GZIP=pigz BZIP2=pbzip2
Variables _GZIP, _BZIP2, _LZOP are used internally because original env vars are reserved by the tools. The use of GZIP in gzip tool is obsolete since 2015. However, alternative implementations (e.g., pigz) still rely on it. BZIP2, BZIP, LZOP vars are not obsolescent.
The credit goes to @grsecurity.
As a sidenote, for multi-threaded lzma, xz compression one can use: $ export XZ_OPT="--threads=0"
Signed-off-by: Denis Efremov efremov@linux.com Signed-off-by: Masahiro Yamada masahiroy@kernel.org (cherry picked from commit 8dfb61dcbaceb19a5ded5e9c9dcf8d05acc32294) Signed-off-by: Matthias Maennich maennich@google.com Change-Id: Idaa5daac69c820c78e32b4630f3d72739e74be12
Makefile | 25 +++++++++++++++++++++++-- arch/arm/boot/deflate_xip_data.sh | 2 +- arch/ia64/Makefile | 2 +- arch/m68k/Makefile | 8 ++++---- arch/parisc/Makefile | 2 +- kernel/gen_kheaders.sh | 2 +- scripts/Makefile.lib | 12 ++++++------ scripts/Makefile.package | 8 ++++---- scripts/package/buildtar | 6 +++--- scripts/xz_wrap.sh | 2 +- 10 files changed, 45 insertions(+), 24 deletions(-)
diff --git a/Makefile b/Makefile index 7c001e21e28e..3521ba6097a7 100644 --- a/Makefile +++ b/Makefile @@ -426,6 +426,26 @@ PYTHON2 = python2 PYTHON3 = python3 CHECK = sparse BASH = bash +GZIP = gzip +BZIP2 = bzip2 +LZOP = lzop +LZMA = lzma +LZ4 = lz4c +XZ = xz
+# GZIP, BZIP2, LZOP env vars are used by the tools. Support them as the command +# line interface, but use _GZIP, _BZIP2, _LZOP internally. +_GZIP := $(GZIP) +_BZIP2 := $(BZIP2) +_LZOP := $(LZOP)
+# Reset GZIP, BZIP2, LZOP in this Makefile +override GZIP= +override BZIP2= +override LZOP=
+# Reset GZIP, BZIP2, LZOP in recursive invocations +MAKEOVERRIDES += GZIP= BZIP2= LZOP= CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF) @@ -474,6 +494,7 @@ CLANG_FLAGS := export ARCH SRCARCH CONFIG_SHELL BASH HOSTCC KBUILD_HOSTCFLAGS CROSS_COMPILE AS LD CC export CPP AR NM STRIP OBJCOPY OBJDUMP OBJSIZE PAHOLE LEX YACC AWK INSTALLKERNEL export PERL PYTHON PYTHON2 PYTHON3 CHECK CHECKFLAGS MAKE UTS_MACHINE HOSTCXX +export _GZIP _BZIP2 _LZOP LZMA LZ4 XZ export KBUILD_HOSTCXXFLAGS KBUILD_HOSTLDFLAGS KBUILD_HOSTLDLIBS LDFLAGS_MODULE export KBUILD_CPPFLAGS NOSTDINC_FLAGS LINUXINCLUDE OBJCOPYFLAGS KBUILD_LDFLAGS @@ -981,10 +1002,10 @@ export mod_strip_cmd mod_compress_cmd = true ifdef CONFIG_MODULE_COMPRESS ifdef CONFIG_MODULE_COMPRESS_GZIP
- mod_compress_cmd = gzip -n -f
- mod_compress_cmd = $(_GZIP) -n -f endif # CONFIG_MODULE_COMPRESS_GZIP ifdef CONFIG_MODULE_COMPRESS_XZ
- mod_compress_cmd = xz -f
- mod_compress_cmd = $(XZ) -f endif # CONFIG_MODULE_COMPRESS_XZ
endif # CONFIG_MODULE_COMPRESS export mod_compress_cmd diff --git a/arch/arm/boot/deflate_xip_data.sh b/arch/arm/boot/deflate_xip_data.sh index 40937248cebe..739f0464321e 100755 --- a/arch/arm/boot/deflate_xip_data.sh +++ b/arch/arm/boot/deflate_xip_data.sh @@ -56,7 +56,7 @@ trap 'rm -f "$XIPIMAGE.tmp"; exit 1' 1 2 3 # substitute the data section by a compressed version $DD if="$XIPIMAGE" count=$data_start iflag=count_bytes of="$XIPIMAGE.tmp" $DD if="$XIPIMAGE" skip=$data_start iflag=skip_bytes | -gzip -9 >> "$XIPIMAGE.tmp" +$_GZIP -9 >> "$XIPIMAGE.tmp" # replace kernel binary mv -f "$XIPIMAGE.tmp" "$XIPIMAGE" diff --git a/arch/ia64/Makefile b/arch/ia64/Makefile index 32240000dc0c..f817f3d5e758 100644 --- a/arch/ia64/Makefile +++ b/arch/ia64/Makefile @@ -40,7 +40,7 @@ $(error Sorry, you need a newer version of the assember, one that is built from endif quiet_cmd_gzip = GZIP $@ -cmd_gzip = cat $(real-prereqs) | gzip -n -f -9 > $@ +cmd_gzip = cat $(real-prereqs) | $(_GZIP) -n -f -9 > $@ quiet_cmd_objcopy = OBJCOPY $@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@ diff --git a/arch/m68k/Makefile b/arch/m68k/Makefile index 5d9288384096..ce6db5e5a5a3 100644 --- a/arch/m68k/Makefile +++ b/arch/m68k/Makefile @@ -135,10 +135,10 @@ vmlinux.gz: vmlinux ifndef CONFIG_KGDB cp vmlinux vmlinux.tmp $(STRIP) vmlinux.tmp
- gzip -9c vmlinux.tmp >vmlinux.gz
- $(_GZIP) -9c vmlinux.tmp >vmlinux.gz rm vmlinux.tmp
else
- gzip -9c vmlinux >vmlinux.gz
- $(_GZIP) -9c vmlinux >vmlinux.gz
endif bzImage: vmlinux.bz2 @@ -148,10 +148,10 @@ vmlinux.bz2: vmlinux ifndef CONFIG_KGDB cp vmlinux vmlinux.tmp $(STRIP) vmlinux.tmp
- bzip2 -1c vmlinux.tmp >vmlinux.bz2
- $(_BZIP2) -1c vmlinux.tmp >vmlinux.bz2 rm vmlinux.tmp
else
- bzip2 -1c vmlinux >vmlinux.bz2
- $(_BZIP2) -1c vmlinux >vmlinux.bz2
endif archclean: diff --git a/arch/parisc/Makefile b/arch/parisc/Makefile index 36b834f1c933..ad4ba8b5d1e4 100644 --- a/arch/parisc/Makefile +++ b/arch/parisc/Makefile @@ -156,7 +156,7 @@ vmlinuz: bzImage $(OBJCOPY) $(boot)/bzImage $@ else vmlinuz: vmlinux
- @gzip -cf -9 $< > $@
- @$(_GZIP) -cf -9 $< > $@
endif install: diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh index e13ca842eb7e..c1510f0ab3ea 100755 --- a/kernel/gen_kheaders.sh +++ b/kernel/gen_kheaders.sh @@ -88,7 +88,7 @@ find $cpio_dir -type f -print0 | find $cpio_dir -printf "./%P\n" | LC_ALL=C sort | \ tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \ --owner=0 --group=0 --numeric-owner --no-recursion \
- -Jcf $tarfile -C $cpio_dir/ -T - > /dev/null
- -I $XZ -cf $tarfile -C $cpio_dir/ -T - > /dev/null
echo $headers_md5 > kernel/kheaders.md5 echo "$this_file_md5" >> kernel/kheaders.md5 diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib index 342618a2bccb..75d3684aceec 100644 --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -230,7 +230,7 @@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@ # --------------------------------------------------------------------------- quiet_cmd_gzip = GZIP $@
cmd_gzip = cat $(real-prereqs) | gzip -n -f -9 > $@
cmd_gzip = cat $(real-prereqs) | $(_GZIP) -n -f -9 > $@# DTC # --------------------------------------------------------------------------- @@ -322,19 +322,19 @@ printf "%08x\n" $$dec_size | \ ) quiet_cmd_bzip2 = BZIP2 $@
cmd_bzip2 = { cat $(real-prereqs) | bzip2 -9; $(size_append); } > $@
cmd_bzip2 = { cat $(real-prereqs) | $(_BZIP2) -9; $(size_append); } > $@# Lzma # --------------------------------------------------------------------------- quiet_cmd_lzma = LZMA $@
cmd_lzma = { cat $(real-prereqs) | lzma -9; $(size_append); } > $@
cmd_lzma = { cat $(real-prereqs) | $(LZMA) -9; $(size_append); } > $@quiet_cmd_lzo = LZO $@
cmd_lzo = { cat $(real-prereqs) | lzop -9; $(size_append); } > $@
cmd_lzo = { cat $(real-prereqs) | $(_LZOP) -9; $(size_append); } > $@quiet_cmd_lz4 = LZ4 $@
cmd_lz4 = { cat $(real-prereqs) | lz4c -l -c1 stdin stdout; \
cmd_lz4 = { cat $(real-prereqs) | $(LZ4) -l -c1 stdin stdout; \ $(size_append); } > $@# U-Boot mkimage @@ -381,7 +381,7 @@ quiet_cmd_xzkern = XZKERN $@ $(size_append); } > $@ quiet_cmd_xzmisc = XZMISC $@
cmd_xzmisc = cat $(real-prereqs) | xz --check=crc32 --lzma2=dict=1MiB > $@
cmd_xzmisc = cat $(real-prereqs) | $(XZ) --check=crc32 --lzma2=dict=1MiB > $@# ASM offsets # --------------------------------------------------------------------------- diff --git a/scripts/Makefile.package b/scripts/Makefile.package index 56eadcc48d46..4ea20e6b7e2b 100644 --- a/scripts/Makefile.package +++ b/scripts/Makefile.package @@ -45,7 +45,7 @@ if test "$(objtree)" != "$(srctree)"; then \ false; \ fi ; \ $(srctree)/scripts/setlocalversion --save-scmversion; \ -tar -cz $(RCS_TAR_IGNORE) -f $(2).tar.gz \ +tar -I $(_GZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \ --transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3); \ rm -f $(objtree)/.scmversion @@ -127,9 +127,9 @@ util/PERF-VERSION-GEN $(CURDIR)/$(perf-tar)/); \ tar rf $(perf-tar).tar $(perf-tar)/HEAD $(perf-tar)/PERF-VERSION-FILE; \ rm -r $(perf-tar); \ $(if $(findstring tar-src,$@),, \ -$(if $(findstring bz2,$@),bzip2, \ -$(if $(findstring gz,$@),gzip, \ -$(if $(findstring xz,$@),xz, \ +$(if $(findstring bz2,$@),$(_BZIP2), \ +$(if $(findstring gz,$@),$(_GZIP), \ +$(if $(findstring xz,$@),$(XZ), \ $(error unknown target $@)))) \ -f -9 $(perf-tar).tar) diff --git a/scripts/package/buildtar b/scripts/package/buildtar index 2f66c81e4021..4ae3698cd84a 100755 --- a/scripts/package/buildtar +++ b/scripts/package/buildtar @@ -28,15 +28,15 @@ case "${1}" in opts= ;; targz-pkg)
opts=--gzip
opts="-I ${_GZIP}"
opts=--bzip2
opts="-I ${_BZIP2}"
opts=--xz
opts="-I ${XZ}"diff --git a/scripts/xz_wrap.sh b/scripts/xz_wrap.sh index 7a2d372f4885..76e9cbcfbeab 100755 --- a/scripts/xz_wrap.sh +++ b/scripts/xz_wrap.sh @@ -20,4 +20,4 @@ case $SRCARCH in sparc) BCJ=--sparc ;; esac -exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB +exec $XZ --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB
On Wed, Aug 26, 2020 at 10:07:17PM +0300, Denis Efremov wrote:
Hi,
This patch introduces build error, see fix: e4a42c82e943 kbuild: fix broken builds because of GZIP,BZIP2,LZOP variables
Thanks for that, I've picked it up now.
greg k-h
On Wed, Aug 26, 2020 at 05:28:22PM +0100, Matthias Maennich wrote:
Hi,
please pick up the following patches for 5.4.
Those are build time optimizations for kernel/gen_kheaders.sh, and - by removing bashisms - dropping the dependency to /bin/bash.
In addition, this enables build time improvements across the tree by optionally allowing to use alternative implementations for various compression tools, e.g. GZIP=pigz.
The documentation-only change is not strictly necessary, but keeps kernel/gen_kheaders.sh in sync with mainline.
Looks good, now queued up, thanks.
greg k-h
linux-stable-mirror@lists.linaro.org
-
Denis Efremov -
Greg Kroah-Hartman -
Matthias Maennich