Re: # Toplevel Makefile doesn't process module build correctly on recursive make calls - Linux-kselftest-mirror

24 Oct 2023

On Fri, Oct 20, 2023 at 06:21:40AM +0200, Nicolas Schier wrote:
...
On Thu, Oct 19, 2023 at 03:50:05PM -0300, Marcos Paulo de Souza wrote:
...
On Sat, Oct 14, 2023 at 05:35:55PM +0900, Masahiro Yamada wrote:
...
On Tue, Oct 10, 2023 at 5:43 AM Marcos Paulo de Souza mpdesouza@suse.de wrote:
...
Hi all,
I found an issue while moving the livepatch kselftest modules to be built on the
fly, instead of building them on kernel building.
If, for some reason, there is a recursive make invocation that starts from the
top level Makefile and in the leaf Makefile it tries to build a module (using M=
in the make invocation), it doesn't produce the module. This happens because the
toplevel Makefile checks for M= only once. This is controlled by the
sub_make_done variable, which is exported after checking the command line
options are passed to the top level Makefile. Once this variable is set it's
the M= setting is never checked again on the recursive call.
This can be observed when cleaning the bpf kselftest dir. When calling
    $ make TARGETS="bpf" SKIP_TARGETS="" kselftest-clean


What happens:
    1. It checks for some command line settings (like M=) was passed (it wasn't),
    set some definitions and exports sub_make_done.

    2. Jump into tools/testing/selftests/bpf, and calls the clean target.

    3. The clean target is overwritten to remove some files and then jump to
    bpf_testmod dir and call clean there

    4. On bpf_testmod/Makefile, the clean target will execute
            $(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean

    5. The KDIR is to toplevel dir. The top Makefile will check that sub_make_done was
    already set, ignoring the M= setting.

    6. As M= wasn't checked, KBUILD_EXTMOD isn't set, and the clean target applies
    to the kernel as a whole, making it clean all generated code/objects and
    everything.


One way to avoid it is to call "unexport sub_make_done" on
tools/testing/selftests/bpf/bpf_testmod/Makefile before processing the all
target, forcing the toplevel Makefile to process the M=, producing the module
file correctly.
If the M=dir points to /lib/modules/.../build, then it fails with "m2c: No such
file", which I already reported here[1]. At the time this problem was treated
like a problem with kselftest infrastructure.
Important: The process works fine if the initial make invocation is targeted to a
different directory (using -C), since it doesn't goes through the toplevel
Makefile, and sub_make_done variable is not set.
I attached a minimal reproducer, that can be used to better understand the
problem. The "make testmod" and "make testmod-clean" have the same effect that
can be seem with the bpf kselftests. There is a unexport call commented on
test-mods/Makefile, and once that is called the process works as expected.
Is there a better way to fix this? Is this really a problem, or am I missing
something?
Or, using KBUILD_EXTMOD will work too.
Yes, that works, only if set to /lib/modules:
$ make kselftest TARGETS=bpf SKIP_TARGETS=""
make[3]: Entering directory '/home/mpdesouza/git/linux/tools/testing/selftests/bpf'
  MOD      bpf_testmod.ko
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: gcc (SUSE Linux) 13.2.1 20230803 [revision cc279d6c64562f05019e1d12d0d825f9391b5553]
  You are using:           gcc (SUSE Linux) 13.2.1 20230912 [revision b96e66fd4ef3e36983969fb8cdd1956f551a074b]
  CC [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.o
  MODPOST /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/Module.symvers
  CC [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.mod.o
  LD [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko
  BTF [M] /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko
Skipping BTF generation for /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko due to unavailability of vmlinux
  BINARY   xdp_synproxy
...
But if we set the KBUILD_EXTMOD to toplevel Makefile, it fails with a different
strange issue:
$ make kselftest TARGETS=bpf SKIP_TARGETS=""
  BINARY   urandom_read
  MOD      bpf_testmod.ko
m2c    -o scripts/Makefile.build -e scripts/Makefile.build scripts/Makefile.build.mod
make[6]: m2c: No such file or directory
make[6]: *** [<builtin>: scripts/Makefile.build] Error 127
make[5]: *** [Makefile:1913: /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod] Error 2
make[4]: *** [Makefile:19: all] Error 2
make[3]: *** [Makefile:229: /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod.ko] Error 2
make[3]: Leaving directory '/home/mpdesouza/git/linux/tools/testing/selftests/bpf'
make[2]: *** [Makefile:175: all] Error 2
make[1]: *** [/home/mpdesouza/git/linux/Makefile:1362: kselftest] Error 2
I attached a patch that can reproduce the case where it works, and the case
where it doesn't by changing the value of KDIR.
I understand that KBUILD_EXTMOD, as the name implies, was designed to build
"external" modules, and not ones that live inside kernel, but how could this be
solved?
It seems to me as if there is some confusion about in-tree vs.
out-of-tree kmods.
KBUILD_EXTMOD and M are almost the same and indicate that you want to
build _external_ (=out-of-tree) kernel modules.  In-tree modules are
only those that stay in-tree _and_ are built along with the kernel.
Thus, 'make modules KBUILD_EXTMOD=fs/ext4' could be used to build ext4
kmod as "out-of-tree" kernel module, that even taints the kernel if it
gets loaded.
If you want bpf_testmod.ko to be an in-tree kmod, it has to be build
during the usual kernel build, not by running 'make kselftest'.
If you use 'make -C $(KDIR)' for building out-of-tree kmods, KDIR has to
point to the kernel build directory.  (Or it may point to the source
tree if you give O=$(BUILDDIR) as well).
Thanks for the explanation Nicolas. In this, I believe that the BPF module
should be moved into lib/, like lib/livepatch, when then be built along with
other in-tree modules.
Currently there is a bug when running the kselftests-clean target with bpf:
make kselftest-clean TARGETS=bpf SKIP_TARGETS=""
As the M= argument is ignore on the toplevel Makefile, this make invocation
applies the clean to all built kernel objects/modules/everything, which is bug
IMO.
There is a statement in the BPF docs saying that the selftests should be run
inside the tools/testing/selftests/bpf directory. At the same time, kselftests
should comply with all the targets defined in the documention, like gen_tar, and
run_tests. In this case should the build process be fixed, or just make
kselftests less restrict?
(CCing kselftests and bpf ML)
...
HTH.
Kind regards,
Nicolas
...
For the sake of my initial about livepatch kselftests, KBUILD_EXTMOD
will suffice, since we will target /lib/modules, but I would like to know what
we can do in this case. Do you have other suggestions?
Thanks in advance,
  Marcos
...
--
Best Regards
Masahiro Yamada
...

diff --git a/tools/testing/selftests/bpf/bpf_testmod/Makefile b/tools/testing/selftests/bpf/bpf_testmod/Makefile
index 15cb36c4483a..1dce76f35405 100644
--- a/tools/testing/selftests/bpf/bpf_testmod/Makefile
+++ b/tools/testing/selftests/bpf/bpf_testmod/Makefile
@@ -1,5 +1,6 @@
 BPF_TESTMOD_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
-KDIR ?= $(abspath $(BPF_TESTMOD_DIR)/../../../../..)
+#KDIR ?= $(abspath $(BPF_TESTMOD_DIR)/../../../../..)
+KDIR ?= /lib/modules/$(shell uname -r)/build
 
 ifeq ($(V),1)
 Q =
@@ -12,9 +13,10 @@ MODULES = bpf_testmod.ko
 obj-m += bpf_testmod.o
 CFLAGS_bpf_testmod.o = -I$(src)
 
+export KBUILD_EXTMOD := $(BPF_TESTMOD_DIR)



all:

+$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) modules


+$(Q)make -C $(KDIR) modules

clean:

+$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean



+$(Q)make -C $(KDIR) clean


    