From: Björn Töpel bjorn@rivosinc.com
When the BPF selftests are cross-compiled, only the a host version of bpftool is built. This version of bpftool is used to generate various intermediates, e.g., skeletons.
The test runners are also using bpftool. The Makefile will symlink bpftool from the selftest/bpf root, where the test runners will look for the tool:
| ... | $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \ | $(OUTPUT)/$(if $2,$2/)bpftool
There are two issues for cross-compilation builds:
1. There is no native (cross-compilation target) build of bpftool 2. The bootstrap variant of bpftool is never cross-compiled (by design)
Make sure that a native/cross-compiled version of bpftool is built, and if CROSS_COMPILE is set, symlink to the native/non-bootstrap version.
Signed-off-by: Björn Töpel bjorn@rivosinc.com --- tools/testing/selftests/bpf/Makefile | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-)
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index b2eb3201b85a..b706750f71e2 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -157,8 +157,9 @@ $(notdir $(TEST_GEN_PROGS) \ $(TEST_CUSTOM_PROGS)): %: $(OUTPUT)/% ;
# sort removes libbpf duplicates when not cross-building -MAKE_DIRS := $(sort $(BUILD_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \ - $(HOST_BUILD_DIR)/bpftool $(HOST_BUILD_DIR)/resolve_btfids \ +MAKE_DIRS := $(sort $(BUILD_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \ + $(BUILD_DIR)/bpftool $(HOST_BUILD_DIR)/bpftool \ + $(HOST_BUILD_DIR)/resolve_btfids \ $(RUNQSLOWER_OUTPUT) $(INCLUDE_DIR)) $(MAKE_DIRS): $(call msg,MKDIR,,$@) @@ -208,6 +209,14 @@ $(OUTPUT)/bpf_testmod.ko: $(VMLINUX_BTF) $(wildcard bpf_testmod/Makefile bpf_tes $(Q)cp bpf_testmod/bpf_testmod.ko $@
DEFAULT_BPFTOOL := $(HOST_SCRATCH_DIR)/sbin/bpftool +ifneq ($(CROSS_COMPILE),) +CROSS_BPFTOOL := $(SCRATCH_DIR)/sbin/bpftool +TRUNNER_BPFTOOL := $(CROSS_BPFTOOL) +USE_BOOTSTRAP := "" +else +TRUNNER_BPFTOOL := $(DEFAULT_BPFTOOL) +USE_BOOTSTRAP := "bootstrap" +endif
$(OUTPUT)/runqslower: $(BPFOBJ) | $(DEFAULT_BPFTOOL) $(RUNQSLOWER_OUTPUT) $(Q)$(MAKE) $(submake_extras) -C $(TOOLSDIR)/bpf/runqslower \ @@ -255,6 +264,18 @@ $(DEFAULT_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \ LIBBPF_DESTDIR=$(HOST_SCRATCH_DIR)/ \ prefix= DESTDIR=$(HOST_SCRATCH_DIR)/ install-bin
+ifneq ($(CROSS_COMPILE),) +$(CROSS_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \ + $(BPFOBJ) | $(BUILD_DIR)/bpftool + $(Q)$(MAKE) $(submake_extras) -C $(BPFTOOLDIR) \ + ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) \ + EXTRA_CFLAGS='-g -O0' \ + OUTPUT=$(BUILD_DIR)/bpftool/ \ + LIBBPF_OUTPUT=$(BUILD_DIR)/libbpf/ \ + LIBBPF_DESTDIR=$(SCRATCH_DIR)/ \ + prefix= DESTDIR=$(SCRATCH_DIR)/ install-bin +endif + all: docs
docs: @@ -518,11 +539,12 @@ endif $(OUTPUT)/$(TRUNNER_BINARY): $(TRUNNER_TEST_OBJS) \ $(TRUNNER_EXTRA_OBJS) $$(BPFOBJ) \ $(RESOLVE_BTFIDS) \ + $(TRUNNER_BPFTOOL) \ | $(TRUNNER_BINARY)-extras $$(call msg,BINARY,,$$@) $(Q)$$(CC) $$(CFLAGS) $$(filter %.a %.o,$$^) $$(LDLIBS) -o $$@ $(Q)$(RESOLVE_BTFIDS) --btf $(TRUNNER_OUTPUT)/btf_data.bpf.o $$@ - $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \ + $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/$(USE_BOOTSTRAP)/bpftool \ $(OUTPUT)/$(if $2,$2/)bpftool
endef
base-commit: 06744f24696e1e7598412c3df61a538b57ebec22
Björn Töpel bjorn@kernel.org writes:
From: Björn Töpel bjorn@rivosinc.com
When the BPF selftests are cross-compiled, only the a host version of bpftool is built. This version of bpftool is used to generate various intermediates, e.g., skeletons.
The test runners are also using bpftool. The Makefile will symlink bpftool from the selftest/bpf root, where the test runners will look for the tool:
| ... | $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \ | $(OUTPUT)/$(if $2,$2/)bpftool
There are two issues for cross-compilation builds:
- There is no native (cross-compilation target) build of bpftool
- The bootstrap variant of bpftool is never cross-compiled (by design)
Make sure that a native/cross-compiled version of bpftool is built, and if CROSS_COMPILE is set, symlink to the native/non-bootstrap version.
...and the grand master plan is to add BPF CI support for riscv64, where this patch a prerequisite to [1]. I would suspect that other platforms might benefit from cross-compilation builds as well.
On 13/02/2023 14:30, Björn Töpel wrote:
Björn Töpel bjorn@kernel.org writes:
From: Björn Töpel bjorn@rivosinc.com
When the BPF selftests are cross-compiled, only the a host version of bpftool is built. This version of bpftool is used to generate various intermediates, e.g., skeletons.
The test runners are also using bpftool. The Makefile will symlink bpftool from the selftest/bpf root, where the test runners will look for the tool:
| ... | $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \ | $(OUTPUT)/$(if $2,$2/)bpftool
There are two issues for cross-compilation builds:
- There is no native (cross-compilation target) build of bpftool
- The bootstrap variant of bpftool is never cross-compiled (by design)
Make sure that a native/cross-compiled version of bpftool is built, and if CROSS_COMPILE is set, symlink to the native/non-bootstrap version.
...and the grand master plan is to add BPF CI support for riscv64, where this patch a prerequisite to [1]. I would suspect that other platforms might benefit from cross-compilation builds as well.
Similar use case. There also seems to be a lot of issues building these tests out of tree.
I have some potential fixes up to 6.1 but linux-next seems to have introduced a few more issues on top.
Zachary Leaf zachary.leaf@arm.com writes:
On 13/02/2023 14:30, Björn Töpel wrote:
Björn Töpel bjorn@kernel.org writes:
From: Björn Töpel bjorn@rivosinc.com
When the BPF selftests are cross-compiled, only the a host version of bpftool is built. This version of bpftool is used to generate various intermediates, e.g., skeletons.
The test runners are also using bpftool. The Makefile will symlink bpftool from the selftest/bpf root, where the test runners will look for the tool:
| ... | $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \ | $(OUTPUT)/$(if $2,$2/)bpftool
There are two issues for cross-compilation builds:
- There is no native (cross-compilation target) build of bpftool
- The bootstrap variant of bpftool is never cross-compiled (by design)
Make sure that a native/cross-compiled version of bpftool is built, and if CROSS_COMPILE is set, symlink to the native/non-bootstrap version.
...and the grand master plan is to add BPF CI support for riscv64, where this patch a prerequisite to [1]. I would suspect that other platforms might benefit from cross-compilation builds as well.
Similar use case. There also seems to be a lot of issues building these tests out of tree.
I have some potential fixes up to 6.1 but linux-next seems to have introduced a few more issues on top.
Ah, yes. FWIW, the BPF CI builds the selftests *in-tree*, so with this patch (and my PRs) the BPF CI is capable of cross-compiling.
Björn
2023-02-10 09:43 UTC+0100 ~ Björn Töpel bjorn@kernel.org
From: Björn Töpel bjorn@rivosinc.com
When the BPF selftests are cross-compiled, only the a host version of bpftool is built. This version of bpftool is used to generate various intermediates, e.g., skeletons.
The test runners are also using bpftool. The Makefile will symlink bpftool from the selftest/bpf root, where the test runners will look for the tool:
| ... | $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \ | $(OUTPUT)/$(if $2,$2/)bpftool
There are two issues for cross-compilation builds:
- There is no native (cross-compilation target) build of bpftool
- The bootstrap variant of bpftool is never cross-compiled (by design)
Make sure that a native/cross-compiled version of bpftool is built, and if CROSS_COMPILE is set, symlink to the native/non-bootstrap version.
Signed-off-by: Björn Töpel bjorn@rivosinc.com
tools/testing/selftests/bpf/Makefile | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-)
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index b2eb3201b85a..b706750f71e2 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -157,8 +157,9 @@ $(notdir $(TEST_GEN_PROGS) \ $(TEST_CUSTOM_PROGS)): %: $(OUTPUT)/% ; # sort removes libbpf duplicates when not cross-building -MAKE_DIRS := $(sort $(BUILD_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \
$(HOST_BUILD_DIR)/bpftool $(HOST_BUILD_DIR)/resolve_btfids \+MAKE_DIRS := $(sort $(BUILD_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \
$(BUILD_DIR)/bpftool $(HOST_BUILD_DIR)/bpftool \$(HOST_BUILD_DIR)/resolve_btfids \ $(RUNQSLOWER_OUTPUT) $(INCLUDE_DIR))$(MAKE_DIRS): $(call msg,MKDIR,,$@) @@ -208,6 +209,14 @@ $(OUTPUT)/bpf_testmod.ko: $(VMLINUX_BTF) $(wildcard bpf_testmod/Makefile bpf_tes $(Q)cp bpf_testmod/bpf_testmod.ko $@ DEFAULT_BPFTOOL := $(HOST_SCRATCH_DIR)/sbin/bpftool +ifneq ($(CROSS_COMPILE),) +CROSS_BPFTOOL := $(SCRATCH_DIR)/sbin/bpftool +TRUNNER_BPFTOOL := $(CROSS_BPFTOOL) +USE_BOOTSTRAP := "" +else +TRUNNER_BPFTOOL := $(DEFAULT_BPFTOOL) +USE_BOOTSTRAP := "bootstrap" +endif $(OUTPUT)/runqslower: $(BPFOBJ) | $(DEFAULT_BPFTOOL) $(RUNQSLOWER_OUTPUT) $(Q)$(MAKE) $(submake_extras) -C $(TOOLSDIR)/bpf/runqslower \ @@ -255,6 +264,18 @@ $(DEFAULT_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \ LIBBPF_DESTDIR=$(HOST_SCRATCH_DIR)/ \ prefix= DESTDIR=$(HOST_SCRATCH_DIR)/ install-bin +ifneq ($(CROSS_COMPILE),) +$(CROSS_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \
$(BPFOBJ) | $(BUILD_DIR)/bpftool- $(Q)$(MAKE) $(submake_extras) -C $(BPFTOOLDIR) \
ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) \EXTRA_CFLAGS='-g -O0' \OUTPUT=$(BUILD_DIR)/bpftool/ \LIBBPF_OUTPUT=$(BUILD_DIR)/libbpf/ \LIBBPF_DESTDIR=$(SCRATCH_DIR)/ \prefix= DESTDIR=$(SCRATCH_DIR)/ install-bin+endif
all: docs docs: @@ -518,11 +539,12 @@ endif $(OUTPUT)/$(TRUNNER_BINARY): $(TRUNNER_TEST_OBJS) \ $(TRUNNER_EXTRA_OBJS) $$(BPFOBJ) \ $(RESOLVE_BTFIDS) \
$(TRUNNER_BPFTOOL) \ | $(TRUNNER_BINARY)-extras
- $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \
- $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/$(USE_BOOTSTRAP)/bpftool \
Nit: You'll have a double slash in this path when USE_BOOSTRAP is empty (.../tools/build/bpftool//bpftool), but it probably doesn't matter much.
$(OUTPUT)/$(if $2,$2/)bpftoolendef
base-commit: 06744f24696e1e7598412c3df61a538b57ebec22
The changes look good to me, thanks!
Acked-by: Quentin Monnet quentin@isovalent.com
Jean-Philippe, I know you do some cross-compiling with bpftool, how does this look from your side?
Hi Quentin,
On Mon, Feb 13, 2023 at 08:53:31PM +0000, Quentin Monnet wrote:
Jean-Philippe, I know you do some cross-compiling with bpftool, how does this look from your side?
I don't have enough time for BPF at the moment unfortunately. Zachary is looking at cross-compiling the selftests for arm64
Thanks, Jean
Hi Bjorn,
Thanks for the patch, I've tested it and it works for me.
I have a minor suggestion but otherwise happy to see this getting fixed.
On 10/02/2023 08:43, Björn Töpel wrote:
From: Björn Töpel bjorn@rivosinc.com
When the BPF selftests are cross-compiled, only the a host version of bpftool is built. This version of bpftool is used to generate various intermediates, e.g., skeletons.
The test runners are also using bpftool. The Makefile will symlink bpftool from the selftest/bpf root, where the test runners will look for the tool:
| ... | $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \ | $(OUTPUT)/$(if $2,$2/)bpftool
There are two issues for cross-compilation builds:
- There is no native (cross-compilation target) build of bpftool
- The bootstrap variant of bpftool is never cross-compiled (by design)
Make sure that a native/cross-compiled version of bpftool is built, and if CROSS_COMPILE is set, symlink to the native/non-bootstrap version.
Signed-off-by: Björn Töpel bjorn@rivosinc.com
tools/testing/selftests/bpf/Makefile | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-)
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index b2eb3201b85a..b706750f71e2 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -157,8 +157,9 @@ $(notdir $(TEST_GEN_PROGS) \ $(TEST_CUSTOM_PROGS)): %: $(OUTPUT)/% ; # sort removes libbpf duplicates when not cross-building -MAKE_DIRS := $(sort $(BUILD_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \
$(HOST_BUILD_DIR)/bpftool $(HOST_BUILD_DIR)/resolve_btfids \+MAKE_DIRS := $(sort $(BUILD_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \
$(BUILD_DIR)/bpftool $(HOST_BUILD_DIR)/bpftool \$(HOST_BUILD_DIR)/resolve_btfids \ $(RUNQSLOWER_OUTPUT) $(INCLUDE_DIR))$(MAKE_DIRS): $(call msg,MKDIR,,$@) @@ -208,6 +209,14 @@ $(OUTPUT)/bpf_testmod.ko: $(VMLINUX_BTF) $(wildcard bpf_testmod/Makefile bpf_tes $(Q)cp bpf_testmod/bpf_testmod.ko $@ DEFAULT_BPFTOOL := $(HOST_SCRATCH_DIR)/sbin/bpftool +ifneq ($(CROSS_COMPILE),) +CROSS_BPFTOOL := $(SCRATCH_DIR)/sbin/bpftool +TRUNNER_BPFTOOL := $(CROSS_BPFTOOL) +USE_BOOTSTRAP := "" +else +TRUNNER_BPFTOOL := $(DEFAULT_BPFTOOL) +USE_BOOTSTRAP := "bootstrap" +endif $(OUTPUT)/runqslower: $(BPFOBJ) | $(DEFAULT_BPFTOOL) $(RUNQSLOWER_OUTPUT) $(Q)$(MAKE) $(submake_extras) -C $(TOOLSDIR)/bpf/runqslower \
...
-TEST_GEN_PROGS_EXTENDED += $(DEFAULT_BPFTOOL) +TEST_GEN_PROGS_EXTENDED += $(TRUNNER_BPFTOOL)
Ensure the target arch bpftool is copied into the kselftest_bpf_install dir by selftests/lib.mk instead of always the default/host version.
Thanks, Zach
@@ -255,6 +264,18 @@ $(DEFAULT_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \ LIBBPF_DESTDIR=$(HOST_SCRATCH_DIR)/ \ prefix= DESTDIR=$(HOST_SCRATCH_DIR)/ install-bin +ifneq ($(CROSS_COMPILE),) +$(CROSS_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \
$(BPFOBJ) | $(BUILD_DIR)/bpftool- $(Q)$(MAKE) $(submake_extras) -C $(BPFTOOLDIR) \
ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) \EXTRA_CFLAGS='-g -O0' \OUTPUT=$(BUILD_DIR)/bpftool/ \LIBBPF_OUTPUT=$(BUILD_DIR)/libbpf/ \LIBBPF_DESTDIR=$(SCRATCH_DIR)/ \prefix= DESTDIR=$(SCRATCH_DIR)/ install-bin+endif
all: docs docs: @@ -518,11 +539,12 @@ endif $(OUTPUT)/$(TRUNNER_BINARY): $(TRUNNER_TESAT_OBJS) \ $(TRUNNER_EXTRA_OBJS) $$(BPFOBJ) \ $(RESOLVE_BTFIDS) related to \
$(TRUNNER_BPFTOOL) \ | $(TRUNNER_BINARY)-extras
- $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \
- $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/$(USE_BOOTSTRAP)/bpftool > $(OUTPUT)/$(if $2,$2/)bpftool
endef
base-commit: 06744f24696e1e7598412c3df61a538b57ebec22
Zachary Leaf zachary.leaf@arm.com writes:
Hi Bjorn,
Thanks for the patch, I've tested it and it works for me.
Good!
I have a minor suggestion but otherwise happy to see this getting fixed.
...
-TEST_GEN_PROGS_EXTENDED += $(DEFAULT_BPFTOOL) +TEST_GEN_PROGS_EXTENDED += $(TRUNNER_BPFTOOL)
Ensure the target arch bpftool is copied into the kselftest_bpf_install dir by selftests/lib.mk instead of always the default/host version.
Good one. I'll spin a v2, and also fix Quentin's "double-slash".
Björn
linux-kselftest-mirror@lists.linaro.org
-
Björn Töpel -
Jean-Philippe Brucker -
Quentin Monnet -
Zachary Leaf