On 2/19/20 5:27 PM, Alexei Starovoitov wrote:
> On Wed, Feb 19, 2020 at 03:59:41PM -0600, Daniel Díaz wrote:
>>>
>>> When I download a specific kernel release, how can I know what LLVM
>>> git-hash or version I need (to use BPF-selftests)?
>
> as discussed we're going to add documentation-like file that will
> list required commits in tools.
> This will be enforced for future llvm/pahole commits.
>
>>> Do you think it is reasonable to require end-users to compile their own
>>> bleeding edge version of LLVM, to use BPF-selftests?
>
> absolutely.
+ linux-kselftest(a)vger.kernel.org
End-users in this context are users and not necessarily developers.
> If a developer wants to send a patch they must run all selftests and
> all of them must pass in their environment.
> "but I'm adding a tracing feature and don't care about networking tests
> failing"... is not acceptable.
This is a reasonable expectation when a developers sends bpf patches.
>
>>> I do hope that some end-users of BPF-selftests will be CI-systems.
>>> That also implies that CI-system maintainers need to constantly do
>>> "latest built from sources" of LLVM git-tree to keep up. Is that a
>>> reasonable requirement when buying a CI-system in the cloud?
>
> "buying CI-system in the cloud" ?
> If I could buy such system I would pay for it out of my own pocket to save
> maintainer's and developer's time.
>
>> We [1] are end users of kselftests and many other test suites [2]. We
>> run all of our testing on every git-push on linux-stable-rc, mainline,
>> and linux-next -- approximately 1 million tests per week. We have a
>> dedicated engineering team looking after this CI infrastructure and
>> test results, and as such, I can wholeheartedly echo Jesper's
>> sentiment here: We would really like to help kernel maintainers and
>> developers by automatically testing their code in real hardware, but
>> the BPF kselftests are difficult to work with from a CI perspective.
>> We have caught and reported [3] many [4] build [5] failures [6] in the
>> past for libbpf/Perf, but building is just one of the pieces. We are
>> unable to run the entire BPF kselftests because only a part of the
>> code builds, so our testing is very limited there.
>>
>> We hope that this situation can be improved and that our and everyone
>> else's automated testing can help you guys too. For this to work out,
>> we need some help.
>
It would be helpful understand what "help" is in this context.
> I don't understand what kind of help you need. Just install the latest tools.
What would be helpful is to write bpf tests such that older tests that
worked on older llvm versions continue to work and with some indication
on which tests require new bleeding edge tools.
> Both the latest llvm and the latest pahole are required.
It would be helpful if you can elaborate why latest tools are a
requirement.
> If by 'help' you mean to tweak selftests to skip tests then it's a nack.
> We have human driven CI. Every developer must run selftests/bpf before
> emailing the patches. Myself and Daniel run them as well before applying.
> These manual runs is the only thing that keeps bpf tree going.
> If selftests get to skip tests humans will miss those errors.
> When I don't see '0 SKIPPED, 0 FAILED' I go and investigate.
> Anything but zero is a path to broken kernels.
>
> Imagine the tests would get skipped when pahole is too old.
> That would mean all of the kernel features from year 2019
> would get skipped. Is there a point of running such selftests?
> I think the value is not just zero. The value is negative.
> Such selftests that run old stuff would give false believe
> that they do something meaningful.
> "but CI can do build only tests"... If 'helping' such CI means hurting the
> key developer/maintainer workflow such CI is on its own.
>
Skipping tests will be useless. I am with you on that. However,
figuring out how to maintain some level of backward compatibility
to run at least older tests and warn users to upgrade would be
helpful.
I suspect currently users are ignoring bpf failures because they
are unable to keep up with the requirement to install newer tools
to run the tests. This isn't great either.
Users that care are sharing their pain to see if they can get some
help or explanation on why new tools are required every so often.
I don't think everybody understands why. :)
thanks,
-- Shuah
On 2/20/20 7:58 AM, Christophe Leroy wrote:
> ping
>
> On 02/06/2020 08:11 AM, Christophe Leroy wrote:
>> Commit 46d1a0f03d66 ("selftests/lkdtm: Add tests for LKDTM targets")
>> added generation of lkdtm test scripts.
>>
>> Ignore those generated scripts when performing 'git status'
>>
>> Fixes: 46d1a0f03d66 ("selftests/lkdtm: Add tests for LKDTM targets")
>> Signed-off-by: Christophe Leroy <christophe.leroy(a)c-s.fr>
>
> Without this, 'git status' now reports the following crap and real
> problems are drowned in the middle, that's annoying.
>
I will pull this in. Please cc linux-kselftest mailing list in the
future.
thanks,
-- Shuah
Hi Linus,
Please pull the following Kselftest fixes update for Linux 5.6-rc3
This Kselftest update for Linux 5.6-rc3 consists of fixes to build
failures and other test bugs.
diff is attached.
thanks,
-- Shuah
----------------------------------------------------------------
The following changes since commit bb6d3fb354c5ee8d6bde2d576eb7220ea09862b9:
Linux 5.6-rc1 (2020-02-09 16:08:48 -0800)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest
tags/linux-kselftest-5.6-rc3
for you to fetch changes up to 9a0584f05687947d5a0b87f046bcd2592a55e67c:
selftests: openat2: fix build error on newer glibc (2020-02-13
13:15:45 -0700)
----------------------------------------------------------------
linux-kselftest-5.6-rc3
This Kselftest update for Linux 5.6-rc3 consists of fixes to build
failures and other test bugs.
----------------------------------------------------------------
Aleksa Sarai (1):
selftests: openat2: fix build error on newer glibc
Colin Ian King (1):
selftests: fix spelling mistaked "chaigned" -> "chained"
Dmitry Safonov (1):
selftests: use LDLIBS for libraries instead of LDFLAGS
Jiri Benc (2):
selftests: allow detection of build failures
selftests: fix too long argument
Nikita Sobolev (1):
Kernel selftests: tpm2: check for tpm support
Steven Rostedt (VMware) (1):
selftests/ftrace: Have pid filter test use instance flag
tools/testing/selftests/Makefile | 12 +++++++++--
.../ftrace/test.d/ftrace/func-filter-pid.tc | 8 +-------
tools/testing/selftests/futex/functional/Makefile | 2 +-
tools/testing/selftests/lib.mk | 23
++++++++++++----------
tools/testing/selftests/net/Makefile | 4 ++--
tools/testing/selftests/openat2/helpers.c | 2 +-
tools/testing/selftests/openat2/resolve_test.c | 2 +-
tools/testing/selftests/rtc/Makefile | 2 +-
tools/testing/selftests/timens/Makefile | 2 +-
tools/testing/selftests/tpm2/test_smoke.sh | 13 ++++++++++--
tools/testing/selftests/tpm2/test_space.sh | 9 ++++++++-
11 files changed, 50 insertions(+), 29 deletions(-)
----------------------------------------------------------------
This series adds new functions to the mmu interval notifier API to
allow device drivers with MMUs to dynamically mirror a process' page
tables based on device faults and invalidation callbacks. The Nouveau
driver is updated to use the extended API and a set of stand alone self
tests is added to help validate and maintain correctness.
The patches are based on linux-5.5.0-rc6 and are for Jason's rdma/hmm tree
since I believe he is planning some interval notifier changes.
Changes v5 -> v6:
Rebase to linux-5.5.0-rc6
Refactored mmu interval notifier patches
Converted nouveau to use the new mmu interval notifier API
Changes v4 -> v5:
Added mmu interval notifier insert/remove/update callable from the
invalidate() callback
Updated HMM tests to use the new core interval notifier API
Changes v1 -> v4:
https://lore.kernel.org/linux-mm/20191104222141.5173-1-rcampbell@nvidia.com
Ralph Campbell (6):
mm/mmu_notifier: add mmu_interval_notifier_insert_safe()
mm/mmu_notifier: add mmu_interval_notifier_put()
mm/notifier: add mmu_interval_notifier_update()
mm/mmu_notifier: add mmu_interval_notifier_find()
nouveau: use new mmu interval notifiers
mm/hmm/test: add self tests for HMM
MAINTAINERS | 3 +
drivers/gpu/drm/nouveau/nouveau_svm.c | 313 ++++--
include/linux/mmu_notifier.h | 29 +
lib/Kconfig.debug | 11 +
lib/Makefile | 1 +
lib/test_hmm.c | 1368 ++++++++++++++++++++++++
mm/mmu_notifier.c | 223 +++-
tools/testing/selftests/vm/.gitignore | 1 +
tools/testing/selftests/vm/Makefile | 3 +
tools/testing/selftests/vm/config | 2 +
tools/testing/selftests/vm/hmm-tests.c | 1354 +++++++++++++++++++++++
tools/testing/selftests/vm/run_vmtests | 16 +
tools/testing/selftests/vm/test_hmm.sh | 97 ++
13 files changed, 3289 insertions(+), 132 deletions(-)
create mode 100644 lib/test_hmm.c
create mode 100644 tools/testing/selftests/vm/hmm-tests.c
create mode 100755 tools/testing/selftests/vm/test_hmm.sh
--
2.20.1
On 2/19/20 3:38 PM, Heidi Fahim wrote:
> On Wed, Feb 19, 2020 at 2:18 PM shuah <shuah(a)kernel.org
> <mailto:shuah@kernel.org>> wrote:
>
> Hi Heidi,
>
> On 2/18/20 3:19 PM, Heidi Fahim wrote:
> > Implemented small fix so that the script changes work directories
> to the
> > root of the linux kernel source tree from which kunit.py is run. This
> > enables the user to run kunit from any working directory. Originally
> > considered using os.path.join but this is more error prone as we
> would
> > have to find all file path usages and modify them accordingly. Using
> > os.chdir ensures that the entire script is run within /linux.
> >
> > Signed-off-by: Heidi Fahim <heidifahim(a)google.com
> <mailto:heidifahim@google.com>>
> > Reviewed-by: Brendan Higgins <brendanhiggins(a)google.com
> <mailto:brendanhiggins@google.com>>
>
> Thanks for the patch. In the future please include changes from v1
> to v2.
>
> I am assuming this v2 addresses Frank's comments.
>
>
> Sorry about that! Yes the only change here was in the commit message
> addressing Frank's comment.
>
Great. I will pull this in for 5.6-rc4.
thanks,
-- Shuah
Implemented small fix so that the script changes work directories to the
root of the linux kernel source tree from which kunit.py is run. This
enables the user to run kunit from any working directory. Originally
considered using os.path.join but this is more error prone as we would
have to find all file path usages and modify them accordingly. Using
os.chdir ensures that the entire script is run within /linux.
Signed-off-by: Heidi Fahim <heidifahim(a)google.com>
Reviewed-by: Brendan Higgins <brendanhiggins(a)google.com>
---
tools/testing/kunit/kunit.py | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index 3f552e847a14..060d960a7029 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -26,6 +26,8 @@ KunitResult = namedtuple('KunitResult', ['status','result'])
KunitRequest = namedtuple('KunitRequest', ['raw_output','timeout', 'jobs',
'build_dir', 'defconfig', 'json'])
+KernelDirectoryPath = sys.argv[0].split('tools/testing/kunit/')[0]
+
class KunitStatus(Enum):
SUCCESS = auto()
CONFIG_FAILURE = auto()
@@ -37,6 +39,13 @@ def create_default_kunitconfig():
shutil.copyfile('arch/um/configs/kunit_defconfig',
kunit_kernel.kunitconfig_path)
+def get_kernel_root_path():
+ parts = sys.argv[0] if not __file__ else __file__
+ parts = os.path.realpath(parts).split('tools/testing/kunit')
+ if len(parts) != 2:
+ sys.exit(1)
+ return parts[0]
+
def run_tests(linux: kunit_kernel.LinuxSourceTree,
request: KunitRequest) -> KunitResult:
config_start = time.time()
@@ -130,6 +139,9 @@ def main(argv, linux=None):
cli_args = parser.parse_args(argv)
if cli_args.subcommand == 'run':
+ if get_kernel_root_path():
+ os.chdir(get_kernel_root_path())
+
if cli_args.build_dir:
if not os.path.exists(cli_args.build_dir):
os.mkdir(cli_args.build_dir)
--
2.25.0.265.gbab2e86ba0-goog
This patch series extend the "bpftool feature" subcommand with the
new positional arguments:
- "section", which allows to select a specific section of probes (i.e.
"system_config", "program_types", "map_types");
- "filter_in", which allows to select only probes which matches the
given regex pattern;
- "filter_out", which allows to filter out probes which do not match the
given regex pattern.
The main motivation behind those changes is ability the fact that some
probes (for example those related to "trace" or "write_user" helpers)
emit dmesg messages which might be confusing for people who are running
on production environments. For details see the Cilium issue[0].
[0] https://github.com/cilium/cilium/issues/10048
Michal Rostecki (6):
bpftool: Move out sections to separate functions
bpftool: Allow to select a specific section to probe
bpftool: Add arguments for filtering in and filtering out probes
bpftool: Update documentation of "bpftool feature" command
bpftool: Update bash completion for "bpftool feature" command
selftests/bpf: Add test for "bpftool feature" command
.../bpftool/Documentation/bpftool-feature.rst | 37 +-
tools/bpf/bpftool/bash-completion/bpftool | 32 +-
tools/bpf/bpftool/feature.c | 592 +++++++++++++-----
tools/testing/selftests/.gitignore | 5 +-
tools/testing/selftests/bpf/Makefile | 3 +-
tools/testing/selftests/bpf/test_bpftool.py | 294 +++++++++
tools/testing/selftests/bpf/test_bpftool.sh | 5 +
7 files changed, 811 insertions(+), 157 deletions(-)
create mode 100644 tools/testing/selftests/bpf/test_bpftool.py
create mode 100755 tools/testing/selftests/bpf/test_bpftool.sh
--
2.25.0
When running the ftrace selftests, 2 failures and 6 unresolved
cases were observed. The failures can be avoided by setting
a sysctl prior to test execution (fixed in patch 1) and the
unresolved cases result from absence of testing modules which
are built based on CONFIG options being set and program
availability (fixed in patch 2).
These seem more like "unsupported" than "unresolved" errors,
since for the ftrace tests "unresolved" cases cause the test
(and thus kselftest) to report failure. With these changes
in place, the unresolved cases become unsupported and the
test failures disappear, resulting in the ftracetest program
exiting with "ok" status.
Alan Maguire (2):
ftrace/selftests: workaround cgroup RT scheduling issues
ftrace/selftest: absence of modules/programs should trigger
unsupported errors
tools/testing/selftests/ftrace/ftracetest | 23 ++++++++++++++++++++++
.../ftrace/test.d/direct/ftrace-direct.tc | 2 +-
.../ftrace/test.d/direct/kprobe-direct.tc | 2 +-
.../selftests/ftrace/test.d/event/trace_printk.tc | 2 +-
.../ftrace/test.d/ftrace/func_mod_trace.tc | 2 +-
.../ftrace/test.d/kprobe/kprobe_module.tc | 2 +-
.../selftests/ftrace/test.d/selftest/bashisms.tc | 2 +-
7 files changed, 29 insertions(+), 6 deletions(-)
--
1.8.3.1
When kunit tests are run on native (i.e. non-UML) environments, the results
of test execution are often intermixed with dmesg output. This patch
series attempts to solve this by providing a debugfs representation
of the results of the last test run, available as
/sys/kernel/debug/kunit/<testsuite>/results
Changes since v2:
- updated kunit_status2str() to kunit_status_to_string() and made it
static inline in include/kunit/test.h (Brendan)
- added log string to struct kunit_suite and kunit_case, with log
pointer in struct kunit pointing at the case log. This allows us
to collect kunit_[err|info|warning]() messages at the same time
as we printk() them. This solves for the most part the sharing
of log messages between test execution and debugfs since we
just print the suite log (which contains the test suite preamble)
and the individual test logs. The only exception is the suite-level
status, which we cannot store in the suite log as it would mean
we'd print the suite and its status prior to the suite's results.
(Brendan, patch 1)
- dropped debugfs-based kunit run patch for now so as not to cause
problems with tests currently under development (Brendan)
- fixed doc issues with code block (Brendan, patch 3)
Changes since v1:
- trimmed unneeded include files in lib/kunit/debugfs.c (Greg)
- renamed global debugfs functions to be prefixed with kunit_ (Greg)
- removed error checking for debugfs operations (Greg)
Alan Maguire (2):
kunit: add debugfs /sys/kernel/debug/kunit/<suite>/results display
kunit: update documentation to describe debugfs representation
Documentation/dev-tools/kunit/usage.rst | 12 +++
include/kunit/test.h | 52 ++++++++++--
lib/kunit/Makefile | 3 +-
lib/kunit/debugfs.c | 105 ++++++++++++++++++++++++
lib/kunit/debugfs.h | 16 ++++
lib/kunit/kunit-test.c | 4 +-
lib/kunit/test.c | 136 ++++++++++++++++++++++++--------
7 files changed, 286 insertions(+), 42 deletions(-)
create mode 100644 lib/kunit/debugfs.c
create mode 100644 lib/kunit/debugfs.h
--
1.8.3.1