Hi,
Jan and Kirill: I've tentatively removed your review and ACK,
respectively, for patch 12 (the last dump_page patch), because even
though they are logically the same as what you reviewed in v5, the
base is Matthew's new patch instead of my earlier patch. (Trying to err
on the side of caution with these tags.)
There is a git repo and branch, for convenience in reviewing:
git@github.com:johnhubbard/linux.git track_user_pages_v6
============================================================
Changes since v5:
* Rebased onto Linux 5.6.0-rc1.
* Swapped in Matthew Wilcox's more comprehensive dump_page() patch, and
moved it later in this series so that it immediately precedes my
subsequent dump_page() patch, for slightly easier reviews and commit
log history.
* Fixed "the last bug!" in the /proc/vmstat patch, by moving the
mod_node_page_state() call in put_compound_page() so that it only
happens in the FOLL_PIN case.
* Added a couple more ACKs from Kirill.
* Tweaked the "Future steps" in this cover letter to add a little
detail about what comes next.
============================================================
Changes since v4:
* Added documentation about the huge page behavior of the new
/proc/vmstat items.
* Added a missing mode_node_page_state() call to put_compound_head().
* Fixed a tracepoint call in page_ref_sub_return().
* Added a trailing underscore to a URL in pin_user_pages.rst, to fix
a broken generated link.
* Added ACKs and reviewed-by's from Jan Kara and Kirill Shutemov.
* Rebased onto today's linux.git, and
* I am experimenting here with "git format-patch --base=<commit>".
This generated the "base-commit:" tag you'll see at the end of this
cover letter. I was inspired to do so after trying out a new
get-lore-mbox.py tool (it's very nice), mentioned in a recent LWN
article (https://lwn.net/Articles/811528/ ). That tool relies on the
base-commit tag for some things.
============================================================
Changes since v3:
* Rebased onto latest linux.git
* Added ACKs and reviewed-by's from Kirill Shutemov and Jan Kara.
* /proc/vmstat:
* Renamed items, after realizing that I hate the previous names:
nr_foll_pin_requested --> nr_foll_pin_acquired
nr_foll_pin_returned --> nr_foll_pin_released
* Removed the CONFIG_DEBUG_VM guard, and collapsed away a wrapper
routine: now just calls mod_node_page_state() directly.
* Tweaked the WARN_ON_ONCE() statements in mm/hugetlb.c to be more
informative, and added comments above them as well.
* Fixed gup_benchmark: signed int --> unsigned long.
* One or two minor formatting changes.
============================================================
Changes since v2:
* Rebased onto linux.git, because the akpm tree for 5.6 has been merged.
* Split the tracking patch into even more patches, as requested.
* Merged Matthew Wilcox's dump_page() changes into mine, as part of the
first patch.
* Renamed: page_dma_pinned() --> page_maybe_dma_pinned(), in response to
Kirill Shutemov's review.
* Moved a WARN to the top of a routine, and fixed a typo in the commit
description of patch #7, also as suggested by Kirill.
============================================================
Changes since v1:
* Split the tracking patch into 6 smaller patches
* Rebased onto today's linux-next/akpm (there weren't any conflicts).
* Fixed an "unsigned int" vs. "int" problem in gup_benchmark, reported
by Nathan Chancellor. (I don't see it in my local builds, probably
because they use gcc, but an LLVM test found the mismatch.)
* Fixed a huge page pincount problem (add/subtract vs.
increment/decrement), spotted by Jan Kara.
============================================================
There is a reasonable case to be made for merging two of the patches
(patches 7 and 8), given that patch 7 provides tracking that has upper
limits on the number of pins that can be done with huge pages. Let me
know if anyone wants those merged, but unless there is some weird chance
of someone grabbing patch 7 and not patch 8, I don't really see the
need. Meanwhile, it's easier to review in this form.
Also, patch 3 has been revived. Earlier reviewers asked for it to be
merged into the tracking patch (one cannot please everyone, heh), but
now it's back out on it's own.
This activates tracking of FOLL_PIN pages. This is in support of fixing
the get_user_pages()+DMA problem described in [1]-[4].
FOLL_PIN support is now in the main linux tree. However, the
patch to use FOLL_PIN to track pages was *not* submitted, because Leon
saw an RDMA test suite failure that involved (I think) page refcount
overflows when huge pages were used.
This patch definitively solves that kind of overflow problem, by adding
an exact pincount, for compound pages (of order > 1), in the 3rd struct
page of a compound page. If available, that form of pincounting is used,
instead of the GUP_PIN_COUNTING_BIAS approach. Thanks again to Jan Kara
for that idea.
Other interesting changes:
* dump_page(): added one, or two new things to report for compound
pages: head refcount (for all compound pages), and map_pincount (for
compound pages of order > 1).
* Documentation/core-api/pin_user_pages.rst: removed the "TODO" for the
huge page refcount upper limit problems, and added notes about how it
works now. Also added a note about the dump_page() enhancements.
* Added some comments in gup.c and mm.h, to explain that there are two
ways to count pinned pages: exact (for compound pages of order > 1)
and fuzzy (GUP_PIN_COUNTING_BIAS: for all other pages).
============================================================
General notes about the tracking patch:
This is a prerequisite to solving the problem of proper interactions
between file-backed pages, and [R]DMA activities, as discussed in [1],
[2], [3], [4] and in a remarkable number of email threads since about
2017. :)
In contrast to earlier approaches, the page tracking can be
incrementally applied to the kernel call sites that, until now, have
been simply calling get_user_pages() ("gup"). In other words, opt-in by
changing from this:
get_user_pages() (sets FOLL_GET)
put_page()
to this:
pin_user_pages() (sets FOLL_PIN)
unpin_user_page()
============================================================
Future steps:
* Convert more subsystems from get_user_pages() to pin_user_pages().
The first probably needs to be bio/biovecs, because any filesystem
testing is too difficult without those in place.
* Change VFS and filesystems to respond appropriately when encountering
dma-pinned pages.
* Work with Ira and others to connect this all up with file system
leases.
[1] Some slow progress on get_user_pages() (Apr 2, 2019):
https://lwn.net/Articles/784574/
[2] DMA and get_user_pages() (LPC: Dec 12, 2018):
https://lwn.net/Articles/774411/
[3] The trouble with get_user_pages() (Apr 30, 2018):
https://lwn.net/Articles/753027/
[4] LWN kernel index: get_user_pages()
https://lwn.net/Kernel/Index/#Memory_management-get_user_pages
John Hubbard (11):
mm/gup: split get_user_pages_remote() into two routines
mm/gup: pass a flags arg to __gup_device_* functions
mm: introduce page_ref_sub_return()
mm/gup: pass gup flags to two more routines
mm/gup: require FOLL_GET for get_user_pages_fast()
mm/gup: track FOLL_PIN pages
mm/gup: page->hpage_pinned_refcount: exact pin counts for huge pages
mm/gup: /proc/vmstat: pin_user_pages (FOLL_PIN) reporting
mm/gup_benchmark: support pin_user_pages() and related calls
selftests/vm: run_vmtests: invoke gup_benchmark with basic FOLL_PIN
coverage
mm: dump_page(): additional diagnostics for huge pinned pages
Matthew Wilcox (Oracle) (1):
mm: Improve dump_page() for compound pages
Documentation/core-api/pin_user_pages.rst | 86 ++--
include/linux/mm.h | 108 ++++-
include/linux/mm_types.h | 7 +-
include/linux/mmzone.h | 2 +
include/linux/page_ref.h | 9 +
mm/debug.c | 44 +-
mm/gup.c | 451 ++++++++++++++++-----
mm/gup_benchmark.c | 71 +++-
mm/huge_memory.c | 29 +-
mm/hugetlb.c | 60 ++-
mm/page_alloc.c | 2 +
mm/rmap.c | 6 +
mm/vmstat.c | 2 +
tools/testing/selftests/vm/gup_benchmark.c | 15 +-
tools/testing/selftests/vm/run_vmtests | 22 +
15 files changed, 734 insertions(+), 180 deletions(-)
base-commit: bb6d3fb354c5ee8d6bde2d576eb7220ea09862b9
--
2.25.0
This series adds basic self tests for HMM and are intended for Jason
Gunthorpe's rdma tree which has a number of HMM patches applied.
Changes v7 -> v8:
Rebased to Jason's rdma/hmm tree, plus Jason's 6 patch series
"Small hmm_range_fault() cleanups".
Applied a number of changes from Jason's comments.
Changes v6 -> v7:
Rebased to linux-5.6.0-rc6
Reverted back to just using mmu_interval_notifier_insert() and making
this series only introduce HMM self tests.
Changes v5 -> v6:
Rebased to linux-5.5.0-rc6
Refactored mmu interval notifier patches
Converted nouveau to use the new mmu interval notifier API
Changes v4 -> v5:
Added mmu interval notifier insert/remove/update callable from the
invalidate() callback
Updated HMM tests to use the new core interval notifier API
Changes v1 -> v4:
https://lore.kernel.org/linux-mm/20191104222141.5173-1-rcampbell@nvidia.com
Ralph Campbell (3):
mm/hmm/test: add selftest driver for HMM
mm/hmm/test: add selftests for HMM
MAINTAINERS: add HMM selftests
MAINTAINERS | 3 +
include/uapi/linux/test_hmm.h | 59 ++
lib/Kconfig.debug | 12 +
lib/Makefile | 1 +
lib/test_hmm.c | 1177 +++++++++++++++++++++
tools/testing/selftests/vm/.gitignore | 1 +
tools/testing/selftests/vm/Makefile | 3 +
tools/testing/selftests/vm/config | 2 +
tools/testing/selftests/vm/hmm-tests.c | 1353 ++++++++++++++++++++++++
tools/testing/selftests/vm/run_vmtests | 16 +
tools/testing/selftests/vm/test_hmm.sh | 97 ++
11 files changed, 2724 insertions(+)
create mode 100644 include/uapi/linux/test_hmm.h
create mode 100644 lib/test_hmm.c
create mode 100644 tools/testing/selftests/vm/hmm-tests.c
create mode 100755 tools/testing/selftests/vm/test_hmm.sh
--
2.20.1
When kunit tests are run on native (i.e. non-UML) environments, the results
of test execution are often intermixed with dmesg output. This patch
series attempts to solve this by providing a debugfs representation
of the results of the last test run, available as
/sys/kernel/debug/kunit/<testsuite>/results
Changes since v7:
- renamed KUNIT_INDENT[2] to KUNIT_SUBTEST_INDENT, KUNIT_SUBSUBTEST_INDENT
and added more description to their definitions to clarify why they
are defined as they are (Shuah)
- defined KUNIT_SUBSUBTEST_INDENT directly as 8 spaces to avoid
checkpatch error (Shuah)
Changes since v6:
- fixed regexp parsing in kunit_parser.py to ensure test results are read
successfully with 4-space indentation (Brendan, patch 3)
Changes since v5:
- replaced undefined behaviour use of snprintf(buf, ..., buf) in
kunit_log() with a function to append string to existing log
(Frank, patch 1)
- added clarification on log size limitations to documentation
(Frank, patch 4)
Changes since v4:
- added suite-level log expectations to kunit log test (Brendan, patch 2)
- added log expectations (of it being NULL) for case where
CONFIG_KUNIT_DEBUGFS=n to kunit log test (patch 2)
- added patch 3 which replaces subtest tab indentation with 4 space
indentation as per TAP 14 spec (Frank, patch 3)
Changes since v3:
- added CONFIG_KUNIT_DEBUGFS to support conditional compilation of debugfs
representation, including string logging (Frank, patch 1)
- removed unneeded NULL check for test_case in
kunit_suite_for_each_test_case() (Frank, patch 1)
- added kunit log test to verify logging multiple strings works
(Frank, patch 2)
- rephrased description of results file (Frank, patch 3)
Changes since v2:
- updated kunit_status2str() to kunit_status_to_string() and made it
static inline in include/kunit/test.h (Brendan)
- added log string to struct kunit_suite and kunit_case, with log
pointer in struct kunit pointing at the case log. This allows us
to collect kunit_[err|info|warning]() messages at the same time
as we printk() them. This solves for the most part the sharing
of log messages between test execution and debugfs since we
just print the suite log (which contains the test suite preamble)
and the individual test logs. The only exception is the suite-level
status, which we cannot store in the suite log as it would mean
we'd print the suite and its status prior to the suite's results.
(Brendan, patch 1)
- dropped debugfs-based kunit run patch for now so as not to cause
problems with tests currently under development (Brendan)
- fixed doc issues with code block (Brendan, patch 3)
Changes since v1:
- trimmed unneeded include files in lib/kunit/debugfs.c (Greg)
- renamed global debugfs functions to be prefixed with kunit_ (Greg)
- removed error checking for debugfs operations (Greg)
Alan Maguire (4):
kunit: add debugfs /sys/kernel/debug/kunit/<suite>/results display
kunit: add log test
kunit: subtests should be indented 4 spaces according to TAP
kunit: update documentation to describe debugfs representation
Documentation/dev-tools/kunit/usage.rst | 14 +++
include/kunit/test.h | 63 ++++++++++++--
lib/kunit/Kconfig | 8 ++
lib/kunit/Makefile | 4 +
lib/kunit/assert.c | 79 ++++++++---------
lib/kunit/debugfs.c | 116 +++++++++++++++++++++++++
lib/kunit/debugfs.h | 30 +++++++
lib/kunit/kunit-test.c | 44 +++++++++-
lib/kunit/test.c | 148 +++++++++++++++++++++++++-------
tools/testing/kunit/kunit_parser.py | 10 +--
10 files changed, 430 insertions(+), 86 deletions(-)
create mode 100644 lib/kunit/debugfs.c
create mode 100644 lib/kunit/debugfs.h
--
1.8.3.1
Hi!
Shuah please consider applying to the kselftest tree.
This set is an attempt to make running tests for different
sets of data easier. The direct motivation is the tls
test which we'd like to run for TLS 1.2 and TLS 1.3,
but currently there is no easy way to invoke the same
tests with different parameters.
Tested all users of kselftest_harness.h.
v2:
- don't run tests by fixture
- don't pass params as an explicit argument
v3:
- go back to the orginal implementation with an extra
parameter, and running by fixture (Kees);
- add LIST_APPEND helper (Kees);
- add a dot between fixture and param name (Kees);
- rename the params to variants (Tim);
v4:
- whitespace fixes.
v5 (Kees):
- move a comment;
- remove a temporary variable;
- reword the commit message on patch 4.
v1: https://lore.kernel.org/netdev/20200313031752.2332565-1-kuba@kernel.org/
v2: https://lore.kernel.org/netdev/20200314005501.2446494-1-kuba@kernel.org/
v3: https://lore.kernel.org/netdev/20200316225647.3129354-1-kuba@kernel.org/
v4: https://lore.kernel.org/netdev/20200317010419.3268916-1-kuba@kernel.org/
Jakub Kicinski (5):
kselftest: factor out list manipulation to a helper
kselftest: create fixture objects
kselftest: run tests by fixture
kselftest: add fixture variants
selftests: tls: run all tests for TLS 1.2 and TLS 1.3
Documentation/dev-tools/kselftest.rst | 3 +-
tools/testing/selftests/kselftest_harness.h | 234 +++++++++++++++-----
tools/testing/selftests/net/tls.c | 93 ++------
3 files changed, 202 insertions(+), 128 deletions(-)
--
2.25.1
The second patch was already posted independently but because
of the changes in the first patch, the second one now depends
on it. Hence posting it now as a part of this series.
The last version (v2) of the second patch can be found at:
https://patchwork.ozlabs.org/patch/1225969/
Sandipan Das (2):
selftests: vm: Do not override definition of ARCH
selftests: vm: Fix 64-bit test builds for powerpc64le
tools/testing/selftests/vm/Makefile | 4 ++--
tools/testing/selftests/vm/run_vmtests | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
--
2.17.1
Avoid using /usr/bin/timeout unnecessarily if timeout is set to 0
(disabled) in the "settings" file for a specific test.
NOTE: without this change (and adding timeout=0 in the corresponding
settings file - tools/testing/selftests/seccomp/settings) the
seccomp_bpf selftest is always failing with a timeout event during the
syscall_restart step.
Signed-off-by: Andrea Righi <andrea.righi(a)canonical.com>
---
tools/testing/selftests/kselftest/runner.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh
index e84d901f8567..2cd3c8def0f6 100644
--- a/tools/testing/selftests/kselftest/runner.sh
+++ b/tools/testing/selftests/kselftest/runner.sh
@@ -32,7 +32,7 @@ tap_prefix()
tap_timeout()
{
# Make sure tests will time out if utility is available.
- if [ -x /usr/bin/timeout ] ; then
+ if [ -x /usr/bin/timeout ] && [ $kselftest_timeout -gt 0 ] ; then
/usr/bin/timeout "$kselftest_timeout" "$1"
else
"$1"
--
2.25.1
From: Tim Bird <tim.bird(a)sony.com>
It is useful for CI systems to be able to query the list
of targets provided by kselftest by default, so that they
can construct their own loop over the targets if desired.
Signed-off-by: Tim Bird <tim.bird(a)sony.com>
---
tools/testing/selftests/Makefile | 3 +++
1 file changed, 3 insertions(+)
diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 63430e2..9955e71 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -246,4 +246,7 @@ clean:
$(MAKE) OUTPUT=$$BUILD_TARGET -C $$TARGET clean;\
done;
+show_targets:
+ @echo $(TARGETS)
+
.PHONY: khdr all run_tests hotplug run_hotplug clean_hotplug run_pstore_crash install clean
--
2.1.4
The following 4 tests in timers can take longer than the default 45
seconds that added in commit 852c8cbf (selftests/kselftest/runner.sh:
Add 45 second timeout per test) to run:
* nsleep-lat - 2m7.350s
* set-timer-lat - 2m0.66s
* inconsistency-check - 1m45.074s
* raw_skew - 2m0.013s
Thus they will be marked as failed with the current 45s setting:
not ok 3 selftests: timers: nsleep-lat # TIMEOUT
not ok 4 selftests: timers: set-timer-lat # TIMEOUT
not ok 6 selftests: timers: inconsistency-check # TIMEOUT
not ok 7 selftests: timers: raw_skew # TIMEOUT
Disable the timeout setting for timers can make these tests finish
properly:
ok 3 selftests: timers: nsleep-lat
ok 4 selftests: timers: set-timer-lat
ok 6 selftests: timers: inconsistency-check
ok 7 selftests: timers: raw_skew
https://bugs.launchpad.net/bugs/1864626
Signed-off-by: Po-Hsu Lin <po-hsu.lin(a)canonical.com>
---
tools/testing/selftests/timers/Makefile | 1 +
tools/testing/selftests/timers/settings | 1 +
2 files changed, 2 insertions(+)
create mode 100644 tools/testing/selftests/timers/settings
diff --git a/tools/testing/selftests/timers/Makefile b/tools/testing/selftests/timers/Makefile
index 7656c7c..0e73a16 100644
--- a/tools/testing/selftests/timers/Makefile
+++ b/tools/testing/selftests/timers/Makefile
@@ -13,6 +13,7 @@ DESTRUCTIVE_TESTS = alarmtimer-suspend valid-adjtimex adjtick change_skew \
TEST_GEN_PROGS_EXTENDED = $(DESTRUCTIVE_TESTS)
+TEST_FILES := settings
include ../lib.mk
diff --git a/tools/testing/selftests/timers/settings b/tools/testing/selftests/timers/settings
new file mode 100644
index 0000000..e7b9417
--- /dev/null
+++ b/tools/testing/selftests/timers/settings
@@ -0,0 +1 @@
+timeout=0
--
2.7.4