On top of mm-stable.
This is my current set of tests for testing COW handling of anonymous memory, especially when interacting with GUP. I developed these tests while working on PageAnonExclusive and managed to clean them up just now.
On current upstream Linux, all tests pass except the hugetlb tests that rely on vmsplice -- these tests should pass as soon as vmsplice properly uses FOLL_PIN instead of FOLL_GET.
I'm working on additional tests for COW handling in private mappings, focusing on long-term R/O pinning e.g., of the shared zeropage, pagecache pages and KSM pages. These tests, however, will go into a different file. So this is everything I have regarding tests for anonymous memory.
Cc: Andrew Morton akpm@linux-foundation.org Cc: Shuah Khan shuah@kernel.org Cc: Jason Gunthorpe jgg@nvidia.com Cc: John Hubbard jhubbard@nvidia.com Cc: Nadav Amit namit@vmware.com Cc: Peter Xu peterx@redhat.com Cc: Andrea Arcangeli aarcange@redhat.com Cc: Vlastimil Babka vbabka@suse.cz Cc: Mike Rapoport rppt@kernel.org Cc: Christoph von Recklinghausen crecklin@redhat.com Cc: Don Dutile ddutile@redhat.com
David Hildenbrand (7): selftests/vm: anon_cow: test COW handling of anonymous memory selftests/vm: factor out pagemap_is_populated() into vm_util selftests/vm: anon_cow: THP tests selftests/vm: anon_cow: hugetlb tests selftests/vm: anon_cow: add liburing test cases mm/gup_test: start/stop/read functionality for PIN LONGTERM test selftests/vm: anon_cow: add R/O longterm tests via gup_test
mm/gup_test.c | 140 +++ mm/gup_test.h | 12 + tools/testing/selftests/vm/.gitignore | 1 + tools/testing/selftests/vm/Makefile | 25 +- tools/testing/selftests/vm/anon_cow.c | 1126 ++++++++++++++++++++ tools/testing/selftests/vm/check_config.sh | 31 + tools/testing/selftests/vm/madv_populate.c | 8 - tools/testing/selftests/vm/run_vmtests.sh | 3 + tools/testing/selftests/vm/vm_util.c | 15 + tools/testing/selftests/vm/vm_util.h | 2 + 10 files changed, 1353 insertions(+), 10 deletions(-) create mode 100644 tools/testing/selftests/vm/anon_cow.c create mode 100644 tools/testing/selftests/vm/check_config.sh
Let's start adding tests for our COW handling of anonymous memory. We'll focus on basic tests that we can achieve without additional libraries or gup_test extensions.
We'll add THP and hugetlb tests separately.
Signed-off-by: David Hildenbrand david@redhat.com --- tools/testing/selftests/vm/.gitignore | 1 + tools/testing/selftests/vm/Makefile | 4 +- tools/testing/selftests/vm/anon_cow.c | 401 ++++++++++++++++++++++ tools/testing/selftests/vm/run_vmtests.sh | 3 + tools/testing/selftests/vm/vm_util.c | 7 + tools/testing/selftests/vm/vm_util.h | 1 + 6 files changed, 416 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/vm/anon_cow.c
diff --git a/tools/testing/selftests/vm/.gitignore b/tools/testing/selftests/vm/.gitignore index 7b9dc2426f18..8a536c731e3c 100644 --- a/tools/testing/selftests/vm/.gitignore +++ b/tools/testing/selftests/vm/.gitignore @@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only +anon_cow hugepage-mmap hugepage-mremap hugepage-shm diff --git a/tools/testing/selftests/vm/Makefile b/tools/testing/selftests/vm/Makefile index 4ae879f70f4c..bc759534ec28 100644 --- a/tools/testing/selftests/vm/Makefile +++ b/tools/testing/selftests/vm/Makefile @@ -25,7 +25,8 @@ MAKEFLAGS += --no-builtin-rules
CFLAGS = -Wall -I ../../../../usr/include $(EXTRA_CFLAGS) $(KHDR_INCLUDES) LDLIBS = -lrt -lpthread -TEST_GEN_FILES = compaction_test +TEST_GEN_FILES = anon_cow +TEST_GEN_FILES += compaction_test TEST_GEN_FILES += gup_test TEST_GEN_FILES += hmm-tests TEST_GEN_FILES += hugetlb-madvise @@ -95,6 +96,7 @@ TEST_FILES += va_128TBswitch.sh
include ../lib.mk
+$(OUTPUT)/anon_cow: vm_util.c $(OUTPUT)/madv_populate: vm_util.c $(OUTPUT)/soft-dirty: vm_util.c $(OUTPUT)/split_huge_page_test: vm_util.c diff --git a/tools/testing/selftests/vm/anon_cow.c b/tools/testing/selftests/vm/anon_cow.c new file mode 100644 index 000000000000..9d2b15c829e6 --- /dev/null +++ b/tools/testing/selftests/vm/anon_cow.c @@ -0,0 +1,401 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * COW (Copy On Write) tests for anonymous memory. + * + * Copyright 2022, Red Hat, Inc. + * + * Author(s): David Hildenbrand david@redhat.com + */ +#define _GNU_SOURCE +#include <stdlib.h> +#include <string.h> +#include <stdbool.h> +#include <stdint.h> +#include <unistd.h> +#include <errno.h> +#include <fcntl.h> +#include <dirent.h> +#include <assert.h> +#include <sys/mman.h> +#include <sys/wait.h> + +#include "../kselftest.h" +#include "vm_util.h" + +static size_t pagesize; +static int pagemap_fd; + +struct comm_pipes { + int child_ready[2]; + int parent_ready[2]; +}; + +static int setup_comm_pipes(struct comm_pipes *comm_pipes) +{ + if (pipe(comm_pipes->child_ready) < 0) + return -errno; + if (pipe(comm_pipes->parent_ready) < 0) { + close(comm_pipes->child_ready[0]); + close(comm_pipes->child_ready[1]); + return -errno; + } + + return 0; +} + +static void close_comm_pipes(struct comm_pipes *comm_pipes) +{ + close(comm_pipes->child_ready[0]); + close(comm_pipes->child_ready[1]); + close(comm_pipes->parent_ready[0]); + close(comm_pipes->parent_ready[1]); +} + +static int child_memcmp_fn(char *mem, size_t size, + struct comm_pipes *comm_pipes) +{ + char *old = malloc(size); + char buf; + + /* Backup the original content. */ + memcpy(old, mem, size); + + /* Wait until the parent modified the page. */ + write(comm_pipes->child_ready[1], "0", 1); + while (read(comm_pipes->parent_ready[0], &buf, 1) != 1) + ; + + /* See if we still read the old values. */ + return memcmp(old, mem, size); +} + +static int child_vmsplice_memcmp_fn(char *mem, size_t size, + struct comm_pipes *comm_pipes) +{ + struct iovec iov = { + .iov_base = mem, + .iov_len = size, + }; + size_t cur, total, transferred; + char *old, *new; + int fds[2]; + char buf; + + old = malloc(size); + new = malloc(size); + + /* Backup the original content. */ + memcpy(old, mem, size); + + if (pipe(fds) < 0) + return -errno; + + /* Trigger a read-only pin. */ + transferred = vmsplice(fds[1], &iov, 1, 0); + if (transferred < 0) + return -errno; + if (transferred == 0) + return -EINVAL; + + /* Unmap it from our page tables. */ + if (munmap(mem, size) < 0) + return -errno; + + /* Wait until the parent modified it. */ + write(comm_pipes->child_ready[1], "0", 1); + while (read(comm_pipes->parent_ready[0], &buf, 1) != 1) + ; + + /* See if we still read the old values via the pipe. */ + for (total = 0; total < transferred; total += cur) { + cur = read(fds[0], new + total, transferred - total); + if (cur < 0) + return -errno; + } + + return memcmp(old, new, transferred); +} + +typedef int (*child_fn)(char *mem, size_t size, struct comm_pipes *comm_pipes); + +static void do_test_cow_in_parent(char *mem, size_t size, child_fn fn) +{ + struct comm_pipes comm_pipes; + char buf; + int ret; + + ret = setup_comm_pipes(&comm_pipes); + if (ret) { + ksft_test_result_fail("pipe() failed\n"); + return; + } + + ret = fork(); + if (ret < 0) { + ksft_test_result_fail("fork() failed\n"); + goto close_comm_pipes; + } else if (!ret) { + exit(fn(mem, size, &comm_pipes)); + } + + while (read(comm_pipes.child_ready[0], &buf, 1) != 1) + ; + /* Modify the page. */ + memset(mem, 0xff, size); + write(comm_pipes.parent_ready[1], "0", 1); + + wait(&ret); + if (WIFEXITED(ret)) + ret = WEXITSTATUS(ret); + else + ret = -EINVAL; + + ksft_test_result(!ret, "No leak from parent into child\n"); +close_comm_pipes: + close_comm_pipes(&comm_pipes); +} + +static void test_cow_in_parent(char *mem, size_t size) +{ + do_test_cow_in_parent(mem, size, child_memcmp_fn); +} + +static void test_vmsplice_in_child(char *mem, size_t size) +{ + do_test_cow_in_parent(mem, size, child_vmsplice_memcmp_fn); +} + +static void do_test_vmsplice_in_parent(char *mem, size_t size, + bool before_fork) +{ + struct iovec iov = { + .iov_base = mem, + .iov_len = size, + }; + size_t cur, total, transferred; + struct comm_pipes comm_pipes; + char *old, *new; + int ret, fds[2]; + char buf; + + old = malloc(size); + new = malloc(size); + + memcpy(old, mem, size); + + ret = setup_comm_pipes(&comm_pipes); + if (ret) { + ksft_test_result_fail("pipe() failed\n"); + goto free; + } + + if (pipe(fds) < 0) { + ksft_test_result_fail("pipe() failed\n"); + goto close_comm_pipes; + } + + if (before_fork) { + transferred = vmsplice(fds[1], &iov, 1, 0); + if (transferred <= 0) { + ksft_test_result_fail("vmsplice() failed\n"); + goto close_pipe; + } + } + + ret = fork(); + if (ret < 0) { + ksft_test_result_fail("fork() failed\n"); + goto close_pipe; + } else if (!ret) { + write(comm_pipes.child_ready[1], "0", 1); + while (read(comm_pipes.parent_ready[0], &buf, 1) != 1) + ; + /* Modify page content in the child. */ + memset(mem, 0xff, size); + exit(0); + } + + if (!before_fork) { + transferred = vmsplice(fds[1], &iov, 1, 0); + if (transferred <= 0) { + ksft_test_result_fail("vmsplice() failed\n"); + wait(&ret); + goto close_pipe; + } + } + + while (read(comm_pipes.child_ready[0], &buf, 1) != 1) + ; + if (munmap(mem, size) < 0) { + ksft_test_result_fail("munmap() failed\n"); + goto close_pipe; + } + write(comm_pipes.parent_ready[1], "0", 1); + + /* Wait until the child is done writing. */ + wait(&ret); + if (!WIFEXITED(ret)) { + ksft_test_result_fail("wait() failed\n"); + goto close_pipe; + } + + /* See if we still read the old values. */ + for (total = 0; total < transferred; total += cur) { + cur = read(fds[0], new + total, transferred - total); + if (cur < 0) { + ksft_test_result_fail("read() failed\n"); + goto close_pipe; + } + } + + ksft_test_result(!memcmp(old, new, transferred), + "No leak from child into parent\n"); +close_pipe: + close(fds[0]); + close(fds[1]); +close_comm_pipes: + close_comm_pipes(&comm_pipes); +free: + free(old); + free(new); +} + +static void test_vmsplice_before_fork(char *mem, size_t size) +{ + do_test_vmsplice_in_parent(mem, size, true); +} + +static void test_vmsplice_after_fork(char *mem, size_t size) +{ + do_test_vmsplice_in_parent(mem, size, false); +} + +typedef void (*test_fn)(char *mem, size_t size); + +static void do_run_with_base_page(test_fn fn, bool swapout) +{ + char *mem; + int ret; + + mem = mmap(NULL, pagesize, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (mem == MAP_FAILED) { + ksft_test_result_fail("mmap() failed\n"); + return; + } + + ret = madvise(mem, pagesize, MADV_NOHUGEPAGE); + /* Ignore if not around on a kernel. */ + if (ret && errno != EINVAL) { + ksft_test_result_fail("MADV_NOHUGEPAGE failed\n"); + goto munmap; + } + + /* Populate a base page. */ + memset(mem, 0, pagesize); + + if (swapout) { + madvise(mem, pagesize, MADV_PAGEOUT); + if (!pagemap_is_swapped(pagemap_fd, mem)) { + ksft_test_result_skip("MADV_PAGEOUT did not work, is swap enabled?\n"); + goto munmap; + } + } + + fn(mem, pagesize); +munmap: + munmap(mem, pagesize); +} + +static void run_with_base_page(test_fn fn, const char *desc) +{ + ksft_print_msg("[RUN] %s ... with base page\n", desc); + do_run_with_base_page(fn, false); +} + +static void run_with_base_page_swap(test_fn fn, const char *desc) +{ + ksft_print_msg("[RUN] %s ... with swapped out base page\n", desc); + do_run_with_base_page(fn, true); +} + +struct test_case { + const char *desc; + test_fn fn; +}; + +static const struct test_case test_cases[] = { + /* + * Basic COW tests for fork() without any GUP. If we miss to break COW, + * either the child can observe modifications by the parent or the + * other way around. + */ + { + "Basic COW after fork()", + test_cow_in_parent, + }, + /* + * vmsplice() [R/O GUP] + unmap in the child; modify in the parent. If + * we miss to break COW, the child observes modifications by the parent. + * This is CVE-2020-29374 reported by Jann Horn. + */ + { + "vmsplice() + unmap in child", + test_vmsplice_in_child + }, + /* + * vmsplice() [R/O GUP] in parent before fork(), unmap in parent after + * fork(); modify in the child. If we miss to break COW, the parent + * observes modifications by the child. + */ + { + "vmsplice() before fork(), unmap in parent after fork()", + test_vmsplice_before_fork, + }, + /* + * vmsplice() [R/O GUP] + unmap in parent after fork(); modify in the + * child. If we miss to break COW, the parent observes modifications by + * the child. + */ + { + "vmsplice() + unmap in parent after fork()", + test_vmsplice_after_fork, + }, +}; + +static void run_test_case(struct test_case const *test_case) +{ + run_with_base_page(test_case->fn, test_case->desc); + run_with_base_page_swap(test_case->fn, test_case->desc); +} + +static void run_test_cases(void) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(test_cases); i++) + run_test_case(&test_cases[i]); +} + +int main(int argc, char **argv) +{ + int nr_test_cases = ARRAY_SIZE(test_cases); + int err; + + pagesize = getpagesize(); + + ksft_print_header(); + ksft_set_plan(nr_test_cases * 2); + + pagemap_fd = open("/proc/self/pagemap", O_RDONLY); + if (pagemap_fd < 0) + ksft_exit_fail_msg("opening pagemap failed\n"); + + run_test_cases(); + + err = ksft_get_fail_cnt(); + if (err) + ksft_exit_fail_msg("%d out of %d tests failed\n", + err, ksft_test_num()); + return ksft_exit_pass(); +} diff --git a/tools/testing/selftests/vm/run_vmtests.sh b/tools/testing/selftests/vm/run_vmtests.sh index e780e76c26b8..af59380bc254 100755 --- a/tools/testing/selftests/vm/run_vmtests.sh +++ b/tools/testing/selftests/vm/run_vmtests.sh @@ -197,4 +197,7 @@ fi
run_test ./soft-dirty
+# COW tests for anonymous memory +run_test ./anon_cow + exit $exitcode diff --git a/tools/testing/selftests/vm/vm_util.c b/tools/testing/selftests/vm/vm_util.c index b58ab11a7a30..6456027e1985 100644 --- a/tools/testing/selftests/vm/vm_util.c +++ b/tools/testing/selftests/vm/vm_util.c @@ -28,6 +28,13 @@ bool pagemap_is_softdirty(int fd, char *start) return entry & 0x0080000000000000ull; }
+bool pagemap_is_swapped(int fd, char *start) +{ + uint64_t entry = pagemap_get_entry(fd, start); + + return entry & 0x4000000000000000ull; +} + void clear_softdirty(void) { int ret; diff --git a/tools/testing/selftests/vm/vm_util.h b/tools/testing/selftests/vm/vm_util.h index 2e512bd57ae1..bb8ec8d69623 100644 --- a/tools/testing/selftests/vm/vm_util.h +++ b/tools/testing/selftests/vm/vm_util.h @@ -4,6 +4,7 @@
uint64_t pagemap_get_entry(int fd, char *start); bool pagemap_is_softdirty(int fd, char *start); +bool pagemap_is_swapped(int fd, char *start); void clear_softdirty(void); uint64_t read_pmd_pagesize(void); uint64_t check_huge(void *addr);
On Tue, Sep 27, 2022 at 01:01:14PM +0200, David Hildenbrand wrote:
Let's start adding tests for our COW handling of anonymous memory. We'll focus on basic tests that we can achieve without additional libraries or gup_test extensions.
We'll add THP and hugetlb tests separately.
Signed-off-by: David Hildenbrand david@redhat.com
tools/testing/selftests/vm/.gitignore | 1 + tools/testing/selftests/vm/Makefile | 4 +- tools/testing/selftests/vm/anon_cow.c | 401 ++++++++++++++++++++++ tools/testing/selftests/vm/run_vmtests.sh | 3 + tools/testing/selftests/vm/vm_util.c | 7 + tools/testing/selftests/vm/vm_util.h | 1 + 6 files changed, 416 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/vm/anon_cow.c
diff --git a/tools/testing/selftests/vm/.gitignore b/tools/testing/selftests/vm/.gitignore index 7b9dc2426f18..8a536c731e3c 100644 --- a/tools/testing/selftests/vm/.gitignore +++ b/tools/testing/selftests/vm/.gitignore @@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only +anon_cow hugepage-mmap hugepage-mremap hugepage-shm diff --git a/tools/testing/selftests/vm/Makefile b/tools/testing/selftests/vm/Makefile index 4ae879f70f4c..bc759534ec28 100644 --- a/tools/testing/selftests/vm/Makefile +++ b/tools/testing/selftests/vm/Makefile @@ -25,7 +25,8 @@ MAKEFLAGS += --no-builtin-rules CFLAGS = -Wall -I ../../../../usr/include $(EXTRA_CFLAGS) $(KHDR_INCLUDES) LDLIBS = -lrt -lpthread -TEST_GEN_FILES = compaction_test +TEST_GEN_FILES = anon_cow +TEST_GEN_FILES += compaction_test TEST_GEN_FILES += gup_test TEST_GEN_FILES += hmm-tests TEST_GEN_FILES += hugetlb-madvise @@ -95,6 +96,7 @@ TEST_FILES += va_128TBswitch.sh include ../lib.mk +$(OUTPUT)/anon_cow: vm_util.c $(OUTPUT)/madv_populate: vm_util.c $(OUTPUT)/soft-dirty: vm_util.c $(OUTPUT)/split_huge_page_test: vm_util.c diff --git a/tools/testing/selftests/vm/anon_cow.c b/tools/testing/selftests/vm/anon_cow.c new file mode 100644 index 000000000000..9d2b15c829e6 --- /dev/null +++ b/tools/testing/selftests/vm/anon_cow.c @@ -0,0 +1,401 @@ +// SPDX-License-Identifier: GPL-2.0-only +/*
- COW (Copy On Write) tests for anonymous memory.
- Copyright 2022, Red Hat, Inc.
- Author(s): David Hildenbrand david@redhat.com
- */
+#define _GNU_SOURCE +#include <stdlib.h> +#include <string.h> +#include <stdbool.h> +#include <stdint.h> +#include <unistd.h> +#include <errno.h> +#include <fcntl.h> +#include <dirent.h> +#include <assert.h> +#include <sys/mman.h> +#include <sys/wait.h>
+#include "../kselftest.h" +#include "vm_util.h"
+static size_t pagesize; +static int pagemap_fd;
+struct comm_pipes {
- int child_ready[2];
- int parent_ready[2];
+};
+static int setup_comm_pipes(struct comm_pipes *comm_pipes) +{
- if (pipe(comm_pipes->child_ready) < 0)
return -errno;- if (pipe(comm_pipes->parent_ready) < 0) {
close(comm_pipes->child_ready[0]);close(comm_pipes->child_ready[1]);return -errno;- }
- return 0;
+}
+static void close_comm_pipes(struct comm_pipes *comm_pipes) +{
- close(comm_pipes->child_ready[0]);
- close(comm_pipes->child_ready[1]);
- close(comm_pipes->parent_ready[0]);
- close(comm_pipes->parent_ready[1]);
+}
+static int child_memcmp_fn(char *mem, size_t size,
struct comm_pipes *comm_pipes)+{
- char *old = malloc(size);
- char buf;
- /* Backup the original content. */
- memcpy(old, mem, size);
- /* Wait until the parent modified the page. */
- write(comm_pipes->child_ready[1], "0", 1);
- while (read(comm_pipes->parent_ready[0], &buf, 1) != 1)
;- /* See if we still read the old values. */
- return memcmp(old, mem, size);
+}
+static int child_vmsplice_memcmp_fn(char *mem, size_t size,
struct comm_pipes *comm_pipes)+{
- struct iovec iov = {
.iov_base = mem,.iov_len = size,- };
- size_t cur, total, transferred;
- char *old, *new;
- int fds[2];
- char buf;
- old = malloc(size);
- new = malloc(size);
- /* Backup the original content. */
- memcpy(old, mem, size);
- if (pipe(fds) < 0)
return -errno;- /* Trigger a read-only pin. */
- transferred = vmsplice(fds[1], &iov, 1, 0);
- if (transferred < 0)
return -errno;- if (transferred == 0)
return -EINVAL;- /* Unmap it from our page tables. */
- if (munmap(mem, size) < 0)
return -errno;- /* Wait until the parent modified it. */
- write(comm_pipes->child_ready[1], "0", 1);
- while (read(comm_pipes->parent_ready[0], &buf, 1) != 1)
;- /* See if we still read the old values via the pipe. */
- for (total = 0; total < transferred; total += cur) {
cur = read(fds[0], new + total, transferred - total);if (cur < 0)
Hi David, I was looking at some coccinelle reports for linux-next and saw the following warning for this comparison:
WARNING: Unsigned expression compared with zero: cur < 0
I think 'cur' needs to be of type 'ssize_t' for this comparison to work.
The same warning is reported for the variable 'transferred' above, and also for do_test_iouring() and do_test_vmsplice_in_parent() in "selftests/vm: anon_cow: add liburing test cases".
return -errno;- }
- return memcmp(old, new, transferred);
+}
+typedef int (*child_fn)(char *mem, size_t size, struct comm_pipes *comm_pipes);
+static void do_test_cow_in_parent(char *mem, size_t size, child_fn fn) +{
- struct comm_pipes comm_pipes;
- char buf;
- int ret;
- ret = setup_comm_pipes(&comm_pipes);
- if (ret) {
ksft_test_result_fail("pipe() failed\n");return;- }
- ret = fork();
- if (ret < 0) {
ksft_test_result_fail("fork() failed\n");goto close_comm_pipes;- } else if (!ret) {
exit(fn(mem, size, &comm_pipes));- }
- while (read(comm_pipes.child_ready[0], &buf, 1) != 1)
;- /* Modify the page. */
- memset(mem, 0xff, size);
- write(comm_pipes.parent_ready[1], "0", 1);
- wait(&ret);
- if (WIFEXITED(ret))
ret = WEXITSTATUS(ret);- else
ret = -EINVAL;- ksft_test_result(!ret, "No leak from parent into child\n");
+close_comm_pipes:
- close_comm_pipes(&comm_pipes);
+}
+static void test_cow_in_parent(char *mem, size_t size) +{
- do_test_cow_in_parent(mem, size, child_memcmp_fn);
+}
+static void test_vmsplice_in_child(char *mem, size_t size) +{
- do_test_cow_in_parent(mem, size, child_vmsplice_memcmp_fn);
+}
+static void do_test_vmsplice_in_parent(char *mem, size_t size,
bool before_fork)+{
- struct iovec iov = {
.iov_base = mem,.iov_len = size,- };
- size_t cur, total, transferred;
- struct comm_pipes comm_pipes;
- char *old, *new;
- int ret, fds[2];
- char buf;
- old = malloc(size);
- new = malloc(size);
- memcpy(old, mem, size);
- ret = setup_comm_pipes(&comm_pipes);
- if (ret) {
ksft_test_result_fail("pipe() failed\n");goto free;- }
- if (pipe(fds) < 0) {
ksft_test_result_fail("pipe() failed\n");goto close_comm_pipes;- }
- if (before_fork) {
transferred = vmsplice(fds[1], &iov, 1, 0);if (transferred <= 0) {ksft_test_result_fail("vmsplice() failed\n");goto close_pipe;}- }
- ret = fork();
- if (ret < 0) {
ksft_test_result_fail("fork() failed\n");goto close_pipe;- } else if (!ret) {
write(comm_pipes.child_ready[1], "0", 1);while (read(comm_pipes.parent_ready[0], &buf, 1) != 1);/* Modify page content in the child. */memset(mem, 0xff, size);exit(0);- }
- if (!before_fork) {
transferred = vmsplice(fds[1], &iov, 1, 0);if (transferred <= 0) {ksft_test_result_fail("vmsplice() failed\n");wait(&ret);goto close_pipe;}- }
- while (read(comm_pipes.child_ready[0], &buf, 1) != 1)
;- if (munmap(mem, size) < 0) {
ksft_test_result_fail("munmap() failed\n");goto close_pipe;- }
- write(comm_pipes.parent_ready[1], "0", 1);
- /* Wait until the child is done writing. */
- wait(&ret);
- if (!WIFEXITED(ret)) {
ksft_test_result_fail("wait() failed\n");goto close_pipe;- }
- /* See if we still read the old values. */
- for (total = 0; total < transferred; total += cur) {
cur = read(fds[0], new + total, transferred - total);if (cur < 0) {ksft_test_result_fail("read() failed\n");goto close_pipe;}- }
- ksft_test_result(!memcmp(old, new, transferred),
"No leak from child into parent\n");+close_pipe:
- close(fds[0]);
- close(fds[1]);
+close_comm_pipes:
- close_comm_pipes(&comm_pipes);
+free:
- free(old);
- free(new);
+}
+static void test_vmsplice_before_fork(char *mem, size_t size) +{
- do_test_vmsplice_in_parent(mem, size, true);
+}
+static void test_vmsplice_after_fork(char *mem, size_t size) +{
- do_test_vmsplice_in_parent(mem, size, false);
+}
+typedef void (*test_fn)(char *mem, size_t size);
+static void do_run_with_base_page(test_fn fn, bool swapout) +{
- char *mem;
- int ret;
- mem = mmap(NULL, pagesize, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);- if (mem == MAP_FAILED) {
ksft_test_result_fail("mmap() failed\n");return;- }
- ret = madvise(mem, pagesize, MADV_NOHUGEPAGE);
- /* Ignore if not around on a kernel. */
- if (ret && errno != EINVAL) {
ksft_test_result_fail("MADV_NOHUGEPAGE failed\n");goto munmap;- }
- /* Populate a base page. */
- memset(mem, 0, pagesize);
- if (swapout) {
madvise(mem, pagesize, MADV_PAGEOUT);if (!pagemap_is_swapped(pagemap_fd, mem)) {ksft_test_result_skip("MADV_PAGEOUT did not work, is swap enabled?\n");goto munmap;}- }
- fn(mem, pagesize);
+munmap:
- munmap(mem, pagesize);
+}
+static void run_with_base_page(test_fn fn, const char *desc) +{
- ksft_print_msg("[RUN] %s ... with base page\n", desc);
- do_run_with_base_page(fn, false);
+}
+static void run_with_base_page_swap(test_fn fn, const char *desc) +{
- ksft_print_msg("[RUN] %s ... with swapped out base page\n", desc);
- do_run_with_base_page(fn, true);
+}
+struct test_case {
- const char *desc;
- test_fn fn;
+};
+static const struct test_case test_cases[] = {
- /*
* Basic COW tests for fork() without any GUP. If we miss to break COW,* either the child can observe modifications by the parent or the* other way around.*/- {
"Basic COW after fork()",test_cow_in_parent,- },
- /*
* vmsplice() [R/O GUP] + unmap in the child; modify in the parent. If* we miss to break COW, the child observes modifications by the parent.* This is CVE-2020-29374 reported by Jann Horn.*/- {
"vmsplice() + unmap in child",test_vmsplice_in_child- },
- /*
* vmsplice() [R/O GUP] in parent before fork(), unmap in parent after* fork(); modify in the child. If we miss to break COW, the parent* observes modifications by the child.*/- {
"vmsplice() before fork(), unmap in parent after fork()",test_vmsplice_before_fork,- },
- /*
* vmsplice() [R/O GUP] + unmap in parent after fork(); modify in the* child. If we miss to break COW, the parent observes modifications by* the child.*/- {
"vmsplice() + unmap in parent after fork()",test_vmsplice_after_fork,- },
+};
+static void run_test_case(struct test_case const *test_case) +{
- run_with_base_page(test_case->fn, test_case->desc);
- run_with_base_page_swap(test_case->fn, test_case->desc);
+}
+static void run_test_cases(void) +{
- int i;
- for (i = 0; i < ARRAY_SIZE(test_cases); i++)
run_test_case(&test_cases[i]);+}
+int main(int argc, char **argv) +{
- int nr_test_cases = ARRAY_SIZE(test_cases);
- int err;
- pagesize = getpagesize();
- ksft_print_header();
- ksft_set_plan(nr_test_cases * 2);
- pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
- if (pagemap_fd < 0)
ksft_exit_fail_msg("opening pagemap failed\n");- run_test_cases();
- err = ksft_get_fail_cnt();
- if (err)
ksft_exit_fail_msg("%d out of %d tests failed\n",err, ksft_test_num());- return ksft_exit_pass();
+} diff --git a/tools/testing/selftests/vm/run_vmtests.sh b/tools/testing/selftests/vm/run_vmtests.sh index e780e76c26b8..af59380bc254 100755 --- a/tools/testing/selftests/vm/run_vmtests.sh +++ b/tools/testing/selftests/vm/run_vmtests.sh @@ -197,4 +197,7 @@ fi run_test ./soft-dirty +# COW tests for anonymous memory +run_test ./anon_cow
exit $exitcode diff --git a/tools/testing/selftests/vm/vm_util.c b/tools/testing/selftests/vm/vm_util.c index b58ab11a7a30..6456027e1985 100644 --- a/tools/testing/selftests/vm/vm_util.c +++ b/tools/testing/selftests/vm/vm_util.c @@ -28,6 +28,13 @@ bool pagemap_is_softdirty(int fd, char *start) return entry & 0x0080000000000000ull; } +bool pagemap_is_swapped(int fd, char *start) +{
- uint64_t entry = pagemap_get_entry(fd, start);
- return entry & 0x4000000000000000ull;
+}
void clear_softdirty(void) { int ret; diff --git a/tools/testing/selftests/vm/vm_util.h b/tools/testing/selftests/vm/vm_util.h index 2e512bd57ae1..bb8ec8d69623 100644 --- a/tools/testing/selftests/vm/vm_util.h +++ b/tools/testing/selftests/vm/vm_util.h @@ -4,6 +4,7 @@ uint64_t pagemap_get_entry(int fd, char *start); bool pagemap_is_softdirty(int fd, char *start); +bool pagemap_is_swapped(int fd, char *start); void clear_softdirty(void); uint64_t read_pmd_pagesize(void); uint64_t check_huge(void *addr); -- 2.37.3
- /* See if we still read the old values via the pipe. */
- for (total = 0; total < transferred; total += cur) {
cur = read(fds[0], new + total, transferred - total);if (cur < 0)Hi David, I was looking at some coccinelle reports for linux-next and saw the following warning for this comparison:
WARNING: Unsigned expression compared with zero: cur < 0
I think 'cur' needs to be of type 'ssize_t' for this comparison to work.
The same warning is reported for the variable 'transferred' above, and also for do_test_iouring() and do_test_vmsplice_in_parent() in "selftests/vm: anon_cow: add liburing test cases".
Thanks for reporting! Indeed, we need a signed value.
On 27.09.22 13:01, David Hildenbrand wrote:
Let's start adding tests for our COW handling of anonymous memory. We'll focus on basic tests that we can achieve without additional libraries or gup_test extensions.
We'll add THP and hugetlb tests separately.
Signed-off-by: David Hildenbrand david@redhat.com
The following fixup on top:
diff --git a/tools/testing/selftests/vm/anon_cow.c b/tools/testing/selftests/vm/anon_cow.c index 9d2b15c829e6..4613294af758 100644 --- a/tools/testing/selftests/vm/anon_cow.c +++ b/tools/testing/selftests/vm/anon_cow.c @@ -76,7 +76,7 @@ static int child_vmsplice_memcmp_fn(char *mem, size_t size, .iov_base = mem, .iov_len = size, }; - size_t cur, total, transferred; + ssize_t cur, total, transferred; char *old, *new; int fds[2]; char buf; @@ -172,7 +172,7 @@ static void do_test_vmsplice_in_parent(char *mem, size_t size, .iov_base = mem, .iov_len = size, }; - size_t cur, total, transferred; + ssize_t cur, total, transferred; struct comm_pipes comm_pipes; char *old, *new; int ret, fds[2];
We'll reuse it in the anon_cow test next.
Signed-off-by: David Hildenbrand david@redhat.com --- tools/testing/selftests/vm/madv_populate.c | 8 -------- tools/testing/selftests/vm/vm_util.c | 8 ++++++++ tools/testing/selftests/vm/vm_util.h | 1 + 3 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/tools/testing/selftests/vm/madv_populate.c b/tools/testing/selftests/vm/madv_populate.c index 715a42e8e2cd..60547245e479 100644 --- a/tools/testing/selftests/vm/madv_populate.c +++ b/tools/testing/selftests/vm/madv_populate.c @@ -27,14 +27,6 @@
static size_t pagesize;
-static bool pagemap_is_populated(int fd, char *start) -{ - uint64_t entry = pagemap_get_entry(fd, start); - - /* Present or swapped. */ - return entry & 0xc000000000000000ull; -} - static void sense_support(void) { char *addr; diff --git a/tools/testing/selftests/vm/vm_util.c b/tools/testing/selftests/vm/vm_util.c index 6456027e1985..d37dbabeb875 100644 --- a/tools/testing/selftests/vm/vm_util.c +++ b/tools/testing/selftests/vm/vm_util.c @@ -35,6 +35,14 @@ bool pagemap_is_swapped(int fd, char *start) return entry & 0x4000000000000000ull; }
+bool pagemap_is_populated(int fd, char *start) +{ + uint64_t entry = pagemap_get_entry(fd, start); + + /* Present or swapped. */ + return entry & 0xc000000000000000ull; +} + void clear_softdirty(void) { int ret; diff --git a/tools/testing/selftests/vm/vm_util.h b/tools/testing/selftests/vm/vm_util.h index bb8ec8d69623..691d063e79cc 100644 --- a/tools/testing/selftests/vm/vm_util.h +++ b/tools/testing/selftests/vm/vm_util.h @@ -5,6 +5,7 @@ uint64_t pagemap_get_entry(int fd, char *start); bool pagemap_is_softdirty(int fd, char *start); bool pagemap_is_swapped(int fd, char *start); +bool pagemap_is_populated(int fd, char *start); void clear_softdirty(void); uint64_t read_pmd_pagesize(void); uint64_t check_huge(void *addr);
Let's add various THP variants that we'll run with our existing test cases.
Signed-off-by: David Hildenbrand david@redhat.com --- tools/testing/selftests/vm/anon_cow.c | 259 +++++++++++++++++++++++++- 1 file changed, 258 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/vm/anon_cow.c b/tools/testing/selftests/vm/anon_cow.c index 9d2b15c829e6..9244c8551a95 100644 --- a/tools/testing/selftests/vm/anon_cow.c +++ b/tools/testing/selftests/vm/anon_cow.c @@ -24,6 +24,43 @@
static size_t pagesize; static int pagemap_fd; +static size_t thpsize; + +static void detect_thpsize(void) +{ + int fd = open("/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", + O_RDONLY); + size_t size = 0; + char buf[15]; + int ret; + + if (fd < 0) + return; + + ret = pread(fd, buf, sizeof(buf), 0); + if (ret > 0 && ret < sizeof(buf)) { + buf[ret] = 0; + + size = strtoul(buf, NULL, 10); + if (size < pagesize) + size = 0; + if (size > 0) { + thpsize = size; + ksft_print_msg("[INFO] detected THP size: %zu KiB\n", + thpsize / 1024); + } + } + + close(fd); +} + +static bool range_is_swapped(void *addr, size_t size) +{ + for (; size; addr += pagesize, size -= pagesize) + if (!pagemap_is_swapped(pagemap_fd, addr)) + return false; + return true; +}
struct comm_pipes { int child_ready[2]; @@ -319,6 +356,206 @@ static void run_with_base_page_swap(test_fn fn, const char *desc) do_run_with_base_page(fn, true); }
+enum thp_run { + THP_RUN_PMD, + THP_RUN_PMD_SWAPOUT, + THP_RUN_PTE, + THP_RUN_PTE_SWAPOUT, + THP_RUN_SINGLE_PTE, + THP_RUN_SINGLE_PTE_SWAPOUT, + THP_RUN_PARTIAL_MREMAP, + THP_RUN_PARTIAL_SHARED, +}; + +static void do_run_with_thp(test_fn fn, enum thp_run thp_run) +{ + char *mem, *mmap_mem, *tmp, *mremap_mem = MAP_FAILED; + size_t size, mmap_size, mremap_size; + int ret; + + /* For alignment purposes, we need twice the thp size. */ + mmap_size = 2 * thpsize; + mmap_mem = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (mmap_mem == MAP_FAILED) { + ksft_test_result_fail("mmap() failed\n"); + return; + } + + /* We need a THP-aligned memory area. */ + mem = (char *)(((uintptr_t)mmap_mem + thpsize) & ~(thpsize - 1)); + + ret = madvise(mem, thpsize, MADV_HUGEPAGE); + if (ret) { + ksft_test_result_fail("MADV_HUGEPAGE failed\n"); + goto munmap; + } + + /* + * Try to populate a THP. Touch the first sub-page and test if we get + * another sub-page populated automatically. + */ + mem[0] = 0; + if (!pagemap_is_populated(pagemap_fd, mem + pagesize)) { + ksft_test_result_skip("Did not get a THP populated\n"); + goto munmap; + } + memset(mem, 0, thpsize); + + size = thpsize; + switch (thp_run) { + case THP_RUN_PMD: + case THP_RUN_PMD_SWAPOUT: + break; + case THP_RUN_PTE: + case THP_RUN_PTE_SWAPOUT: + /* + * Trigger PTE-mapping the THP by temporarily mapping a single + * subpage R/O. + */ + ret = mprotect(mem + pagesize, pagesize, PROT_READ); + if (ret) { + ksft_test_result_fail("mprotect() failed\n"); + goto munmap; + } + ret = mprotect(mem + pagesize, pagesize, PROT_READ | PROT_WRITE); + if (ret) { + ksft_test_result_fail("mprotect() failed\n"); + goto munmap; + } + break; + case THP_RUN_SINGLE_PTE: + case THP_RUN_SINGLE_PTE_SWAPOUT: + /* + * Discard all but a single subpage of that PTE-mapped THP. What + * remains is a single PTE mapping a single subpage. + */ + ret = madvise(mem + pagesize, thpsize - pagesize, MADV_DONTNEED); + if (ret) { + ksft_test_result_fail("MADV_DONTNEED failed\n"); + goto munmap; + } + size = pagesize; + break; + case THP_RUN_PARTIAL_MREMAP: + /* + * Remap half of the THP. We need some new memory location + * for that. + */ + mremap_size = thpsize / 2; + mremap_mem = mmap(NULL, mremap_size, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (mem == MAP_FAILED) { + ksft_test_result_fail("mmap() failed\n"); + goto munmap; + } + tmp = mremap(mem + mremap_size, mremap_size, mremap_size, + MREMAP_MAYMOVE | MREMAP_FIXED, mremap_mem); + if (tmp != mremap_mem) { + ksft_test_result_fail("mremap() failed\n"); + goto munmap; + } + size = mremap_size; + break; + case THP_RUN_PARTIAL_SHARED: + /* + * Share the first page of the THP with a child and quit the + * child. This will result in some parts of the THP never + * have been shared. + */ + ret = madvise(mem + pagesize, thpsize - pagesize, MADV_DONTFORK); + if (ret) { + ksft_test_result_fail("MADV_DONTFORK failed\n"); + goto munmap; + } + ret = fork(); + if (ret < 0) { + ksft_test_result_fail("fork() failed\n"); + goto munmap; + } else if (!ret) { + exit(0); + } + wait(&ret); + /* Allow for sharing all pages again. */ + ret = madvise(mem + pagesize, thpsize - pagesize, MADV_DOFORK); + if (ret) { + ksft_test_result_fail("MADV_DOFORK failed\n"); + goto munmap; + } + break; + default: + assert(false); + } + + switch (thp_run) { + case THP_RUN_PMD_SWAPOUT: + case THP_RUN_PTE_SWAPOUT: + case THP_RUN_SINGLE_PTE_SWAPOUT: + madvise(mem, size, MADV_PAGEOUT); + if (!range_is_swapped(mem, size)) { + ksft_test_result_skip("MADV_PAGEOUT did not work, is swap enabled?\n"); + goto munmap; + } + break; + default: + break; + } + + fn(mem, size); +munmap: + munmap(mmap_mem, mmap_size); + if (mremap_mem != MAP_FAILED) + munmap(mremap_mem, mremap_size); +} + +static void run_with_thp(test_fn fn, const char *desc) +{ + ksft_print_msg("[RUN] %s ... with THP\n", desc); + do_run_with_thp(fn, THP_RUN_PMD); +} + +static void run_with_thp_swap(test_fn fn, const char *desc) +{ + ksft_print_msg("[RUN] %s ... with swapped-out THP\n", desc); + do_run_with_thp(fn, THP_RUN_PMD_SWAPOUT); +} + +static void run_with_pte_mapped_thp(test_fn fn, const char *desc) +{ + ksft_print_msg("[RUN] %s ... with PTE-mapped THP\n", desc); + do_run_with_thp(fn, THP_RUN_PTE); +} + +static void run_with_pte_mapped_thp_swap(test_fn fn, const char *desc) +{ + ksft_print_msg("[RUN] %s ... with swapped-out, PTE-mapped THP\n", desc); + do_run_with_thp(fn, THP_RUN_PTE_SWAPOUT); +} + +static void run_with_single_pte_of_thp(test_fn fn, const char *desc) +{ + ksft_print_msg("[RUN] %s ... with single PTE of THP\n", desc); + do_run_with_thp(fn, THP_RUN_SINGLE_PTE); +} + +static void run_with_single_pte_of_thp_swap(test_fn fn, const char *desc) +{ + ksft_print_msg("[RUN] %s ... with single PTE of swapped-out THP\n", desc); + do_run_with_thp(fn, THP_RUN_SINGLE_PTE_SWAPOUT); +} + +static void run_with_partial_mremap_thp(test_fn fn, const char *desc) +{ + ksft_print_msg("[RUN] %s ... with partially mremap()'ed THP\n", desc); + do_run_with_thp(fn, THP_RUN_PARTIAL_MREMAP); +} + +static void run_with_partial_shared_thp(test_fn fn, const char *desc) +{ + ksft_print_msg("[RUN] %s ... with partially shared THP\n", desc); + do_run_with_thp(fn, THP_RUN_PARTIAL_SHARED); +} + struct test_case { const char *desc; test_fn fn; @@ -367,6 +604,16 @@ static void run_test_case(struct test_case const *test_case) { run_with_base_page(test_case->fn, test_case->desc); run_with_base_page_swap(test_case->fn, test_case->desc); + if (thpsize) { + run_with_thp(test_case->fn, test_case->desc); + run_with_thp_swap(test_case->fn, test_case->desc); + run_with_pte_mapped_thp(test_case->fn, test_case->desc); + run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc); + run_with_single_pte_of_thp(test_case->fn, test_case->desc); + run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc); + run_with_partial_mremap_thp(test_case->fn, test_case->desc); + run_with_partial_shared_thp(test_case->fn, test_case->desc); + } }
static void run_test_cases(void) @@ -377,15 +624,25 @@ static void run_test_cases(void) run_test_case(&test_cases[i]); }
+static int tests_per_test_case(void) +{ + int tests = 2; + + if (thpsize) + tests += 8; + return tests; +} + int main(int argc, char **argv) { int nr_test_cases = ARRAY_SIZE(test_cases); int err;
pagesize = getpagesize(); + detect_thpsize();
ksft_print_header(); - ksft_set_plan(nr_test_cases * 2); + ksft_set_plan(nr_test_cases * tests_per_test_case());
pagemap_fd = open("/proc/self/pagemap", O_RDONLY); if (pagemap_fd < 0)
Let's run all existing test cases with all hugetlb sizes we're able to detect.
Note that some tests cases still fail. This will, for example, be fixed once vmsplice properly uses FOLL_PIN instead of FOLL_GET for pinning. With 2 MiB and 1 GiB hugetlb on x86_64, the expected failures are:
# [RUN] vmsplice() + unmap in child ... with hugetlb (2048 kB) not ok 23 No leak from parent into child # [RUN] vmsplice() + unmap in child ... with hugetlb (1048576 kB) not ok 24 No leak from parent into child # [RUN] vmsplice() before fork(), unmap in parent after fork() ... with hugetlb (2048 kB) not ok 35 No leak from child into parent # [RUN] vmsplice() before fork(), unmap in parent after fork() ... with hugetlb (1048576 kB) not ok 36 No leak from child into parent # [RUN] vmsplice() + unmap in parent after fork() ... with hugetlb (2048 kB) not ok 47 No leak from child into parent # [RUN] vmsplice() + unmap in parent after fork() ... with hugetlb (1048576 kB) not ok 48 No leak from child into parent
Signed-off-by: David Hildenbrand david@redhat.com --- tools/testing/selftests/vm/anon_cow.c | 70 ++++++++++++++++++++++++++- 1 file changed, 69 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/vm/anon_cow.c b/tools/testing/selftests/vm/anon_cow.c index 9244c8551a95..06fd046bf1f2 100644 --- a/tools/testing/selftests/vm/anon_cow.c +++ b/tools/testing/selftests/vm/anon_cow.c @@ -25,6 +25,8 @@ static size_t pagesize; static int pagemap_fd; static size_t thpsize; +static int nr_hugetlbsizes; +static size_t hugetlbsizes[10];
static void detect_thpsize(void) { @@ -54,6 +56,31 @@ static void detect_thpsize(void) close(fd); }
+static void detect_hugetlbsizes(void) +{ + DIR *dir = opendir("/sys/kernel/mm/hugepages/"); + + if (!dir) + return; + + while (nr_hugetlbsizes < ARRAY_SIZE(hugetlbsizes)) { + struct dirent *entry = readdir(dir); + size_t kb; + + if (!entry) + break; + if (entry->d_type != DT_DIR) + continue; + if (sscanf(entry->d_name, "hugepages-%zukB", &kb) != 1) + continue; + hugetlbsizes[nr_hugetlbsizes] = kb * 1024; + nr_hugetlbsizes++; + ksft_print_msg("[INFO] detected hugetlb size: %zu KiB\n", + kb); + } + closedir(dir); +} + static bool range_is_swapped(void *addr, size_t size) { for (; size; addr += pagesize, size -= pagesize) @@ -556,6 +583,41 @@ static void run_with_partial_shared_thp(test_fn fn, const char *desc) do_run_with_thp(fn, THP_RUN_PARTIAL_SHARED); }
+static void run_with_hugetlb(test_fn fn, const char *desc, size_t hugetlbsize) +{ + int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB; + char *mem, *dummy; + + ksft_print_msg("[RUN] %s ... with hugetlb (%zu kB)\n", desc, + hugetlbsize / 1024); + + flags |= __builtin_ctzll(hugetlbsize) << MAP_HUGE_SHIFT; + + mem = mmap(NULL, hugetlbsize, PROT_READ | PROT_WRITE, flags, -1, 0); + if (mem == MAP_FAILED) { + ksft_test_result_skip("need more free huge pages\n"); + return; + } + + /* Populate an huge page. */ + memset(mem, 0, hugetlbsize); + + /* + * We need a total of two hugetlb pages to handle COW/unsharing + * properly, otherwise we might get zapped by a SIGBUS. + */ + dummy = mmap(NULL, hugetlbsize, PROT_READ | PROT_WRITE, flags, -1, 0); + if (dummy == MAP_FAILED) { + ksft_test_result_skip("need more free huge pages\n"); + goto munmap; + } + munmap(dummy, hugetlbsize); + + fn(mem, hugetlbsize); +munmap: + munmap(mem, hugetlbsize); +} + struct test_case { const char *desc; test_fn fn; @@ -602,6 +664,8 @@ static const struct test_case test_cases[] = {
static void run_test_case(struct test_case const *test_case) { + int i; + run_with_base_page(test_case->fn, test_case->desc); run_with_base_page_swap(test_case->fn, test_case->desc); if (thpsize) { @@ -614,6 +678,9 @@ static void run_test_case(struct test_case const *test_case) run_with_partial_mremap_thp(test_case->fn, test_case->desc); run_with_partial_shared_thp(test_case->fn, test_case->desc); } + for (i = 0; i < nr_hugetlbsizes; i++) + run_with_hugetlb(test_case->fn, test_case->desc, + hugetlbsizes[i]); }
static void run_test_cases(void) @@ -626,7 +693,7 @@ static void run_test_cases(void)
static int tests_per_test_case(void) { - int tests = 2; + int tests = 2 + nr_hugetlbsizes;
if (thpsize) tests += 8; @@ -640,6 +707,7 @@ int main(int argc, char **argv)
pagesize = getpagesize(); detect_thpsize(); + detect_hugetlbsizes();
ksft_print_header(); ksft_set_plan(nr_test_cases * tests_per_test_case());
io_uring provides a simple mechanism to test long-term, R/W GUP pins -- via fixed buffers -- and can be used to verify that GUP pins stay in sync with the pages in the page table even if a page would temporarily get mapped R/O or concurrent fork() could accidentially end up sharing pinned pages with the child.
Note that this essentially re-introduces local_config support that was removed recently in commit 6f83d6c74ea5 ("Kselftests: remove support of libhugetlbfs from kselftests").
Signed-off-by: David Hildenbrand david@redhat.com --- tools/testing/selftests/vm/Makefile | 21 ++- tools/testing/selftests/vm/anon_cow.c | 190 +++++++++++++++++++++ tools/testing/selftests/vm/check_config.sh | 31 ++++ 3 files changed, 241 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/vm/check_config.sh
diff --git a/tools/testing/selftests/vm/Makefile b/tools/testing/selftests/vm/Makefile index bc759534ec28..7931b84040ed 100644 --- a/tools/testing/selftests/vm/Makefile +++ b/tools/testing/selftests/vm/Makefile @@ -1,7 +1,9 @@ # SPDX-License-Identifier: GPL-2.0 # Makefile for vm selftests
-LOCAL_HDRS += $(top_srcdir)/mm/gup_test.h +LOCAL_HDRS += $(selfdir)/vm/local_config.h $(top_srcdir)/mm/gup_test.h + +include local_config.mk
uname_M := $(shell uname -m 2>/dev/null || echo not) MACHINE ?= $(shell echo $(uname_M) | sed -e 's/aarch64.*/arm64/' -e 's/ppc64.*/ppc64/') @@ -150,8 +152,25 @@ warn_32bit_failure: endif endif
+# ANON_COW_EXTRA_LIBS may get set in local_config.mk, or it may be left empty. +$(OUTPUT)/anon_cow: LDLIBS += $(ANON_COW_EXTRA_LIBS) + $(OUTPUT)/mlock-random-test $(OUTPUT)/memfd_secret: LDLIBS += -lcap
$(OUTPUT)/ksm_tests: LDLIBS += -lnuma
$(OUTPUT)/migration: LDLIBS += -lnuma + +local_config.mk local_config.h: check_config.sh + /bin/sh ./check_config.sh $(CC) + +EXTRA_CLEAN += local_config.mk local_config.h + +ifeq ($(ANON_COW_EXTRA_LIBS),) +all: warn_missing_liburing + +warn_missing_liburing: + @echo ; \ + echo "Warning: missing liburing support. Some COW tests will be skipped." ; \ + echo +endif diff --git a/tools/testing/selftests/vm/anon_cow.c b/tools/testing/selftests/vm/anon_cow.c index 06fd046bf1f2..369b3e15647d 100644 --- a/tools/testing/selftests/vm/anon_cow.c +++ b/tools/testing/selftests/vm/anon_cow.c @@ -19,6 +19,11 @@ #include <sys/mman.h> #include <sys/wait.h>
+#include "local_config.h" +#ifdef LOCAL_CONFIG_HAVE_LIBURING +#include <liburing.h> +#endif /* LOCAL_CONFIG_HAVE_LIBURING */ + #include "../kselftest.h" #include "vm_util.h"
@@ -334,6 +339,170 @@ static void test_vmsplice_after_fork(char *mem, size_t size) do_test_vmsplice_in_parent(mem, size, false); }
+#ifdef LOCAL_CONFIG_HAVE_LIBURING +static void do_test_iouring(char *mem, size_t size, bool use_fork) +{ + struct comm_pipes comm_pipes; + struct io_uring_cqe *cqe; + struct io_uring_sqe *sqe; + struct io_uring ring; + size_t cur, total; + struct iovec iov; + char *buf, *tmp; + int ret, fd; + FILE *file; + + ret = setup_comm_pipes(&comm_pipes); + if (ret) { + ksft_test_result_fail("pipe() failed\n"); + return; + } + + file = tmpfile(); + if (!file) { + ksft_test_result_fail("tmpfile() failed\n"); + goto close_comm_pipes; + } + fd = fileno(file); + assert(fd); + + tmp = malloc(size); + if (!tmp) { + ksft_test_result_fail("malloc() failed\n"); + goto close_file; + } + + /* Skip on errors, as we might just lack kernel support. */ + ret = io_uring_queue_init(1, &ring, 0); + if (ret < 0) { + ksft_test_result_skip("io_uring_queue_init() failed\n"); + goto free_tmp; + } + + /* + * Register the range as a fixed buffer. This will FOLL_WRITE | FOLL_PIN + * | FOLL_LONGTERM the range. + * + * Skip on errors, as we might just lack kernel support or might not + * have sufficient MEMLOCK permissions. + */ + iov.iov_base = mem; + iov.iov_len = size; + ret = io_uring_register_buffers(&ring, &iov, 1); + if (ret) { + ksft_test_result_skip("io_uring_register_buffers() failed\n"); + goto queue_exit; + } + + if (use_fork) { + /* + * fork() and keep the child alive until we're done. Note that + * we expect the pinned page to not get shared with the child. + */ + ret = fork(); + if (ret < 0) { + ksft_test_result_fail("fork() failed\n"); + goto unregister_buffers; + } else if (!ret) { + write(comm_pipes.child_ready[1], "0", 1); + while (read(comm_pipes.parent_ready[0], &buf, 1) != 1) + ; + exit(0); + } + + while (read(comm_pipes.child_ready[0], &buf, 1) != 1) + ; + } else { + /* + * Map the page R/O into the page table. Enable softdirty + * tracking to stop the page from getting mapped R/W immediately + * again by mprotect() optimizations. Note that we don't have an + * easy way to test if that worked (the pagemap does not export + * if the page is mapped R/O vs. R/W). + */ + ret = mprotect(mem, size, PROT_READ); + clear_softdirty(); + ret |= mprotect(mem, size, PROT_READ | PROT_WRITE); + if (ret) { + ksft_test_result_fail("mprotect() failed\n"); + goto unregister_buffers; + } + } + + /* + * Modify the page and write page content as observed by the fixed + * buffer pin to the file so we can verify it. + */ + memset(mem, 0xff, size); + sqe = io_uring_get_sqe(&ring); + if (!sqe) { + ksft_test_result_fail("io_uring_get_sqe() failed\n"); + goto quit_child; + } + io_uring_prep_write_fixed(sqe, fd, mem, size, 0, 0); + + ret = io_uring_submit(&ring); + if (ret < 0) { + ksft_test_result_fail("io_uring_submit() failed\n"); + goto quit_child; + } + + ret = io_uring_wait_cqe(&ring, &cqe); + if (ret < 0) { + ksft_test_result_fail("io_uring_wait_cqe() failed\n"); + goto quit_child; + } + + if (cqe->res != size) { + ksft_test_result_fail("write_fixed failed\n"); + goto quit_child; + } + io_uring_cqe_seen(&ring, cqe); + + /* Read back the file content to the temporary buffer. */ + total = 0; + while (total < size) { + cur = pread(fd, tmp + total, size - total, total); + if (cur < 0) { + ksft_test_result_fail("pread() failed\n"); + goto quit_child; + } + total += cur; + } + + /* Finally, check if we read what we expected. */ + ksft_test_result(!memcmp(mem, tmp, size), + "Longterm R/W pin is reliable\n"); + +quit_child: + if (use_fork) { + write(comm_pipes.parent_ready[1], "0", 1); + wait(&ret); + } +unregister_buffers: + io_uring_unregister_buffers(&ring); +queue_exit: + io_uring_queue_exit(&ring); +free_tmp: + free(tmp); +close_file: + fclose(file); +close_comm_pipes: + close_comm_pipes(&comm_pipes); +} + +static void test_iouring_ro(char *mem, size_t size) +{ + do_test_iouring(mem, size, false); +} + +static void test_iouring_fork(char *mem, size_t size) +{ + do_test_iouring(mem, size, true); +} + +#endif /* LOCAL_CONFIG_HAVE_LIBURING */ + typedef void (*test_fn)(char *mem, size_t size);
static void do_run_with_base_page(test_fn fn, bool swapout) @@ -660,6 +829,27 @@ static const struct test_case test_cases[] = { "vmsplice() + unmap in parent after fork()", test_vmsplice_after_fork, }, +#ifdef LOCAL_CONFIG_HAVE_LIBURING + /* + * Take a R/W longterm pin and then map the page R/O into the page + * table to trigger a write fault on next access. When modifying the + * page, the page content must be visible via the pin. + */ + { + "R/O-mapping a page registered as iouring fixed buffer", + test_iouring_ro, + }, + /* + * Take a R/W longterm pin and then fork() a child. When modifying the + * page, the page content must be visible via the pin. We expect the + * pinned page to not get shared with the child. + */ + { + "fork() with an iouring fixed buffer", + test_iouring_fork, + }, + +#endif /* LOCAL_CONFIG_HAVE_LIBURING */ };
static void run_test_case(struct test_case const *test_case) diff --git a/tools/testing/selftests/vm/check_config.sh b/tools/testing/selftests/vm/check_config.sh new file mode 100644 index 000000000000..9a44c6520925 --- /dev/null +++ b/tools/testing/selftests/vm/check_config.sh @@ -0,0 +1,31 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# +# Probe for libraries and create header files to record the results. Both C +# header files and Makefile include fragments are created. + +OUTPUT_H_FILE=local_config.h +OUTPUT_MKFILE=local_config.mk + +tmpname=$(mktemp) +tmpfile_c=${tmpname}.c +tmpfile_o=${tmpname}.o + +# liburing +echo "#include <sys/types.h>" > $tmpfile_c +echo "#include <liburing.h>" >> $tmpfile_c +echo "int func(void) { return 0; }" >> $tmpfile_c + +CC=${1:?"Usage: $0 <compiler> # example compiler: gcc"} +$CC -c $tmpfile_c -o $tmpfile_o >/dev/null 2>&1 + +if [ -f $tmpfile_o ]; then + echo "#define LOCAL_CONFIG_HAVE_LIBURING 1" > $OUTPUT_H_FILE + echo "ANON_COW_EXTRA_LIBS = -luring" > $OUTPUT_MKFILE +else + echo "// No liburing support found" > $OUTPUT_H_FILE + echo "# No liburing support found, so:" > $OUTPUT_MKFILE + echo "ANON_COW_EXTRA_LIBS = " >> $OUTPUT_MKFILE +fi + +rm ${tmpname}.*
On 27.09.22 13:01, David Hildenbrand wrote:
io_uring provides a simple mechanism to test long-term, R/W GUP pins -- via fixed buffers -- and can be used to verify that GUP pins stay in sync with the pages in the page table even if a page would temporarily get mapped R/O or concurrent fork() could accidentially end up sharing pinned pages with the child.
Note that this essentially re-introduces local_config support that was removed recently in commit 6f83d6c74ea5 ("Kselftests: remove support of libhugetlbfs from kselftests").
Signed-off-by: David Hildenbrand david@redhat.com
The following fixup on top:
diff --git a/tools/testing/selftests/vm/anon_cow.c b/tools/testing/selftests/vm/anon_cow.c index c1450a5d6f53..01417a604eda 100644 --- a/tools/testing/selftests/vm/anon_cow.c +++ b/tools/testing/selftests/vm/anon_cow.c @@ -346,7 +346,7 @@ static void do_test_iouring(char *mem, size_t size, bool use_fork) struct io_uring_cqe *cqe; struct io_uring_sqe *sqe; struct io_uring ring; - size_t cur, total; + ssize_t cur, total; struct iovec iov; char *buf, *tmp; int ret, fd;
We want an easy way to take a R/O or R/W longterm pin on a range and be able to observe the content of the pinned pages, so we can properly test how longterm puns interact with our COW logic.
Signed-off-by: David Hildenbrand david@redhat.com --- mm/gup_test.c | 140 ++++++++++++++++++++++++++++++++++++++++++++++++++ mm/gup_test.h | 12 +++++ 2 files changed, 152 insertions(+)
diff --git a/mm/gup_test.c b/mm/gup_test.c index 12b0a91767d3..2bbfacd31f5a 100644 --- a/mm/gup_test.c +++ b/mm/gup_test.c @@ -203,6 +203,134 @@ static int __gup_test_ioctl(unsigned int cmd, return ret; }
+static DEFINE_MUTEX(pin_longterm_test_mutex); +static struct page **pin_longterm_test_pages; +static unsigned long pin_longterm_test_nr_pages; + +static inline void pin_longterm_test_stop(void) +{ + if (pin_longterm_test_pages) { + if (pin_longterm_test_nr_pages) + unpin_user_pages(pin_longterm_test_pages, + pin_longterm_test_nr_pages); + kfree(pin_longterm_test_pages); + pin_longterm_test_pages = NULL; + pin_longterm_test_nr_pages = 0; + } +} + +static inline int pin_longterm_test_start(unsigned long arg) +{ + long nr_pages, cur_pages, addr, remaining_pages; + int gup_flags = FOLL_LONGTERM; + struct pin_longterm_test args; + struct page **pages; + int ret = 0; + bool fast; + + if (pin_longterm_test_pages) + return -EINVAL; + + if (copy_from_user(&args, (void __user *)arg, sizeof(args))) + return -EFAULT; + + if (args.flags & + ~(PIN_LONGTERM_TEST_FLAG_USE_WRITE|PIN_LONGTERM_TEST_FLAG_USE_FAST)) + return -EINVAL; + if (!IS_ALIGNED(args.addr | args.size, PAGE_SIZE)) + return -EINVAL; + if (args.size > LONG_MAX) + return -EINVAL; + nr_pages = args.size / PAGE_SIZE; + if (!nr_pages) + return -EINVAL; + + pages = kvcalloc(nr_pages, sizeof(void *), GFP_KERNEL); + if (!pages) + return -ENOMEM; + + if (args.flags & PIN_LONGTERM_TEST_FLAG_USE_WRITE) + gup_flags |= FOLL_WRITE; + fast = !!(args.flags & PIN_LONGTERM_TEST_FLAG_USE_FAST); + + if (!fast && mmap_read_lock_killable(current->mm)) { + kfree(pages); + return -EINTR; + } + + pin_longterm_test_pages = pages; + pin_longterm_test_nr_pages = 0; + + while (nr_pages - pin_longterm_test_nr_pages) { + remaining_pages = nr_pages - pin_longterm_test_nr_pages; + addr = args.addr + pin_longterm_test_nr_pages * PAGE_SIZE; + + if (fast) + cur_pages = pin_user_pages_fast(addr, remaining_pages, + gup_flags, pages); + else + cur_pages = pin_user_pages(addr, remaining_pages, + gup_flags, pages, NULL); + if (cur_pages < 0) { + pin_longterm_test_stop(); + ret = cur_pages; + break; + } + pin_longterm_test_nr_pages += cur_pages; + pages += cur_pages; + }; + + if (!fast) + mmap_read_unlock(current->mm); + return ret; +} + +static inline int pin_longterm_test_read(unsigned long arg) +{ + __u64 user_addr; + unsigned long i; + + if (!pin_longterm_test_pages) + return -EINVAL; + + if (copy_from_user(&user_addr, (void __user *)arg, sizeof(user_addr))) + return -EFAULT; + + for (i = 0; i < pin_longterm_test_nr_pages; i++) { + void *addr = page_to_virt(pin_longterm_test_pages[i]); + + if (copy_to_user((void __user *)user_addr, addr, PAGE_SIZE)) + return -EFAULT; + user_addr += PAGE_SIZE; + } + return 0; +} + +static long pin_longterm_test_ioctl(struct file *filep, unsigned int cmd, + unsigned long arg) +{ + int ret = -EINVAL; + + if (mutex_lock_killable(&pin_longterm_test_mutex)) + return -EINTR; + + switch (cmd) { + case PIN_LONGTERM_TEST_START: + ret = pin_longterm_test_start(arg); + break; + case PIN_LONGTERM_TEST_STOP: + pin_longterm_test_stop(); + ret = 0; + break; + case PIN_LONGTERM_TEST_READ: + ret = pin_longterm_test_read(arg); + break; + } + + mutex_unlock(&pin_longterm_test_mutex); + return ret; +} + static long gup_test_ioctl(struct file *filep, unsigned int cmd, unsigned long arg) { @@ -217,6 +345,10 @@ static long gup_test_ioctl(struct file *filep, unsigned int cmd, case PIN_BASIC_TEST: case DUMP_USER_PAGES_TEST: break; + case PIN_LONGTERM_TEST_START: + case PIN_LONGTERM_TEST_STOP: + case PIN_LONGTERM_TEST_READ: + return pin_longterm_test_ioctl(filep, cmd, arg); default: return -EINVAL; } @@ -234,9 +366,17 @@ static long gup_test_ioctl(struct file *filep, unsigned int cmd, return 0; }
+static int gup_test_release(struct inode *inode, struct file *file) +{ + pin_longterm_test_stop(); + + return 0; +} + static const struct file_operations gup_test_fops = { .open = nonseekable_open, .unlocked_ioctl = gup_test_ioctl, + .release = gup_test_release, };
static int __init gup_test_init(void) diff --git a/mm/gup_test.h b/mm/gup_test.h index 887ac1d5f5bc..5b37b54e8bea 100644 --- a/mm/gup_test.h +++ b/mm/gup_test.h @@ -10,6 +10,9 @@ #define GUP_BASIC_TEST _IOWR('g', 4, struct gup_test) #define PIN_BASIC_TEST _IOWR('g', 5, struct gup_test) #define DUMP_USER_PAGES_TEST _IOWR('g', 6, struct gup_test) +#define PIN_LONGTERM_TEST_START _IOW('g', 7, struct pin_longterm_test) +#define PIN_LONGTERM_TEST_STOP _IO('g', 8) +#define PIN_LONGTERM_TEST_READ _IOW('g', 9, __u64)
#define GUP_TEST_MAX_PAGES_TO_DUMP 8
@@ -30,4 +33,13 @@ struct gup_test { __u32 which_pages[GUP_TEST_MAX_PAGES_TO_DUMP]; };
+#define PIN_LONGTERM_TEST_FLAG_USE_WRITE 1 +#define PIN_LONGTERM_TEST_FLAG_USE_FAST 2 + +struct pin_longterm_test { + __u64 addr; + __u64 size; + __u32 flags; +}; + #endif /* __GUP_TEST_H */
Let's trigger a R/O longterm pin on three cases of R/O mapped anonymous pages: * exclusive (never shared) * shared (child still alive) * previously shared (child no longer alive)
... and make sure that the pin is reliable: whatever we write via the page tables has to be observable via the pin.
Signed-off-by: David Hildenbrand david@redhat.com --- tools/testing/selftests/vm/anon_cow.c | 210 ++++++++++++++++++++++++++ 1 file changed, 210 insertions(+)
diff --git a/tools/testing/selftests/vm/anon_cow.c b/tools/testing/selftests/vm/anon_cow.c index 369b3e15647d..055bf13a05d9 100644 --- a/tools/testing/selftests/vm/anon_cow.c +++ b/tools/testing/selftests/vm/anon_cow.c @@ -17,6 +17,7 @@ #include <dirent.h> #include <assert.h> #include <sys/mman.h> +#include <sys/ioctl.h> #include <sys/wait.h>
#include "local_config.h" @@ -24,6 +25,7 @@ #include <liburing.h> #endif /* LOCAL_CONFIG_HAVE_LIBURING */
+#include "../../../../mm/gup_test.h" #include "../kselftest.h" #include "vm_util.h"
@@ -32,6 +34,7 @@ static int pagemap_fd; static size_t thpsize; static int nr_hugetlbsizes; static size_t hugetlbsizes[10]; +static int gup_fd;
static void detect_thpsize(void) { @@ -503,6 +506,170 @@ static void test_iouring_fork(char *mem, size_t size)
#endif /* LOCAL_CONFIG_HAVE_LIBURING */
+enum ro_pin_test { + RO_PIN_TEST_SHARED, + RO_PIN_TEST_PREVIOUSLY_SHARED, + RO_PIN_TEST_RO_EXCLUSIVE, +}; + +static void do_test_ro_pin(char *mem, size_t size, enum ro_pin_test test, + bool fast) +{ + struct pin_longterm_test args; + struct comm_pipes comm_pipes; + char *tmp, buf; + __u64 tmp_val; + int ret; + + if (gup_fd < 0) { + ksft_test_result_skip("gup_test not available\n"); + return; + } + + tmp = malloc(size); + if (!tmp) { + ksft_test_result_fail("malloc() failed\n"); + return; + } + + ret = setup_comm_pipes(&comm_pipes); + if (ret) { + ksft_test_result_fail("pipe() failed\n"); + goto free_tmp; + } + + switch (test) { + case RO_PIN_TEST_SHARED: + case RO_PIN_TEST_PREVIOUSLY_SHARED: + /* + * Share the pages with our child. As the pages are not pinned, + * this should just work. + */ + ret = fork(); + if (ret < 0) { + ksft_test_result_fail("fork() failed\n"); + goto close_comm_pipes; + } else if (!ret) { + write(comm_pipes.child_ready[1], "0", 1); + while (read(comm_pipes.parent_ready[0], &buf, 1) != 1) + ; + exit(0); + } + + /* Wait until our child is ready. */ + while (read(comm_pipes.child_ready[0], &buf, 1) != 1) + ; + + if (test == RO_PIN_TEST_PREVIOUSLY_SHARED) { + /* + * Tell the child to quit now and wait until it quit. + * The pages should now be mapped R/O into our page + * tables, but they are no longer shared. + */ + write(comm_pipes.parent_ready[1], "0", 1); + wait(&ret); + if (!WIFEXITED(ret)) + ksft_print_msg("[INFO] wait() failed\n"); + } + break; + case RO_PIN_TEST_RO_EXCLUSIVE: + /* + * Map the page R/O into the page table. Enable softdirty + * tracking to stop the page from getting mapped R/W immediately + * again by mprotect() optimizations. Note that we don't have an + * easy way to test if that worked (the pagemap does not export + * if the page is mapped R/O vs. R/W). + */ + ret = mprotect(mem, size, PROT_READ); + clear_softdirty(); + ret |= mprotect(mem, size, PROT_READ | PROT_WRITE); + if (ret) { + ksft_test_result_fail("mprotect() failed\n"); + goto close_comm_pipes; + } + break; + default: + assert(false); + } + + /* Take a R/O pin. This should trigger unsharing. */ + args.addr = (__u64)mem; + args.size = size; + args.flags = fast ? PIN_LONGTERM_TEST_FLAG_USE_FAST : 0; + ret = ioctl(gup_fd, PIN_LONGTERM_TEST_START, &args); + if (ret) { + if (errno == EINVAL) + ksft_test_result_skip("PIN_LONGTERM_TEST_START failed\n"); + else + ksft_test_result_fail("PIN_LONGTERM_TEST_START failed\n"); + goto wait; + } + + /* Modify the page. */ + memset(mem, 0xff, size); + + /* + * Read back the content via the pin to the temporary buffer and + * test if we observed the modification. + */ + tmp_val = (__u64)tmp; + ret = ioctl(gup_fd, PIN_LONGTERM_TEST_READ, &tmp_val); + if (ret) + ksft_test_result_fail("PIN_LONGTERM_TEST_READ failed\n"); + else + ksft_test_result(!memcmp(mem, tmp, size), + "Longterm R/O pin is reliable\n"); + + ret = ioctl(gup_fd, PIN_LONGTERM_TEST_STOP); + if (ret) + ksft_print_msg("[INFO] PIN_LONGTERM_TEST_STOP failed\n"); +wait: + switch (test) { + case RO_PIN_TEST_SHARED: + write(comm_pipes.parent_ready[1], "0", 1); + wait(&ret); + if (!WIFEXITED(ret)) + ksft_print_msg("[INFO] wait() failed\n"); + break; + default: + break; + } +close_comm_pipes: + close_comm_pipes(&comm_pipes); +free_tmp: + free(tmp); +} + +static void test_ro_pin_on_shared(char *mem, size_t size) +{ + do_test_ro_pin(mem, size, RO_PIN_TEST_SHARED, false); +} + +static void test_ro_fast_pin_on_shared(char *mem, size_t size) +{ + do_test_ro_pin(mem, size, RO_PIN_TEST_SHARED, true); +} + +static void test_ro_pin_on_ro_previously_shared(char *mem, size_t size) +{ + do_test_ro_pin(mem, size, RO_PIN_TEST_PREVIOUSLY_SHARED, false); +} + +static void test_ro_fast_pin_on_ro_previously_shared(char *mem, size_t size) +{ + do_test_ro_pin(mem, size, RO_PIN_TEST_PREVIOUSLY_SHARED, true); +} + +static void test_ro_pin_on_ro_exclusive(char *mem, size_t size) +{ + do_test_ro_pin(mem, size, RO_PIN_TEST_RO_EXCLUSIVE, false); +} + +static void test_ro_fast_pin_on_ro_exclusive(char *mem, size_t size) +{ + do_test_ro_pin(mem, size, RO_PIN_TEST_RO_EXCLUSIVE, true); +} + typedef void (*test_fn)(char *mem, size_t size);
static void do_run_with_base_page(test_fn fn, bool swapout) @@ -850,6 +1017,48 @@ static const struct test_case test_cases[] = { },
#endif /* LOCAL_CONFIG_HAVE_LIBURING */ + /* + * Take a R/O longterm pin on a R/O-mapped shared anonymous page. + * When modifying the page via the page table, the page content change + * must be visible via the pin. + */ + { + "R/O GUP pin on R/O-mapped shared page", + test_ro_pin_on_shared, + }, + /* Same as above, but using GUP-fast. */ + { + "R/O GUP-fast pin on R/O-mapped shared page", + test_ro_fast_pin_on_shared, + }, + /* + * Take a R/O longterm pin on a R/O-mapped exclusive anonymous page that + * was previously shared. When modifying the page via the page table, + * the page content change must be visible via the pin. + */ + { + "R/O GUP pin on R/O-mapped previously-shared page", + test_ro_pin_on_ro_previously_shared, + }, + /* Same as above, but using GUP-fast. */ + { + "R/O GUP-fast pin on R/O-mapped previously-shared page", + test_ro_fast_pin_on_ro_previously_shared, + }, + /* + * Take a R/O longterm pin on a R/O-mapped exclusive anonymous page. + * When modifying the page via the page table, the page content change + * must be visible via the pin. + */ + { + "R/O GUP pin on R/O-mapped exclusive page", + test_ro_pin_on_ro_exclusive, + }, + /* Same as above, but using GUP-fast. */ + { + "R/O GUP-fast pin on R/O-mapped exclusive page", + test_ro_fast_pin_on_ro_exclusive, + }, };
static void run_test_case(struct test_case const *test_case) @@ -902,6 +1111,7 @@ int main(int argc, char **argv) ksft_print_header(); ksft_set_plan(nr_test_cases * tests_per_test_case());
+ gup_fd = open("/sys/kernel/debug/gup_test", O_RDWR); pagemap_fd = open("/proc/self/pagemap", O_RDONLY); if (pagemap_fd < 0) ksft_exit_fail_msg("opening pagemap failed\n");
On 27.09.22 13:01, David Hildenbrand wrote:
Let's trigger a R/O longterm pin on three cases of R/O mapped anonymous pages:
- exclusive (never shared)
- shared (child still alive)
- previously shared (child no longer alive)
... and make sure that the pin is reliable: whatever we write via the page tables has to be observable via the pin.
Signed-off-by: David Hildenbrand david@redhat.com
The following on top should silence the warning on 32bit:
https://lore.kernel.org/all/202210182101.XOyPZEQV-lkp@intel.com/T/#u
diff --git a/mm/gup_test.c b/mm/gup_test.c index 2bbfacd31f5a..57c343f2dc51 100644 --- a/mm/gup_test.c +++ b/mm/gup_test.c @@ -299,7 +299,8 @@ static inline int pin_longterm_test_read(unsigned long arg) for (i = 0; i < pin_longterm_test_nr_pages; i++) { void *addr = page_to_virt(pin_longterm_test_pages[i]);
- if (copy_to_user((void __user *)user_addr, addr, PAGE_SIZE)) + if (copy_to_user((void __user *)(unsigned long)user_addr, addr, + PAGE_SIZE)) return -EFAULT; user_addr += PAGE_SIZE; }
On 19.10.22 14:31, David Hildenbrand wrote:
On 27.09.22 13:01, David Hildenbrand wrote:
Let's trigger a R/O longterm pin on three cases of R/O mapped anonymous pages:
- exclusive (never shared)
- shared (child still alive)
- previously shared (child no longer alive)
... and make sure that the pin is reliable: whatever we write via the page tables has to be observable via the pin.
Signed-off-by: David Hildenbrand david@redhat.com
The following on top should silence the warning on 32bit:
https://lore.kernel.org/all/202210182101.XOyPZEQV-lkp@intel.com/T/#u
diff --git a/mm/gup_test.c b/mm/gup_test.c index 2bbfacd31f5a..57c343f2dc51 100644 --- a/mm/gup_test.c +++ b/mm/gup_test.c @@ -299,7 +299,8 @@ static inline int pin_longterm_test_read(unsigned long arg) for (i = 0; i < pin_longterm_test_nr_pages; i++) { void *addr = page_to_virt(pin_longterm_test_pages[i]);
if (copy_to_user((void __user *)user_addr, addr, PAGE_SIZE))
if (copy_to_user((void __user *)(unsigned long)user_addr, addr,PAGE_SIZE)) return -EFAULT; user_addr += PAGE_SIZE; }
Ehm, wrong patch, it actually belongs into #6/7
https://lkml.kernel.org/r/20220927110120.106906-7-david@redhat.com
@Andrew, if your confused, I can just resend the whole thing.
linux-kselftest-mirror@lists.linaro.org
-
David Hildenbrand -
Rafael Mendonca