*Changes in v28:* - Fix walk_end and add 17 test cases in selftests patch
*Changes in v27:* - Handle review comments and minor improvements - Add performance improvement patch on top with test for easy review
*Changes in v26:* - Code re-structurring and API changes in PAGEMAP_IOCTL
*Changes in v25*: - Do proper filtering on hole as well (hole got missed earlier)
*Changes in v24*: - Rebase on top of next-20230710 - Place WP markers in case of hole as well
*Changes in v23*: - Set vec_buf_index in loop only when vec_buf_index is set - Return -EFAULT instead of -EINVAL if vec is NULL - Correctly return the walk ending address to the page granularity
*Changes in v22*: - Interface change: - Replace [start start + len) with [start, end) - Return the ending address of the address walk in start
*Changes in v21*: - Abort walk instead of returning error if WP is to be performed on partial hugetlb
*Changes in v20* - Correct PAGE_IS_FILE and add PAGE_IS_PFNZERO
*Changes in v19* - Minor changes and interface updates
*Changes in v18* - Rebase on top of next-20230613 - Minor updates
*Changes in v17* - Rebase on top of next-20230606 - Minor improvements in PAGEMAP_SCAN IOCTL patch
*Changes in v16* - Fix a corner case - Add exclusive PM_SCAN_OP_WP back
*Changes in v15* - Build fix (Add missed build fix in RESEND)
*Changes in v14* - Fix build error caused by #ifdef added at last minute in some configs
*Changes in v13* - Rebase on top of next-20230414 - Give-up on using uffd_wp_range() and write new helpers, flush tlb only once
*Changes in v12* - Update and other memory types to UFFD_FEATURE_WP_ASYNC - Rebaase on top of next-20230406 - Review updates
*Changes in v11* - Rebase on top of next-20230307 - Base patches on UFFD_FEATURE_WP_UNPOPULATED - Do a lot of cosmetic changes and review updates - Remove ENGAGE_WP + !GET operation as it can be performed with UFFDIO_WRITEPROTECT
*Changes in v10* - Add specific condition to return error if hugetlb is used with wp async - Move changes in tools/include/uapi/linux/fs.h to separate patch - Add documentation
*Changes in v9:* - Correct fault resolution for userfaultfd wp async - Fix build warnings and errors which were happening on some configs - Simplify pagemap ioctl's code
*Changes in v8:* - Update uffd async wp implementation - Improve PAGEMAP_IOCTL implementation
*Changes in v7:* - Add uffd wp async - Update the IOCTL to use uffd under the hood instead of soft-dirty flags
*Motivation* The real motivation for adding PAGEMAP_SCAN IOCTL is to emulate Windows GetWriteWatch() and ResetWriteWatch() syscalls [1]. The GetWriteWatch() retrieves the addresses of the pages that are written to in a region of virtual memory.
This syscall is used in Windows applications and games etc. This syscall is being emulated in pretty slow manner in userspace. Our purpose is to enhance the kernel such that we translate it efficiently in a better way. Currently some out of tree hack patches are being used to efficiently emulate it in some kernels. We intend to replace those with these patches. So the whole gaming on Linux can effectively get benefit from this. It means there would be tons of users of this code.
CRIU use case [2] was mentioned by Andrei and Danylo:
Use cases for migrating sparse VMAs are binaries sanitized with ASAN, MSAN or TSAN [3]. All of these sanitizers produce sparse mappings of shadow memory [4]. Being able to migrate such binaries allows to highly reduce the amount of work needed to identify and fix post-migration crashes, which happen constantly.
Andrei's defines the following uses of this code: * it is more granular and allows us to track changed pages more effectively. The current interface can clear dirty bits for the entire process only. In addition, reading info about pages is a separate operation. It means we must freeze the process to read information about all its pages, reset dirty bits, only then we can start dumping pages. The information about pages becomes more and more outdated, while we are processing pages. The new interface solves both these downsides. First, it allows us to read pte bits and clear the soft-dirty bit atomically. It means that CRIU will not need to freeze processes to pre-dump their memory. Second, it clears soft-dirty bits for a specified region of memory. It means CRIU will have actual info about pages to the moment of dumping them. * The new interface has to be much faster because basic page filtering is happening in the kernel. With the old interface, we have to read pagemap for each page.
*Implementation Evolution (Short Summary)* From the definition of GetWriteWatch(), we feel like kernel's soft-dirty feature can be used under the hood with some additions like: * reset soft-dirty flag for only a specific region of memory instead of clearing the flag for the entire process * get and clear soft-dirty flag for a specific region atomically
So we decided to use ioctl on pagemap file to read or/and reset soft-dirty flag. But using soft-dirty flag, sometimes we get extra pages which weren't even written. They had become soft-dirty because of VMA merging and VM_SOFTDIRTY flag. This breaks the definition of GetWriteWatch(). We were able to by-pass this short coming by ignoring VM_SOFTDIRTY until David reported that mprotect etc messes up the soft-dirty flag while ignoring VM_SOFTDIRTY [5]. This wasn't happening until [6] got introduced. We discussed if we can revert these patches. But we could not reach to any conclusion. So at this point, I made couple of tries to solve this whole VM_SOFTDIRTY issue by correcting the soft-dirty implementation: * [7] Correct the bug fixed wrongly back in 2014. It had potential to cause regression. We left it behind. * [8] Keep a list of soft-dirty part of a VMA across splits and merges. I got the reply don't increase the size of the VMA by 8 bytes.
At this point, we left soft-dirty considering it is too much delicate and userfaultfd [9] seemed like the only way forward. From there onward, we have been basing soft-dirty emulation on userfaultfd wp feature where kernel resolves the faults itself when WP_ASYNC feature is used. It was straight forward to add WP_ASYNC feature in userfautlfd. Now we get only those pages dirty or written-to which are really written in reality. (PS There is another WP_UNPOPULATED userfautfd feature is required which is needed to avoid pre-faulting memory before write-protecting [9].)
All the different masks were added on the request of CRIU devs to create interface more generic and better.
[1] https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-g... [2] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com [3] https://github.com/google/sanitizers [4] https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm#64-bit [5] https://lore.kernel.org/all/bfcae708-db21-04b4-0bbe-712badd03071@redhat.com [6] https://lore.kernel.org/all/20220725142048.30450-1-peterx@redhat.com/ [7] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.c... [8] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.c... [9] https://lore.kernel.org/all/20230306213925.617814-1-peterx@redhat.com [10] https://lore.kernel.org/all/20230125144529.1630917-1-mdanylo@google.com
* Original Cover letter from v8* Hello,
Note: Soft-dirty pages and pages which have been written-to are synonyms. As kernel already has soft-dirty feature inside which we have given up to use, we are using written-to terminology while using UFFD async WP under the hood.
It is possible to find and clear soft-dirty pages entirely in userspace. But it isn't efficient: - The mprotect and SIGSEGV handler for bookkeeping - The userfaultfd wp (synchronous) with the handler for bookkeeping
Some benchmarks can be seen here[1]. This series adds features that weren't present earlier: - There is no atomic get soft-dirty/Written-to status and clear present in the kernel. - The pages which have been written-to can not be found in accurate way. (Kernel's soft-dirty PTE bit + sof_dirty VMA bit shows more soft-dirty pages than there actually are.)
Historically, soft-dirty PTE bit tracking has been used in the CRIU project. The procfs interface is enough for finding the soft-dirty bit status and clearing the soft-dirty bit of all the pages of a process. We have the use case where we need to track the soft-dirty PTE bit for only specific pages on-demand. We need this tracking and clear mechanism of a region of memory while the process is running to emulate the getWriteWatch() syscall of Windows.
*(Moved to using UFFD instead of soft-dirty feature to find pages which have been written-to from v7 patch series)*: Stop using the soft-dirty flags for finding which pages have been written to. It is too delicate and wrong as it shows more soft-dirty pages than the actual soft-dirty pages. There is no interest in correcting it [2][3] as this is how the feature was written years ago. It shouldn't be updated to changed behaviour. Peter Xu has suggested using the async version of the UFFD WP [4] as it is based inherently on the PTEs.
So in this patch series, I've added a new mode to the UFFD which is asynchronous version of the write protect. When this variant of the UFFD WP is used, the page faults are resolved automatically by the kernel. The pages which have been written-to can be found by reading pagemap file (!PM_UFFD_WP). This feature can be used successfully to find which pages have been written to from the time the pages were write protected. This works just like the soft-dirty flag without showing any extra pages which aren't soft-dirty in reality.
The information related to pages if the page is file mapped, present and swapped is required for the CRIU project [5][6]. The addition of the required mask, any mask, excluded mask and return masks are also required for the CRIU project [5].
The IOCTL returns the addresses of the pages which match the specific masks. The page addresses are returned in struct page_region in a compact form. The max_pages is needed to support a use case where user only wants to get a specific number of pages. So there is no need to find all the pages of interest in the range when max_pages is specified. The IOCTL returns when the maximum number of the pages are found. The max_pages is optional. If max_pages is specified, it must be equal or greater than the vec_size. This restriction is needed to handle worse case when one page_region only contains info of one page and it cannot be compacted. This is needed to emulate the Windows getWriteWatch() syscall.
The patch series include the detailed selftest which can be used as an example for the uffd async wp test and PAGEMAP_IOCTL. It shows the interface usages as well.
[1] https://lore.kernel.org/lkml/54d4c322-cd6e-eefd-b161-2af2b56aae24@collabora.... [2] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.c... [3] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.c... [4] https://lore.kernel.org/all/Y6Hc2d+7eTKs7AiH@x1n [5] https://lore.kernel.org/all/YyiDg79flhWoMDZB@gmail.com/ [6] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com/
Regards, Muhammad Usama Anjum
Muhammad Usama Anjum (5): fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs fs/proc/task_mmu: Add fast paths to get/clear PAGE_IS_WRITTEN flag tools headers UAPI: Update linux/fs.h with the kernel sources mm/pagemap: add documentation of PAGEMAP_SCAN IOCTL selftests: mm: add pagemap ioctl tests
Peter Xu (1): userfaultfd: UFFD_FEATURE_WP_ASYNC
Documentation/admin-guide/mm/pagemap.rst | 64 + Documentation/admin-guide/mm/userfaultfd.rst | 35 + fs/proc/task_mmu.c | 716 ++++++++ fs/userfaultfd.c | 26 +- include/linux/hugetlb.h | 1 + include/linux/userfaultfd_k.h | 21 +- include/uapi/linux/fs.h | 59 + include/uapi/linux/userfaultfd.h | 9 +- mm/hugetlb.c | 34 +- mm/memory.c | 27 +- tools/include/uapi/linux/fs.h | 59 + tools/testing/selftests/mm/.gitignore | 2 + tools/testing/selftests/mm/Makefile | 3 +- tools/testing/selftests/mm/config | 1 + tools/testing/selftests/mm/pagemap_ioctl.c | 1658 ++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 4 + 16 files changed, 2695 insertions(+), 24 deletions(-) create mode 100644 tools/testing/selftests/mm/pagemap_ioctl.c
From: Peter Xu peterx@redhat.com
This patch adds a new userfaultfd-wp feature UFFD_FEATURE_WP_ASYNC, that allows userfaultfd wr-protect faults to be resolved by the kernel directly.
It can be used like a high accuracy version of soft-dirty, without vma modifications during tracking, and also with ranged support by default rather than for a whole mm when reset the protections due to existence of ioctl(UFFDIO_WRITEPROTECT).
Several goals of such a dirty tracking interface:
1. All types of memory should be supported and tracable. This is nature for soft-dirty but should mention when the context is userfaultfd, because it used to only support anon/shmem/hugetlb. The problem is for a dirty tracking purpose these three types may not be enough, and it's legal to track anything e.g. any page cache writes from mmap.
2. Protections can be applied to partial of a memory range, without vma split/merge fuss. The hope is that the tracking itself should not affect any vma layout change. It also helps when reset happens because the reset will not need mmap write lock which can block the tracee.
3. Accuracy needs to be maintained. This means we need pte markers to work on any type of VMA.
One could question that, the whole concept of async dirty tracking is not really close to fundamentally what userfaultfd used to be: it's not "a fault to be serviced by userspace" anymore. However, using userfaultfd-wp here as a framework is convenient for us in at least:
1. VM_UFFD_WP vma flag, which has a very good name to suite something like this, so we don't need VM_YET_ANOTHER_SOFT_DIRTY. Just use a new feature bit to identify from a sync version of uffd-wp registration.
2. PTE markers logic can be leveraged across the whole kernel to maintain the uffd-wp bit as long as an arch supports, this also applies to this case where uffd-wp bit will be a hint to dirty information and it will not go lost easily (e.g. when some page cache ptes got zapped).
3. Reuse ioctl(UFFDIO_WRITEPROTECT) interface for either starting or resetting a range of memory, while there's no counterpart in the old soft-dirty world, hence if this is wanted in a new design we'll need a new interface otherwise.
We can somehow understand that commonality because uffd-wp was fundamentally a similar idea of write-protecting pages just like soft-dirty.
This implementation allows WP_ASYNC to imply WP_UNPOPULATED, because so far WP_ASYNC seems to not usable if without WP_UNPOPULATE. This also gives us chance to modify impl of WP_ASYNC just in case it could be not depending on WP_UNPOPULATED anymore in the future kernels. It's also fine to imply that because both features will rely on PTE_MARKER_UFFD_WP config option, so they'll show up together (or both missing) in an UFFDIO_API probe.
vma_can_userfault() now allows any VMA if the userfaultfd registration is only about async uffd-wp. So we can track dirty for all kinds of memory including generic file systems (like XFS, EXT4 or BTRFS).
One trick worth mention in do_wp_page() is that we need to manually update vmf->orig_pte here because it can be used later with a pte_same() check - this path always has FAULT_FLAG_ORIG_PTE_VALID set in the flags.
The major defect of this approach of dirty tracking is we need to populate the pgtables when tracking starts. Soft-dirty doesn't do it like that. It's unwanted in the case where the range of memory to track is huge and unpopulated (e.g., tracking updates on a 10G file with mmap() on top, without having any page cache installed yet). One way to improve this is to allow pte markers exist for larger than PTE level for PMD+. That will not change the interface if to implemented, so we can leave that for later.
Co-developed-by: Muhammad Usama Anjum usama.anjum@collabora.com Signed-off-by: Muhammad Usama Anjum usama.anjum@collabora.com Signed-off-by: Peter Xu peterx@redhat.com --- Changes in v24: - Rebase on top of next-20230710 - UFFD_FEATURE_WP_ASYNC has been defined to be (1 << 15) after the rebase
Changes in v18: - Rebase on top of next-20230613 (use ptep_get)
Changes in v17: - Rebase on top of next-20230525
Changes in v12: - Peter added the hugetlb support and revamped some other implementation - Transferred the authorship to Peter - Merge documentation to this patch
Changes in v11: - Fix return code in userfaultfd_register() and minor changes here and there - Rebase on top of next-20230307 - Base patches on UFFD_FEATURE_WP_UNPOPULATED https://lore.kernel.org/all/20230306213925.617814-1-peterx@redhat.com - UFFD_FEATURE_WP_ASYNC depends on UFFD_FEATURE_WP_UNPOPULATED to work (correctly)
Changes in v10: - Build fix - Update comments and add error condition to return error from uffd register if hugetlb pages are present when wp async flag is set
Changes in v9: - Correct the fault resolution with code contributed by Peter
Changes in v7: - Remove UFFDIO_WRITEPROTECT_MODE_ASYNC_WP and add UFFD_FEATURE_WP_ASYNC - Handle automatic page fault resolution in better way (thanks to Peter) --- Documentation/admin-guide/mm/userfaultfd.rst | 35 ++++++++++++++++++++ fs/userfaultfd.c | 26 ++++++++++++--- include/linux/userfaultfd_k.h | 21 +++++++++++- include/uapi/linux/userfaultfd.h | 9 ++++- mm/hugetlb.c | 32 ++++++++++-------- mm/memory.c | 27 +++++++++++++-- 6 files changed, 128 insertions(+), 22 deletions(-)
diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst index 4349a8c2b9783..203e26da5f920 100644 --- a/Documentation/admin-guide/mm/userfaultfd.rst +++ b/Documentation/admin-guide/mm/userfaultfd.rst @@ -244,6 +244,41 @@ write-protected (so future writes will also result in a WP fault). These ioctls support a mode flag (``UFFDIO_COPY_MODE_WP`` or ``UFFDIO_CONTINUE_MODE_WP`` respectively) to configure the mapping this way.
+If the userfaultfd context has ``UFFD_FEATURE_WP_ASYNC`` feature bit set, +any vma registered with write-protection will work in async mode rather +than the default sync mode. + +In async mode, there will be no message generated when a write operation +happens, meanwhile the write-protection will be resolved automatically by +the kernel. It can be seen as a more accurate version of soft-dirty +tracking and it can be different in a few ways: + + - The dirty result will not be affected by vma changes (e.g. vma + merging) because the dirty is only tracked by the pte. + + - It supports range operations by default, so one can enable tracking on + any range of memory as long as page aligned. + + - Dirty information will not get lost if the pte was zapped due to + various reasons (e.g. during split of a shmem transparent huge page). + + - Due to a reverted meaning of soft-dirty (page clean when uffd-wp bit + set; dirty when uffd-wp bit cleared), it has different semantics on + some of the memory operations. For example: ``MADV_DONTNEED`` on + anonymous (or ``MADV_REMOVE`` on a file mapping) will be treated as + dirtying of memory by dropping uffd-wp bit during the procedure. + +The user app can collect the "written/dirty" status by looking up the +uffd-wp bit for the pages being interested in /proc/pagemap. + +The page will not be under track of uffd-wp async mode until the page is +explicitly write-protected by ``ioctl(UFFDIO_WRITEPROTECT)`` with the mode +flag ``UFFDIO_WRITEPROTECT_MODE_WP`` set. Trying to resolve a page fault +that was tracked by async mode userfaultfd-wp is invalid. + +When userfaultfd-wp async mode is used alone, it can be applied to all +kinds of memory. + Memory Poisioning Emulation ---------------------------
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index bbfaf5837a08a..938f29be55422 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -123,6 +123,11 @@ static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx) return ctx->features & UFFD_FEATURE_INITIALIZED; }
+static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx) +{ + return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC); +} + /* * Whether WP_UNPOPULATED is enabled on the uffd context. It is only * meaningful when userfaultfd_wp()==true on the vma and when it's @@ -1318,6 +1323,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, bool basic_ioctls; unsigned long start, end, vma_end; struct vma_iterator vmi; + bool wp_async = userfaultfd_wp_async_ctx(ctx); pgoff_t pgoff;
user_uffdio_register = (struct uffdio_register __user *) arg; @@ -1392,7 +1398,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
/* check not compatible vmas */ ret = -EINVAL; - if (!vma_can_userfault(cur, vm_flags)) + if (!vma_can_userfault(cur, vm_flags, wp_async)) goto out_unlock;
/* @@ -1453,7 +1459,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, for_each_vma_range(vmi, vma, end) { cond_resched();
- BUG_ON(!vma_can_userfault(vma, vm_flags)); + BUG_ON(!vma_can_userfault(vma, vm_flags, wp_async)); BUG_ON(vma->vm_userfaultfd_ctx.ctx && vma->vm_userfaultfd_ctx.ctx != ctx); WARN_ON(!(vma->vm_flags & VM_MAYWRITE)); @@ -1553,6 +1559,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, unsigned long start, end, vma_end; const void __user *buf = (void __user *)arg; struct vma_iterator vmi; + bool wp_async = userfaultfd_wp_async_ctx(ctx); pgoff_t pgoff;
ret = -EFAULT; @@ -1607,7 +1614,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, * provides for more strict behavior to notice * unregistration errors. */ - if (!vma_can_userfault(cur, cur->vm_flags)) + if (!vma_can_userfault(cur, cur->vm_flags, wp_async)) goto out_unlock;
found = true; @@ -1623,7 +1630,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, for_each_vma_range(vmi, vma, end) { cond_resched();
- BUG_ON(!vma_can_userfault(vma, vma->vm_flags)); + BUG_ON(!vma_can_userfault(vma, vma->vm_flags, wp_async));
/* * Nothing to do: this vma is already registered into this @@ -2008,6 +2015,11 @@ static inline int userfaultfd_poison(struct userfaultfd_ctx *ctx, unsigned long return ret; }
+bool userfaultfd_wp_async(struct vm_area_struct *vma) +{ + return userfaultfd_wp_async_ctx(vma->vm_userfaultfd_ctx.ctx); +} + static inline unsigned int uffd_ctx_features(__u64 user_features) { /* @@ -2041,6 +2053,11 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, ret = -EPERM; if ((features & UFFD_FEATURE_EVENT_FORK) && !capable(CAP_SYS_PTRACE)) goto err_out; + + /* WP_ASYNC relies on WP_UNPOPULATED, choose it unconditionally */ + if (features & UFFD_FEATURE_WP_ASYNC) + features |= UFFD_FEATURE_WP_UNPOPULATED; + /* report all available features and ioctls to userland */ uffdio_api.features = UFFD_API_FEATURES; #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR @@ -2053,6 +2070,7 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, #ifndef CONFIG_PTE_MARKER_UFFD_WP uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM; uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED; + uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC; #endif uffdio_api.ioctls = UFFD_API_IOCTLS; ret = -EFAULT; diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index ac8c6854097cd..c98df391bfd8f 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -161,11 +161,22 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma) }
static inline bool vma_can_userfault(struct vm_area_struct *vma, - unsigned long vm_flags) + unsigned long vm_flags, + bool wp_async) { + vm_flags &= __VM_UFFD_FLAGS; + if ((vm_flags & VM_UFFD_MINOR) && (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma))) return false; + + /* + * If wp async enabled, and WP is the only mode enabled, allow any + * memory type. + */ + if (wp_async && (vm_flags == VM_UFFD_WP)) + return true; + #ifndef CONFIG_PTE_MARKER_UFFD_WP /* * If user requested uffd-wp but not enabled pte markers for @@ -175,6 +186,8 @@ static inline bool vma_can_userfault(struct vm_area_struct *vma, if ((vm_flags & VM_UFFD_WP) && !vma_is_anonymous(vma)) return false; #endif + + /* By default, allow any of anon|shmem|hugetlb */ return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) || vma_is_shmem(vma); } @@ -197,6 +210,7 @@ extern int userfaultfd_unmap_prep(struct vm_area_struct *vma, extern void userfaultfd_unmap_complete(struct mm_struct *mm, struct list_head *uf); extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma); +extern bool userfaultfd_wp_async(struct vm_area_struct *vma);
#else /* CONFIG_USERFAULTFD */
@@ -297,6 +311,11 @@ static inline bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma) return false; }
+static inline bool userfaultfd_wp_async(struct vm_area_struct *vma) +{ + return false; +} + #endif /* CONFIG_USERFAULTFD */
static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma) diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h index 62151706c5a38..0dbc81015018b 100644 --- a/include/uapi/linux/userfaultfd.h +++ b/include/uapi/linux/userfaultfd.h @@ -40,7 +40,8 @@ UFFD_FEATURE_EXACT_ADDRESS | \ UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \ UFFD_FEATURE_WP_UNPOPULATED | \ - UFFD_FEATURE_POISON) + UFFD_FEATURE_POISON | \ + UFFD_FEATURE_WP_ASYNC) #define UFFD_API_IOCTLS \ ((__u64)1 << _UFFDIO_REGISTER | \ (__u64)1 << _UFFDIO_UNREGISTER | \ @@ -216,6 +217,11 @@ struct uffdio_api { * (i.e. empty ptes). This will be the default behavior for shmem * & hugetlbfs, so this flag only affects anonymous memory behavior * when userfault write-protection mode is registered. + * + * UFFD_FEATURE_WP_ASYNC indicates that userfaultfd write-protection + * asynchronous mode is supported in which the write fault is + * automatically resolved and write-protection is un-set. + * It implies UFFD_FEATURE_WP_UNPOPULATED. */ #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) #define UFFD_FEATURE_EVENT_FORK (1<<1) @@ -232,6 +238,7 @@ struct uffdio_api { #define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12) #define UFFD_FEATURE_WP_UNPOPULATED (1<<13) #define UFFD_FEATURE_POISON (1<<14) +#define UFFD_FEATURE_WP_ASYNC (1<<15) __u64 features;
__u64 ioctls; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 154cc5b315727..a073e6ed8900b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6191,21 +6191,27 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, /* Handle userfault-wp first, before trying to lock more pages */ if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) && (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) { - struct vm_fault vmf = { - .vma = vma, - .address = haddr, - .real_address = address, - .flags = flags, - }; + if (!userfaultfd_wp_async(vma)) { + struct vm_fault vmf = { + .vma = vma, + .address = haddr, + .real_address = address, + .flags = flags, + };
- spin_unlock(ptl); - if (pagecache_folio) { - folio_unlock(pagecache_folio); - folio_put(pagecache_folio); + spin_unlock(ptl); + if (pagecache_folio) { + folio_unlock(pagecache_folio); + folio_put(pagecache_folio); + } + hugetlb_vma_unlock_read(vma); + mutex_unlock(&hugetlb_fault_mutex_table[hash]); + return handle_userfault(&vmf, VM_UFFD_WP); } - hugetlb_vma_unlock_read(vma); - mutex_unlock(&hugetlb_fault_mutex_table[hash]); - return handle_userfault(&vmf, VM_UFFD_WP); + + entry = huge_pte_clear_uffd_wp(entry); + set_huge_pte_at(mm, haddr, ptep, entry); + /* Fallthrough to CoW */ }
/* diff --git a/mm/memory.c b/mm/memory.c index 7ff73a197cced..94a24ac2270b2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3466,11 +3466,28 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; struct vm_area_struct *vma = vmf->vma; struct folio *folio = NULL; + pte_t pte;
if (likely(!unshare)) { if (userfaultfd_pte_wp(vma, ptep_get(vmf->pte))) { - pte_unmap_unlock(vmf->pte, vmf->ptl); - return handle_userfault(vmf, VM_UFFD_WP); + if (!userfaultfd_wp_async(vma)) { + pte_unmap_unlock(vmf->pte, vmf->ptl); + return handle_userfault(vmf, VM_UFFD_WP); + } + + /* + * Nothing needed (cache flush, TLB invalidations, + * etc.) because we're only removing the uffd-wp bit, + * which is completely invisible to the user. + */ + pte = pte_clear_uffd_wp(ptep_get(vmf->pte)); + + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); + /* + * Update this to be prepared for following up CoW + * handling + */ + vmf->orig_pte = pte; }
/* @@ -4965,8 +4982,11 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
if (vma_is_anonymous(vmf->vma)) { if (likely(!unshare) && - userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) + userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) { + if (userfaultfd_wp_async(vmf->vma)) + goto split; return handle_userfault(vmf, VM_UFFD_WP); + } return do_huge_pmd_wp_page(vmf); }
@@ -4978,6 +4998,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) } }
+split: /* COW or write-notify handled on pte level: split pmd. */ __split_huge_pmd(vmf->vma, vmf->pmd, vmf->address, false, NULL);
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear the info about page table entries. The following operations are supported in this ioctl: - Get the information if the pages have Async Write-Protection enabled (``PAGE_IS_WPALLOWED``), have been written to (``PAGE_IS_WRITTEN``), file mapped (``PAGE_IS_FILE``), present (``PAGE_IS_PRESENT``), swapped (``PAGE_IS_SWAPPED``) or page has pfn zero (``PAGE_IS_PFNZERO``). - Find pages which have been written to and/or write protect (atomic ``PM_SCAN_WP_MATCHING + PM_SCAN_CHECK_WPASYNC``) the pages atomically. The (``PM_SCAN_WP_MATCHING``) is used to WP the matched pages. The (``PM_SCAN_CHECK_WPASYNC``) aborts the operation if non-Async-Write-Protected pages are found. Get is automatically performed if output buffer is specified.
This IOCTL can be extended to get information about more PTE bits. The entire address range passed by user [start, end) is scanned until either the user provided buffer is full or max_pages have been found.
Reviewed-by: Andrei Vagin avagin@gmail.com Reviewed-by: Michał Mirosław mirq-linux@rere.qmqm.pl Signed-off-by: Michał Mirosław mirq-linux@rere.qmqm.pl Signed-off-by: Muhammad Usama Anjum usama.anjum@collabora.com --- Changes in v28: - Fix walk_end one last time after doing through testing
Changes in v27: - Add PAGE_IS_HUGE - Iterate until temporary buffer is full to do less iterations - Don't check if PAGE_IS_FILE if no mask needs it as it is very expensive to check per pte - bring is_interesting_page() outside pagemap_scan_output() to remove the horrible return value check - Replace memcpy() with direct copy - rename end_addr to walk_end_addr in pagemap_scan_private - Abort walk if fatal_signal_pending()
Changes in v26: Changes made by Usama: - Fix the wrong breaking of loop if page isn't interesting, skip intsead - Untag the address and save them into struct - Round off the end address to next page - Correct the partial hugetlb page handling and returning the error - Rename PAGE_IS_WPASYNC to PAGE_IS_WPALLOWED - Return walk ending address in walk_end instead of returning in start as there is potential of replacing the memory tag
Changes by Michał: 1. the API: a. return ranges as {begin, end} instead of {begin + len}; b. rename match "flags" to 'page categories' everywhere - this makes it easier to differentiate the ioctl()s categorisation of pages from struct page flags; c. change {required + excluded} to {inverted + required}. This was rejected before, but I'd like to illustrate the difference. Old interface can be translated to the new by: categories_inverted = excluded_mask categories_mask = required_mask | excluded_mask categories_anyof_mask = anyof_mask The new way allows filtering by: A & (B | !C) categories_inverted = C categories_mask = A categories_anyof_mask = B | C e. allow no-op calls 2. the implementation: a. gather the page-categorising and write-protecting code in one place; b. optimization: add whole-vma skipping for WP usecase; c. extracted output limiting code to pagemap_scan_output(); d. extracted range coalescing to pagemap_scan_push_range(); e. extracted THP entry handling to pagemap_scan_thp_entry(); f. added a shortcut for non-WP hugetlb scan; avoids conditional locking; g. extracted scan buffer handling code out of do_pagemap_scan(); h. rework output code to always try to write pending ranges; if EFAULT is generated it always overwrites the original error code; (the case of SIGKILL is needlessly trying to write the output now, but this should be rare case and ignoring it makes the code not needing a goto) 3.Change no-GET operation condition from `arg.return_mask == 0` to `arg.vec == NULL`. This will allow issuing the ioctl with return_mask == 0 to gather matching ranges when the exact category is not interesting. (Anticipated for CRIU scanning a large sparse anonymous mapping).
Changes in v25: - Do proper filtering on hole as well (hole got missed earlier)
Changes in v24: - Place WP markers in case of hole as well
Changes in v23: - Set vec_buf_index to 0 only when vec_buf_index is set - Return -EFAULT instead of -EINVAL if vec is NULL - Correctly return the walk ending address to the page granularity
Changes in v22: - Interface change to return walk ending address to user: - Replace [start start + len) with [start, end) - Return the ending address of the address walk in start
Changes in v21: - Abort walk instead of returning error if WP is to be performed on partial hugetlb - Changed the data types of some variables in pagemap_scan_private to long
Changes in v20: - Correct PAGE_IS_FILE and add PAGE_IS_PFNZERO
Changes in v19: - Interface changes such as renaming, return mask and WP can be used with any flags specified in masks - Internal code changes
Changes in v18: - Rebased on top of next-20230613 - ptep_get() updates - remove pmd_trans_unstable() and add ACTION_AGAIN - Review updates (Micheal)
Changes in v17: - Rebased on next-20230606 - Made make_uffd_wp_*_pte() better and minor changes
Changes in v16: - Fixed a corner case where kernel writes beyond user buffer by one element - Bring back exclusive PM_SCAN_OP_WP - Cosmetic changes
Changes in v15: - Build fix: - Use generic tlb flush function in pagemap_scan_pmd_entry() instead of using x86 specific flush function in do_pagemap_scan() - Remove #ifdef from pagemap_scan_hugetlb_entry() - Use mm instead of undefined vma->vm_mm
Changes in v14: - Fix build error caused by #ifdef added at last minute in some configs
Changes in v13: - Review updates - mmap_read_lock_killable() instead of mmap_read_lock() - Replace uffd_wp_range() with helpers which increases performance drastically for OP_WP operations by reducing the number of tlb flushing etc - Add MMU_NOTIFY_PROTECTION_VMA notification for the memory range
Changes in v12: - Add hugetlb support to cover all memory types - Merge "userfaultfd: Define dummy uffd_wp_range()" with this patch - Review updates to the code
Changes in v11: - Find written pages in a better way - Fix a corner case (thanks Paul) - Improve the code/comments - remove ENGAGE_WP + ! GET operation - shorten the commit message in favour of moving documentation to pagemap.rst
Changes in v10: - move changes in tools/include/uapi/linux/fs.h to separate patch - update commit message
Change in v8: - Correct is_pte_uffd_wp() - Improve readability and error checks - Remove some un-needed code
Changes in v7: - Rebase on top of latest next - Fix some corner cases - Base soft-dirty on the uffd wp async - Update the terminologies - Optimize the memory usage inside the ioctl --- fs/proc/task_mmu.c | 678 ++++++++++++++++++++++++++++++++++++++++ include/linux/hugetlb.h | 1 + include/uapi/linux/fs.h | 59 ++++ mm/hugetlb.c | 2 +- 4 files changed, 739 insertions(+), 1 deletion(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index c1e6531cb02ae..0e219a44e97cd 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -19,6 +19,8 @@ #include <linux/shmem_fs.h> #include <linux/uaccess.h> #include <linux/pkeys.h> +#include <linux/minmax.h> +#include <linux/overflow.h>
#include <asm/elf.h> #include <asm/tlb.h> @@ -1749,11 +1751,687 @@ static int pagemap_release(struct inode *inode, struct file *file) return 0; }
+#define PM_SCAN_CATEGORIES (PAGE_IS_WPALLOWED | PAGE_IS_WRITTEN | \ + PAGE_IS_FILE | PAGE_IS_PRESENT | \ + PAGE_IS_SWAPPED | PAGE_IS_PFNZERO | \ + PAGE_IS_HUGE) +#define PM_SCAN_FLAGS (PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC) + +#define MASKS_OF_INTEREST(a) (a.category_inverted | a.category_mask | \ + a.category_anyof_mask | a.return_mask) + +struct pagemap_scan_private { + struct pm_scan_arg arg; + unsigned long masks_of_interest, cur_vma_category; + struct page_region *vec_buf, cur_buf; + unsigned long vec_buf_len, vec_buf_index, found_pages, walk_end_addr; + struct page_region __user *vec_out; +}; + +static unsigned long pagemap_page_category(struct pagemap_scan_private *p, + struct vm_area_struct *vma, + unsigned long addr, pte_t pte) +{ + unsigned long categories = 0; + + if (pte_present(pte)) { + struct page *page; + + categories |= PAGE_IS_PRESENT; + if (!pte_uffd_wp(pte)) + categories |= PAGE_IS_WRITTEN; + + if (p->masks_of_interest & PAGE_IS_FILE) { + page = vm_normal_page(vma, addr, pte); + if (page && !PageAnon(page)) + categories |= PAGE_IS_FILE; + } + + if (is_zero_pfn(pte_pfn(pte))) + categories |= PAGE_IS_PFNZERO; + } else if (is_swap_pte(pte)) { + swp_entry_t swp; + + categories |= PAGE_IS_SWAPPED; + if (!pte_swp_uffd_wp_any(pte)) + categories |= PAGE_IS_WRITTEN; + + if (p->masks_of_interest & PAGE_IS_FILE) { + swp = pte_to_swp_entry(pte); + if (is_pfn_swap_entry(swp) && + !PageAnon(pfn_swap_entry_to_page(swp))) + categories |= PAGE_IS_FILE; + } + } + + return categories; +} + +static void make_uffd_wp_pte(struct vm_area_struct *vma, + unsigned long addr, pte_t *pte) +{ + pte_t ptent = ptep_get(pte); + + if (pte_present(ptent)) { + pte_t old_pte; + + old_pte = ptep_modify_prot_start(vma, addr, pte); + ptent = pte_mkuffd_wp(ptent); + ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent); + } else if (is_swap_pte(ptent)) { + ptent = pte_swp_mkuffd_wp(ptent); + set_pte_at(vma->vm_mm, addr, pte, ptent); + } else { + set_pte_at(vma->vm_mm, addr, pte, + make_pte_marker(PTE_MARKER_UFFD_WP)); + } +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static unsigned long pagemap_thp_category(pmd_t pmd) +{ + unsigned long categories = PAGE_IS_HUGE; + + if (pmd_present(pmd)) { + categories |= PAGE_IS_PRESENT; + if (!pmd_uffd_wp(pmd)) + categories |= PAGE_IS_WRITTEN; + if (is_zero_pfn(pmd_pfn(pmd))) + categories |= PAGE_IS_PFNZERO; + } else if (is_swap_pmd(pmd)) { + categories |= PAGE_IS_SWAPPED; + if (!pmd_swp_uffd_wp(pmd)) + categories |= PAGE_IS_WRITTEN; + } + + return categories; +} + +static void make_uffd_wp_pmd(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmdp) +{ + pmd_t old, pmd = *pmdp; + + if (pmd_present(pmd)) { + old = pmdp_invalidate_ad(vma, addr, pmdp); + pmd = pmd_mkuffd_wp(old); + set_pmd_at(vma->vm_mm, addr, pmdp, pmd); + } else if (is_migration_entry(pmd_to_swp_entry(pmd))) { + pmd = pmd_swp_mkuffd_wp(pmd); + set_pmd_at(vma->vm_mm, addr, pmdp, pmd); + } +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + +#ifdef CONFIG_HUGETLB_PAGE +static unsigned long pagemap_hugetlb_category(pte_t pte) +{ + unsigned long categories = PAGE_IS_HUGE; + + if (pte_present(pte)) { + categories |= PAGE_IS_PRESENT; + if (!huge_pte_uffd_wp(pte)) + categories |= PAGE_IS_WRITTEN; + if (!PageAnon(pte_page(pte))) + categories |= PAGE_IS_FILE; + if (is_zero_pfn(pte_pfn(pte))) + categories |= PAGE_IS_PFNZERO; + } else if (is_swap_pte(pte)) { + categories |= PAGE_IS_SWAPPED; + if (!pte_swp_uffd_wp_any(pte)) + categories |= PAGE_IS_WRITTEN; + } + + return categories; +} + +static void make_uffd_wp_huge_pte(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t ptent) +{ + if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent)) + return; + + if (is_hugetlb_entry_migration(ptent)) + set_huge_pte_at(vma->vm_mm, addr, ptep, + pte_swp_mkuffd_wp(ptent)); + else if (!huge_pte_none(ptent)) + huge_ptep_modify_prot_commit(vma, addr, ptep, ptent, + huge_pte_mkuffd_wp(ptent)); + else + set_huge_pte_at(vma->vm_mm, addr, ptep, + make_pte_marker(PTE_MARKER_UFFD_WP)); +} +#endif /* CONFIG_HUGETLB_PAGE */ + +static bool pagemap_scan_is_interesting_page(unsigned long categories, + const struct pagemap_scan_private *p) +{ + categories ^= p->arg.category_inverted; + if ((categories & p->arg.category_mask) != p->arg.category_mask) + return false; + if (p->arg.category_anyof_mask && !(categories & p->arg.category_anyof_mask)) + return false; + + return true; +} + +static bool pagemap_scan_is_interesting_vma(unsigned long categories, + const struct pagemap_scan_private *p) +{ + unsigned long required = p->arg.category_mask & PAGE_IS_WPALLOWED; + + categories ^= p->arg.category_inverted; + if ((categories & required) != required) + return false; + + return true; +} + +static int pagemap_scan_test_walk(unsigned long start, unsigned long end, + struct mm_walk *walk) +{ + struct pagemap_scan_private *p = walk->private; + struct vm_area_struct *vma = walk->vma; + unsigned long vma_category = 0; + + if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma)) + vma_category |= PAGE_IS_WPALLOWED; + else if (p->arg.flags & PM_SCAN_CHECK_WPASYNC) + return -EPERM; + + if (vma->vm_flags & VM_PFNMAP) + return 1; + + if (!pagemap_scan_is_interesting_vma(vma_category, p)) + return 1; + + p->cur_vma_category = vma_category; + + return 0; +} + +static bool pagemap_scan_push_range(unsigned long categories, + struct pagemap_scan_private *p, + unsigned long addr, unsigned long end) +{ + struct page_region *cur_buf = &p->cur_buf; + + /* + * When there is no output buffer provided at all, the sentinel values + * won't match here. There is no other way for `cur_buf->end` to be + * non-zero other than it being non-empty. + */ + if (addr == cur_buf->end && categories == cur_buf->categories) { + cur_buf->end = end; + return true; + } + + if (cur_buf->end) { + if (p->vec_buf_index >= p->vec_buf_len) + return false; + + p->vec_buf[p->vec_buf_index++] = *cur_buf; + } + + cur_buf->start = addr; + cur_buf->end = end; + cur_buf->categories = categories; + + return true; +} + +static void pagemap_scan_backout_range(struct pagemap_scan_private *p, + unsigned long addr, unsigned long end, + unsigned long walk_end_addr) +{ + struct page_region *cur_buf = &p->cur_buf; + + if (cur_buf->start != addr) + cur_buf->end = addr; + else + cur_buf->start = cur_buf->end = 0; + + p->walk_end_addr = walk_end_addr; + p->found_pages -= (end - addr) / PAGE_SIZE; +} + +static int pagemap_scan_output(unsigned long categories, + struct pagemap_scan_private *p, + unsigned long addr, unsigned long *end) +{ + unsigned long n_pages, total_pages; + int ret = 0; + + if (!p->vec_buf) + return 0; + + categories &= p->arg.return_mask; + + n_pages = (*end - addr) / PAGE_SIZE; + if (check_add_overflow(p->found_pages, n_pages, &total_pages) || //TODO + total_pages > p->arg.max_pages) { + size_t n_too_much = total_pages - p->arg.max_pages; + *end -= n_too_much * PAGE_SIZE; + n_pages -= n_too_much; + ret = -ENOSPC; + } + + if (!pagemap_scan_push_range(categories, p, addr, *end)) { + *end = addr; + n_pages = 0; + ret = -ENOSPC; + } + + p->found_pages += n_pages; + if (ret) + p->walk_end_addr = *end; + + return ret; +} + +static int pagemap_scan_thp_entry(pmd_t *pmd, unsigned long start, + unsigned long end, struct mm_walk *walk) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + struct pagemap_scan_private *p = walk->private; + struct vm_area_struct *vma = walk->vma; + unsigned long categories; + spinlock_t *ptl; + int ret = 0; + + ptl = pmd_trans_huge_lock(pmd, vma); + if (!ptl) + return -ENOENT; + + categories = p->cur_vma_category | pagemap_thp_category(*pmd); + + if (!pagemap_scan_is_interesting_page(categories, p)) + goto out_unlock; + + ret = pagemap_scan_output(categories, p, start, &end); + if (start == end) + goto out_unlock; + + if (~p->arg.flags & PM_SCAN_WP_MATCHING) + goto out_unlock; + if (~categories & PAGE_IS_WRITTEN) + goto out_unlock; + + /* + * Break huge page into small pages if the WP operation + * need to be performed is on a portion of the huge page. + */ + if (end != start + HPAGE_SIZE) { + spin_unlock(ptl); + split_huge_pmd(vma, pmd, start); + pagemap_scan_backout_range(p, start, end, 0); + /* Indicate to caller for processing these as normal pages */ + return -ENOENT; + } + + make_uffd_wp_pmd(vma, start, pmd); + flush_tlb_range(vma, start, end); +out_unlock: + spin_unlock(ptl); + return ret; +#else /* !CONFIG_TRANSPARENT_HUGEPAGE */ + return -ENOENT; +#endif +} + +static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, + unsigned long end, struct mm_walk *walk) +{ + struct pagemap_scan_private *p = walk->private; + struct vm_area_struct *vma = walk->vma; + pte_t *pte, *start_pte; + unsigned long addr; + bool flush = false; + spinlock_t *ptl; + int ret; + + arch_enter_lazy_mmu_mode(); + + ret = pagemap_scan_thp_entry(pmd, start, end, walk); + if (ret != -ENOENT) { + arch_leave_lazy_mmu_mode(); + return ret; + } + + ret = 0; + start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl); + if (!pte) { + arch_leave_lazy_mmu_mode(); + walk->action = ACTION_AGAIN; + return 0; + } + + for (addr = start; addr != end; pte++, addr += PAGE_SIZE) { + unsigned long categories = p->cur_vma_category | + pagemap_page_category(p, vma, addr, ptep_get(pte)); + unsigned long next = addr + PAGE_SIZE; + + if (!pagemap_scan_is_interesting_page(categories, p)) + continue; + + ret = pagemap_scan_output(categories, p, addr, &next); + if (next == addr) + break; + + if (~p->arg.flags & PM_SCAN_WP_MATCHING) + continue; + if (~categories & PAGE_IS_WRITTEN) + continue; + + make_uffd_wp_pte(vma, addr, pte); + if (!flush) { + start = addr; + flush = true; + } + } + + if (flush) + flush_tlb_range(vma, start, addr); + + pte_unmap_unlock(start_pte, ptl); + arch_leave_lazy_mmu_mode(); + + cond_resched(); + return ret; +} + +#ifdef CONFIG_HUGETLB_PAGE +static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask, + unsigned long start, unsigned long end, + struct mm_walk *walk) +{ + struct pagemap_scan_private *p = walk->private; + struct vm_area_struct *vma = walk->vma; + unsigned long categories; + spinlock_t *ptl; + int ret = 0; + pte_t pte; + + if (~p->arg.flags & PM_SCAN_WP_MATCHING) { + /* Go the short route when not write-protecting pages. */ + + pte = huge_ptep_get(ptep); + categories = p->cur_vma_category | pagemap_hugetlb_category(pte); + + if (!pagemap_scan_is_interesting_page(categories, p)) + return 0; + + return pagemap_scan_output(categories, p, start, &end); + } + + i_mmap_lock_write(vma->vm_file->f_mapping); + ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep); + + pte = huge_ptep_get(ptep); + categories = p->cur_vma_category | pagemap_hugetlb_category(pte); + + if (!pagemap_scan_is_interesting_page(categories, p)) + goto out_unlock; + + ret = pagemap_scan_output(categories, p, start, &end); + if (start == end) + goto out_unlock; + + if (~categories & PAGE_IS_WRITTEN) + goto out_unlock; + + if (end != start + HPAGE_SIZE) { + /* Partial HugeTLB page WP isn't possible. */ + pagemap_scan_backout_range(p, start, end, start); + ret = -EINVAL; + goto out_unlock; + } + + make_uffd_wp_huge_pte(vma, start, ptep, pte); + flush_hugetlb_tlb_range(vma, start, end); + +out_unlock: + spin_unlock(ptl); + i_mmap_unlock_write(vma->vm_file->f_mapping); + + return ret; +} +#else +#define pagemap_scan_hugetlb_entry NULL +#endif + +static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, + int depth, struct mm_walk *walk) +{ + struct pagemap_scan_private *p = walk->private; + struct vm_area_struct *vma = walk->vma; + int ret, err; + + if (!vma || !pagemap_scan_is_interesting_page(p->cur_vma_category, p)) + return 0; + + ret = pagemap_scan_output(p->cur_vma_category, p, addr, &end); + if (addr == end) + return ret; + + if (~p->arg.flags & PM_SCAN_WP_MATCHING) + return ret; + + err = uffd_wp_range(vma, addr, end - addr, true); + if (err < 0) + ret = err; + + return ret; +} + +static const struct mm_walk_ops pagemap_scan_ops = { + .test_walk = pagemap_scan_test_walk, + .pmd_entry = pagemap_scan_pmd_entry, + .pte_hole = pagemap_scan_pte_hole, + .hugetlb_entry = pagemap_scan_hugetlb_entry, +}; + +static int pagemap_scan_get_args(struct pm_scan_arg *arg, + unsigned long uarg) +{ + if (copy_from_user(arg, (void __user *)uarg, sizeof(*arg))) + return -EFAULT; + + if (arg->size != sizeof(struct pm_scan_arg)) + return -EINVAL; + + /* Validate requested features */ + if (arg->flags & ~PM_SCAN_FLAGS) + return -EINVAL; + if ((arg->category_inverted | arg->category_mask | + arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES) + return -EINVAL; + + arg->start = untagged_addr((unsigned long)arg->start); + arg->end = untagged_addr((unsigned long)arg->end); + arg->vec = untagged_addr((unsigned long)arg->vec); + + /* Validate memory pointers */ + if (!IS_ALIGNED(arg->start, PAGE_SIZE)) + return -EINVAL; + if (!access_ok((void __user *)arg->start, arg->end - arg->start)) + return -EFAULT; + if (!arg->vec && arg->vec_len) + return -EFAULT; + if (arg->vec && !access_ok((void __user *)arg->vec, + arg->vec_len * sizeof(struct page_region))) + return -EFAULT; + + /* Fixup default values */ + arg->end = ALIGN(arg->end, PAGE_SIZE); + if (!arg->max_pages) + arg->max_pages = ULONG_MAX; + + return 0; +} + +static int pagemap_scan_writeback_args(struct pm_scan_arg *arg, + unsigned long uargl) +{ + struct pm_scan_arg __user *uarg = (void __user *)uargl; + + if (copy_to_user(&uarg->walk_end, &arg->walk_end, sizeof(arg->walk_end))) + return -EFAULT; + + return 0; +} + +static int pagemap_scan_init_bounce_buffer(struct pagemap_scan_private *p) +{ + if (!p->arg.vec_len) { + /* + * An arbitrary non-page-aligned sentinel value for + * pagemap_scan_push_range(). + */ + p->cur_buf.start = p->cur_buf.end = ULLONG_MAX; + if (p->arg.vec) + p->vec_buf = ZERO_SIZE_PTR; + return 0; + } + + /* + * Allocate a smaller buffer to get output from inside the page + * walk functions and walk the range in PAGEMAP_WALK_SIZE chunks. + * The last range is always stored in p.cur_buf to allow coalescing + * consecutive ranges that have the same categories returned across + * walk_page_range() calls. + */ + p->vec_buf_len = min_t(size_t, PAGEMAP_WALK_SIZE >> PAGE_SHIFT, + p->arg.vec_len - 1); + p->vec_buf = kmalloc_array(p->vec_buf_len, sizeof(*p->vec_buf), + GFP_KERNEL); + if (!p->vec_buf) + return -ENOMEM; + + p->vec_out = (struct page_region __user *)p->arg.vec; + + return 0; +} + +static int pagemap_scan_flush_buffer(struct pagemap_scan_private *p) +{ + const struct page_region *buf = p->vec_buf; + int n = (int)p->vec_buf_index; + + if (!n) + return 0; + + if (copy_to_user(p->vec_out, buf, n * sizeof(*buf))) + return -EFAULT; + + p->arg.vec_len -= n; + p->vec_out += n; + + p->vec_buf_index = 0; + p->vec_buf_len = min_t(size_t, p->vec_buf_len, p->arg.vec_len - 1); + + return n; +} + +static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg) +{ + struct mmu_notifier_range range; + struct pagemap_scan_private p; + unsigned long walk_start; + size_t n_ranges_out = 0; + int ret; + + memset(&p, 0, sizeof(p)); + ret = pagemap_scan_get_args(&p.arg, uarg); + if (ret) + return ret; + + p.masks_of_interest = MASKS_OF_INTEREST(p.arg); + ret = pagemap_scan_init_bounce_buffer(&p); + if (ret) + return ret; + + /* Protection change for the range is going to happen. */ + if (p.arg.flags & PM_SCAN_WP_MATCHING) { + mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0, + mm, p.arg.start, p.arg.end); + mmu_notifier_invalidate_range_start(&range); + } + + walk_start = p.arg.start; + for (; walk_start < p.arg.end; walk_start = p.arg.walk_end) { + int n_out; + + if (fatal_signal_pending(current)) { + ret = -EINTR; + break; + } + + ret = mmap_read_lock_killable(mm); + if (ret) + break; + ret = walk_page_range(mm, walk_start, p.arg.end, + &pagemap_scan_ops, &p); + mmap_read_unlock(mm); + + n_out = pagemap_scan_flush_buffer(&p); + if (n_out < 0) + ret = n_out; + else + n_ranges_out += n_out; + + if (!ret) + p.walk_end_addr = p.arg.end; + + if (ret != -ENOSPC || p.arg.vec_len - 1 == 0 || + p.found_pages == p.arg.max_pages) + break; + } + + if (p.cur_buf.start != p.cur_buf.end) { + if (copy_to_user(p.vec_out, &p.cur_buf, sizeof(p.cur_buf))) + ret = -EFAULT; + else + ++n_ranges_out; + } + + /* ENOSPC signifies early stop (buffer full) from the walk. */ + if (!ret || ret == -ENOSPC) + ret = n_ranges_out; + + p.arg.walk_end = p.walk_end_addr ? p.walk_end_addr : walk_start; + if (pagemap_scan_writeback_args(&p.arg, uarg)) + ret = -EFAULT; + + if (p.arg.flags & PM_SCAN_WP_MATCHING) + mmu_notifier_invalidate_range_end(&range); + + kfree(p.vec_buf); + return ret; +} + +static long do_pagemap_cmd(struct file *file, unsigned int cmd, + unsigned long arg) +{ + struct mm_struct *mm = file->private_data; + + switch (cmd) { + case PAGEMAP_SCAN: + return do_pagemap_scan(mm, arg); + + default: + return -EINVAL; + } +} + const struct file_operations proc_pagemap_operations = { .llseek = mem_lseek, /* borrow this */ .read = pagemap_read, .open = pagemap_open, .release = pagemap_release, + .unlocked_ioctl = do_pagemap_cmd, + .compat_ioctl = do_pagemap_cmd, }; #endif /* CONFIG_PROC_PAGE_MONITOR */
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 0a393bc02f25b..8f8ff07453f22 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -259,6 +259,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long cp_flags);
bool is_hugetlb_entry_migration(pte_t pte); +bool is_hugetlb_entry_hwpoisoned(pte_t pte); void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
#else /* !CONFIG_HUGETLB_PAGE */ diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index b7b56871029c5..1c9d38af1015e 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -305,4 +305,63 @@ typedef int __bitwise __kernel_rwf_t; #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ RWF_APPEND)
+/* Pagemap ioctl */ +#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg) + +/* Bits are set in flags of the page_region and masks in pm_scan_args */ +#define PAGE_IS_WPALLOWED (1 << 0) +#define PAGE_IS_WRITTEN (1 << 1) +#define PAGE_IS_FILE (1 << 2) +#define PAGE_IS_PRESENT (1 << 3) +#define PAGE_IS_SWAPPED (1 << 4) +#define PAGE_IS_PFNZERO (1 << 5) +#define PAGE_IS_HUGE (1 << 6) + +/* + * struct page_region - Page region with flags + * @start: Start of the region + * @end: End of the region (exclusive) + * @categories: PAGE_IS_* category bitmask for the region + */ +struct page_region { + __u64 start; + __u64 end; + __u64 categories; +}; + +/* Flags for PAGEMAP_SCAN ioctl */ +#define PM_SCAN_WP_MATCHING (1 << 0) /* Write protect the pages matched. */ +#define PM_SCAN_CHECK_WPASYNC (1 << 1) /* Abort the scan when a non-WP-enabled page is found. */ + +/* + * struct pm_scan_arg - Pagemap ioctl argument + * @size: Size of the structure + * @flags: Flags for the IOCTL + * @start: Starting address of the region + * @end: Ending address of the region + * @walk_end Address where the scan stopped (written by kernel). + * walk_end == end informs that the scan completed on entire range. + * @vec: Address of page_region struct array for output + * @vec_len: Length of the page_region struct array + * @max_pages: Optional limit for number of returned pages (0 = disabled) + * @category_inverted: PAGE_IS_* categories which values match if 0 instead of 1 + * @category_mask: Skip pages for which any category doesn't match + * @category_anyof_mask: Skip pages for which no category matches + * @return_mask: PAGE_IS_* categories that are to be reported in `page_region`s returned + */ +struct pm_scan_arg { + __u64 size; + __u64 flags; + __u64 start; + __u64 end; + __u64 walk_end; + __u64 vec; + __u64 vec_len; + __u64 max_pages; + __u64 category_inverted; + __u64 category_mask; + __u64 category_anyof_mask; + __u64 return_mask; +}; + #endif /* _UAPI_LINUX_FS_H */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a073e6ed8900b..3b07db0a4f2d9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5008,7 +5008,7 @@ bool is_hugetlb_entry_migration(pte_t pte) return false; }
-static bool is_hugetlb_entry_hwpoisoned(pte_t pte) +bool is_hugetlb_entry_hwpoisoned(pte_t pte) { swp_entry_t swp;
On Wed, 9 Aug 2023 at 08:16, Muhammad Usama Anjum usama.anjum@collabora.com wrote:
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear the info about page table entries. The following operations are supported in this ioctl:
- Get the information if the pages have Async Write-Protection enabled (``PAGE_IS_WPALLOWED``), have been written to (``PAGE_IS_WRITTEN``), file mapped (``PAGE_IS_FILE``), present (``PAGE_IS_PRESENT``), swapped (``PAGE_IS_SWAPPED``) or page has pfn zero (``PAGE_IS_PFNZERO``).
- Find pages which have been written to and/or write protect (atomic ``PM_SCAN_WP_MATCHING + PM_SCAN_CHECK_WPASYNC``) the pages atomically. The (``PM_SCAN_WP_MATCHING``) is used to WP the matched pages. The (``PM_SCAN_CHECK_WPASYNC``) aborts the operation if non-Async-Write-Protected pages are found. Get is automatically performed if output buffer is specified.
This IOCTL can be extended to get information about more PTE bits. The entire address range passed by user [start, end) is scanned until either the user provided buffer is full or max_pages have been found.
Reviewed-by: Andrei Vagin avagin@gmail.com Reviewed-by: Michał Mirosław mirq-linux@rere.qmqm.pl
Still applies, thanks! Please find some mostly-polishing comments below.
Signed-off-by: Michał Mirosław mirq-linux@rere.qmqm.pl Signed-off-by: Muhammad Usama Anjum usama.anjum@collabora.com
Changes in v28:
- Fix walk_end one last time after doing through testing
Changes in v27:
- Add PAGE_IS_HUGE
It seems to be missing from the commitmsg, though. But maybe listing all the flags there is redundant, since a doc is coming anyway and the values are listed in the .h?
[...]
--- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c
[...]
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static unsigned long pagemap_thp_category(pmd_t pmd) +{
unsigned long categories = PAGE_IS_HUGE;if (pmd_present(pmd)) {categories |= PAGE_IS_PRESENT;if (!pmd_uffd_wp(pmd))categories |= PAGE_IS_WRITTEN;if (is_zero_pfn(pmd_pfn(pmd)))categories |= PAGE_IS_PFNZERO;} else if (is_swap_pmd(pmd)) {categories |= PAGE_IS_SWAPPED;if (!pmd_swp_uffd_wp(pmd))categories |= PAGE_IS_WRITTEN;}return categories;+}
I guess THPs can't be file-backed currently, but can we somehow mark this assumption so it can be easily found if the capability arrives?
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+#ifdef CONFIG_HUGETLB_PAGE +static unsigned long pagemap_hugetlb_category(pte_t pte) +{
unsigned long categories = PAGE_IS_HUGE;if (pte_present(pte)) {categories |= PAGE_IS_PRESENT;if (!huge_pte_uffd_wp(pte))categories |= PAGE_IS_WRITTEN;if (!PageAnon(pte_page(pte)))categories |= PAGE_IS_FILE;if (is_zero_pfn(pte_pfn(pte)))categories |= PAGE_IS_PFNZERO;} else if (is_swap_pte(pte)) {categories |= PAGE_IS_SWAPPED;if (!pte_swp_uffd_wp_any(pte))categories |= PAGE_IS_WRITTEN;}
BTW, can a HugeTLB page be file-backed and swapped out?
+static void pagemap_scan_backout_range(struct pagemap_scan_private *p,
unsigned long addr, unsigned long end,unsigned long walk_end_addr)+{
struct page_region *cur_buf = &p->cur_buf;if (cur_buf->start != addr)cur_buf->end = addr;elsecur_buf->start = cur_buf->end = 0;p->walk_end_addr = walk_end_addr;p->found_pages -= (end - addr) / PAGE_SIZE;+}
When would `walk_end_addr` be different from `end`? Maybe it would be easier to understand if the `p->walk_end_addr` update was pushed to the callers? (Actually: the one that wants to change it.)
+static int pagemap_scan_output(unsigned long categories,
struct pagemap_scan_private *p,unsigned long addr, unsigned long *end)+{
unsigned long n_pages, total_pages;int ret = 0;if (!p->vec_buf)return 0;categories &= p->arg.return_mask;n_pages = (*end - addr) / PAGE_SIZE;if (check_add_overflow(p->found_pages, n_pages, &total_pages) || //TODOtotal_pages > p->arg.max_pages) {
Re: TODO: Is there anything left to change here?
size_t n_too_much = total_pages - p->arg.max_pages;*end -= n_too_much * PAGE_SIZE;n_pages -= n_too_much;ret = -ENOSPC;}
[...]
+static int pagemap_scan_thp_entry(pmd_t *pmd, unsigned long start,
unsigned long end, struct mm_walk *walk)+{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
struct pagemap_scan_private *p = walk->private;struct vm_area_struct *vma = walk->vma;unsigned long categories;spinlock_t *ptl;int ret = 0;ptl = pmd_trans_huge_lock(pmd, vma);if (!ptl)return -ENOENT;categories = p->cur_vma_category | pagemap_thp_category(*pmd);if (!pagemap_scan_is_interesting_page(categories, p))goto out_unlock;ret = pagemap_scan_output(categories, p, start, &end);if (start == end)goto out_unlock;if (~p->arg.flags & PM_SCAN_WP_MATCHING)goto out_unlock;if (~categories & PAGE_IS_WRITTEN)goto out_unlock;/** Break huge page into small pages if the WP operation* need to be performed is on a portion of the huge page.
"needs to be performed on ..."
*/if (end != start + HPAGE_SIZE) {spin_unlock(ptl);split_huge_pmd(vma, pmd, start);pagemap_scan_backout_range(p, start, end, 0);/* Indicate to caller for processing these as normal pages */
Nit: "Report as if there was no THP." ?
+static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
unsigned long end, struct mm_walk *walk)+{
struct pagemap_scan_private *p = walk->private;struct vm_area_struct *vma = walk->vma;pte_t *pte, *start_pte;unsigned long addr;bool flush = false;spinlock_t *ptl;int ret;arch_enter_lazy_mmu_mode();ret = pagemap_scan_thp_entry(pmd, start, end, walk);if (ret != -ENOENT) {arch_leave_lazy_mmu_mode();return ret;}ret = 0;start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);if (!pte) {arch_leave_lazy_mmu_mode();walk->action = ACTION_AGAIN;return 0;}for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {unsigned long categories = p->cur_vma_category |pagemap_page_category(p, vma, addr, ptep_get(pte));unsigned long next = addr + PAGE_SIZE;if (!pagemap_scan_is_interesting_page(categories, p))continue;ret = pagemap_scan_output(categories, p, addr, &next);if (next == addr)break;if (~p->arg.flags & PM_SCAN_WP_MATCHING)continue;if (~categories & PAGE_IS_WRITTEN)continue;make_uffd_wp_pte(vma, addr, pte);if (!flush) {start = addr;flush = true;}
A quick improvement:
/* instead of `flush` declaration */ unsigned long flush_end = 0;
if (!flush_end) start = addr; flush_end = next;
}if (flush)flush_tlb_range(vma, start, addr);
And here: if (flush_end) flush_tlb_range(vma, start, flush_end);
pte_unmap_unlock(start_pte, ptl);arch_leave_lazy_mmu_mode();cond_resched();return ret;+}
[...]
+static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg) +{
struct mmu_notifier_range range;struct pagemap_scan_private p;unsigned long walk_start;size_t n_ranges_out = 0;int ret;memset(&p, 0, sizeof(p));ret = pagemap_scan_get_args(&p.arg, uarg);if (ret)return ret;p.masks_of_interest = MASKS_OF_INTEREST(p.arg);
Please inline the macro as here is the only use of it.
[...]
walk_start = p.arg.start;for (; walk_start < p.arg.end; walk_start = p.arg.walk_end) {
Nit: the initialization statement can now be part of the for().
int n_out;if (fatal_signal_pending(current)) {ret = -EINTR;break;}ret = mmap_read_lock_killable(mm);if (ret)break;ret = walk_page_range(mm, walk_start, p.arg.end,&pagemap_scan_ops, &p);mmap_read_unlock(mm);n_out = pagemap_scan_flush_buffer(&p);if (n_out < 0)ret = n_out;elsen_ranges_out += n_out;if (!ret)p.walk_end_addr = p.arg.end;if (ret != -ENOSPC || p.arg.vec_len - 1 == 0 ||p.found_pages == p.arg.max_pages)break;
The second condition is equivalent to `p.arg.vec_len == 1`, but why is that an ending condition? Isn't the last entry enough to gather one more range? (The walk could have returned -ENOSPC due to buffer full and after flushing it could continue with the last free entry.)
[...]
--- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -259,6 +259,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long cp_flags);
bool is_hugetlb_entry_migration(pte_t pte); +bool is_hugetlb_entry_hwpoisoned(pte_t pte); void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
#else /* !CONFIG_HUGETLB_PAGE */ diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index b7b56871029c5..1c9d38af1015e 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -305,4 +305,63 @@ typedef int __bitwise __kernel_rwf_t; #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ RWF_APPEND)
+/* Pagemap ioctl */ +#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg)
+/* Bits are set in flags of the page_region and masks in pm_scan_args */
"Bitmasks provided in pm_scan_args masks and reported in page_region.categories."
+#define PAGE_IS_WPALLOWED (1 << 0) +#define PAGE_IS_WRITTEN (1 << 1) +#define PAGE_IS_FILE (1 << 2) +#define PAGE_IS_PRESENT (1 << 3) +#define PAGE_IS_SWAPPED (1 << 4) +#define PAGE_IS_PFNZERO (1 << 5) +#define PAGE_IS_HUGE (1 << 6)
+/*
- struct page_region - Page region with flags
- @start: Start of the region
- @end: End of the region (exclusive)
- @categories: PAGE_IS_* category bitmask for the region
- */
+struct page_region {
__u64 start;__u64 end;__u64 categories;+};
+/* Flags for PAGEMAP_SCAN ioctl */ +#define PM_SCAN_WP_MATCHING (1 << 0) /* Write protect the pages matched. */ +#define PM_SCAN_CHECK_WPASYNC (1 << 1) /* Abort the scan when a non-WP-enabled page is found. */
+/*
- struct pm_scan_arg - Pagemap ioctl argument
- @size: Size of the structure
- @flags: Flags for the IOCTL
- @start: Starting address of the region
- @end: Ending address of the region
- @walk_end Address where the scan stopped (written by kernel).
walk_end == end informs that the scan completed on entire range.
Can we ensure this holds also for the tagged pointers?
- @vec: Address of page_region struct array for output
- @vec_len: Length of the page_region struct array
- @max_pages: Optional limit for number of returned pages (0 = disabled)
- @category_inverted: PAGE_IS_* categories which values match if 0 instead of 1
- @category_mask: Skip pages for which any category doesn't match
- @category_anyof_mask: Skip pages for which no category matches
- @return_mask: PAGE_IS_* categories that are to be reported in `page_region`s returned
- */
[...]
Best Regards Michał Mirosław
On 8/10/23 10:32 PM, Michał Mirosław wrote:
On Wed, 9 Aug 2023 at 08:16, Muhammad Usama Anjum usama.anjum@collabora.com wrote:
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear the info about page table entries. The following operations are supported in this ioctl:
- Get the information if the pages have Async Write-Protection enabled (``PAGE_IS_WPALLOWED``), have been written to (``PAGE_IS_WRITTEN``), file mapped (``PAGE_IS_FILE``), present (``PAGE_IS_PRESENT``), swapped (``PAGE_IS_SWAPPED``) or page has pfn zero (``PAGE_IS_PFNZERO``).
- Find pages which have been written to and/or write protect (atomic ``PM_SCAN_WP_MATCHING + PM_SCAN_CHECK_WPASYNC``) the pages atomically. The (``PM_SCAN_WP_MATCHING``) is used to WP the matched pages. The (``PM_SCAN_CHECK_WPASYNC``) aborts the operation if non-Async-Write-Protected pages are found. Get is automatically performed if output buffer is specified.
This IOCTL can be extended to get information about more PTE bits. The entire address range passed by user [start, end) is scanned until either the user provided buffer is full or max_pages have been found.
Reviewed-by: Andrei Vagin avagin@gmail.com Reviewed-by: Michał Mirosław mirq-linux@rere.qmqm.pl
Still applies, thanks! Please find some mostly-polishing comments below.
Signed-off-by: Michał Mirosław mirq-linux@rere.qmqm.pl Signed-off-by: Muhammad Usama Anjum usama.anjum@collabora.com
Changes in v28:
- Fix walk_end one last time after doing through testing
Changes in v27:
- Add PAGE_IS_HUGE
It seems to be missing from the commitmsg, though. But maybe listing all the flags there is redundant, since a doc is coming anyway and the values are listed in the .h?
Yeah, fs.h and doc has all these mentioned.
[...]
--- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c
[...]
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static unsigned long pagemap_thp_category(pmd_t pmd) +{
unsigned long categories = PAGE_IS_HUGE;if (pmd_present(pmd)) {categories |= PAGE_IS_PRESENT;if (!pmd_uffd_wp(pmd))categories |= PAGE_IS_WRITTEN;if (is_zero_pfn(pmd_pfn(pmd)))categories |= PAGE_IS_PFNZERO;} else if (is_swap_pmd(pmd)) {categories |= PAGE_IS_SWAPPED;if (!pmd_swp_uffd_wp(pmd))categories |= PAGE_IS_WRITTEN;}return categories;+}
I guess THPs can't be file-backed currently, but can we somehow mark this assumption so it can be easily found if the capability arrives?
Yeah, THPs cannot be file backed. Lets not care for some feature which may not arrive in several years or eternity.
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+#ifdef CONFIG_HUGETLB_PAGE +static unsigned long pagemap_hugetlb_category(pte_t pte) +{
unsigned long categories = PAGE_IS_HUGE;if (pte_present(pte)) {categories |= PAGE_IS_PRESENT;if (!huge_pte_uffd_wp(pte))categories |= PAGE_IS_WRITTEN;if (!PageAnon(pte_page(pte)))categories |= PAGE_IS_FILE;if (is_zero_pfn(pte_pfn(pte)))categories |= PAGE_IS_PFNZERO;} else if (is_swap_pte(pte)) {categories |= PAGE_IS_SWAPPED;if (!pte_swp_uffd_wp_any(pte))categories |= PAGE_IS_WRITTEN;}BTW, can a HugeTLB page be file-backed and swapped out?
Accourding to pagemap_hugetlb_range(), file-backed HugeTLB page cannot be swapped.
+static void pagemap_scan_backout_range(struct pagemap_scan_private *p,
unsigned long addr, unsigned long end,unsigned long walk_end_addr)+{
struct page_region *cur_buf = &p->cur_buf;if (cur_buf->start != addr)cur_buf->end = addr;elsecur_buf->start = cur_buf->end = 0;p->walk_end_addr = walk_end_addr;p->found_pages -= (end - addr) / PAGE_SIZE;+}
When would `walk_end_addr` be different from `end`? Maybe it would be easier to understand if the `p->walk_end_addr` update was pushed to the callers? (Actually: the one that wants to change it.)
Not very major thing, I've made the changes for next revision.
+static int pagemap_scan_output(unsigned long categories,
struct pagemap_scan_private *p,unsigned long addr, unsigned long *end)+{
unsigned long n_pages, total_pages;int ret = 0;if (!p->vec_buf)return 0;categories &= p->arg.return_mask;n_pages = (*end - addr) / PAGE_SIZE;if (check_add_overflow(p->found_pages, n_pages, &total_pages) || //TODOtotal_pages > p->arg.max_pages) {Re: TODO: Is there anything left to change here?
No, I'll remove it.
size_t n_too_much = total_pages - p->arg.max_pages;*end -= n_too_much * PAGE_SIZE;n_pages -= n_too_much;ret = -ENOSPC;}[...]
+static int pagemap_scan_thp_entry(pmd_t *pmd, unsigned long start,
unsigned long end, struct mm_walk *walk)+{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
struct pagemap_scan_private *p = walk->private;struct vm_area_struct *vma = walk->vma;unsigned long categories;spinlock_t *ptl;int ret = 0;ptl = pmd_trans_huge_lock(pmd, vma);if (!ptl)return -ENOENT;categories = p->cur_vma_category | pagemap_thp_category(*pmd);if (!pagemap_scan_is_interesting_page(categories, p))goto out_unlock;ret = pagemap_scan_output(categories, p, start, &end);if (start == end)goto out_unlock;if (~p->arg.flags & PM_SCAN_WP_MATCHING)goto out_unlock;if (~categories & PAGE_IS_WRITTEN)goto out_unlock;/** Break huge page into small pages if the WP operation* need to be performed is on a portion of the huge page."needs to be performed on ..."
Okay. I'll update.
*/if (end != start + HPAGE_SIZE) {spin_unlock(ptl);split_huge_pmd(vma, pmd, start);pagemap_scan_backout_range(p, start, end, 0);/* Indicate to caller for processing these as normal pages */Nit: "Report as if there was no THP." ?
Sure. I'll update.
+static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
unsigned long end, struct mm_walk *walk)+{
struct pagemap_scan_private *p = walk->private;struct vm_area_struct *vma = walk->vma;pte_t *pte, *start_pte;unsigned long addr;bool flush = false;spinlock_t *ptl;int ret;arch_enter_lazy_mmu_mode();ret = pagemap_scan_thp_entry(pmd, start, end, walk);if (ret != -ENOENT) {arch_leave_lazy_mmu_mode();return ret;}ret = 0;start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);if (!pte) {arch_leave_lazy_mmu_mode();walk->action = ACTION_AGAIN;return 0;}for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {unsigned long categories = p->cur_vma_category |pagemap_page_category(p, vma, addr, ptep_get(pte));unsigned long next = addr + PAGE_SIZE;if (!pagemap_scan_is_interesting_page(categories, p))continue;ret = pagemap_scan_output(categories, p, addr, &next);if (next == addr)break;if (~p->arg.flags & PM_SCAN_WP_MATCHING)continue;if (~categories & PAGE_IS_WRITTEN)continue;make_uffd_wp_pte(vma, addr, pte);if (!flush) {start = addr;flush = true;}A quick improvement:
/* instead of `flush` declaration */ unsigned long flush_end = 0;
if (!flush_end) start = addr; flush_end = next;
I'll update.
}if (flush)flush_tlb_range(vma, start, addr);And here: if (flush_end) flush_tlb_range(vma, start, flush_end);
pte_unmap_unlock(start_pte, ptl);arch_leave_lazy_mmu_mode();cond_resched();return ret;+}
[...]
+static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg) +{
struct mmu_notifier_range range;struct pagemap_scan_private p;unsigned long walk_start;size_t n_ranges_out = 0;int ret;memset(&p, 0, sizeof(p));ret = pagemap_scan_get_args(&p.arg, uarg);if (ret)return ret;p.masks_of_interest = MASKS_OF_INTEREST(p.arg);Please inline the macro as here is the only use of it.
Got it.
[...]
walk_start = p.arg.start;for (; walk_start < p.arg.end; walk_start = p.arg.walk_end) {Nit: the initialization statement can now be part of the for().
Sure.
int n_out;if (fatal_signal_pending(current)) {ret = -EINTR;break;}ret = mmap_read_lock_killable(mm);if (ret)break;ret = walk_page_range(mm, walk_start, p.arg.end,&pagemap_scan_ops, &p);mmap_read_unlock(mm);n_out = pagemap_scan_flush_buffer(&p);if (n_out < 0)ret = n_out;elsen_ranges_out += n_out;if (!ret)p.walk_end_addr = p.arg.end;if (ret != -ENOSPC || p.arg.vec_len - 1 == 0 ||p.found_pages == p.arg.max_pages)break;The second condition is equivalent to `p.arg.vec_len == 1`, but why is that an ending condition? Isn't the last entry enough to gather one more range? (The walk could have returned -ENOSPC due to buffer full and after flushing it could continue with the last free entry.)
Now we are walking the entire range walk_page_range(). We don't break loop when we get -ENOSPC as this error may only mean that the temporary buffer is full. So we need check if max pages have been found or output buffer is full or ret is 0 or any other error. When p.arg.vec_len = 1 is end condition as the last entry is in cur. As we have walked over the entire range, cur must be full after which the walk returned.
So current condition is necessary. I've double checked it. I'll change it to `p.arg.vec_len == 1`.
[...]
--- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -259,6 +259,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long cp_flags);
bool is_hugetlb_entry_migration(pte_t pte); +bool is_hugetlb_entry_hwpoisoned(pte_t pte); void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
#else /* !CONFIG_HUGETLB_PAGE */ diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index b7b56871029c5..1c9d38af1015e 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -305,4 +305,63 @@ typedef int __bitwise __kernel_rwf_t; #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ RWF_APPEND)
+/* Pagemap ioctl */ +#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg)
+/* Bits are set in flags of the page_region and masks in pm_scan_args */
"Bitmasks provided in pm_scan_args masks and reported in page_region.categories."
Okay.
+#define PAGE_IS_WPALLOWED (1 << 0) +#define PAGE_IS_WRITTEN (1 << 1) +#define PAGE_IS_FILE (1 << 2) +#define PAGE_IS_PRESENT (1 << 3) +#define PAGE_IS_SWAPPED (1 << 4) +#define PAGE_IS_PFNZERO (1 << 5) +#define PAGE_IS_HUGE (1 << 6)
+/*
- struct page_region - Page region with flags
- @start: Start of the region
- @end: End of the region (exclusive)
- @categories: PAGE_IS_* category bitmask for the region
- */
+struct page_region {
__u64 start;__u64 end;__u64 categories;+};
+/* Flags for PAGEMAP_SCAN ioctl */ +#define PM_SCAN_WP_MATCHING (1 << 0) /* Write protect the pages matched. */ +#define PM_SCAN_CHECK_WPASYNC (1 << 1) /* Abort the scan when a non-WP-enabled page is found. */
+/*
- struct pm_scan_arg - Pagemap ioctl argument
- @size: Size of the structure
- @flags: Flags for the IOCTL
- @start: Starting address of the region
- @end: Ending address of the region
- @walk_end Address where the scan stopped (written by kernel).
walk_end == end informs that the scan completed on entire range.Can we ensure this holds also for the tagged pointers?
No, we cannot.
- @vec: Address of page_region struct array for output
- @vec_len: Length of the page_region struct array
- @max_pages: Optional limit for number of returned pages (0 = disabled)
- @category_inverted: PAGE_IS_* categories which values match if 0 instead of 1
- @category_mask: Skip pages for which any category doesn't match
- @category_anyof_mask: Skip pages for which no category matches
- @return_mask: PAGE_IS_* categories that are to be reported in `page_region`s returned
- */
[...]
Best Regards Michał Mirosław
On Fri, Aug 11, 2023 at 05:02:44PM +0500, Muhammad Usama Anjum wrote:
On 8/10/23 10:32 PM, Michał Mirosław wrote:
On Wed, 9 Aug 2023 at 08:16, Muhammad Usama Anjum usama.anjum@collabora.com wrote:
[...]
--- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c
[...]
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static unsigned long pagemap_thp_category(pmd_t pmd) +{
unsigned long categories = PAGE_IS_HUGE;if (pmd_present(pmd)) {categories |= PAGE_IS_PRESENT;if (!pmd_uffd_wp(pmd))categories |= PAGE_IS_WRITTEN;if (is_zero_pfn(pmd_pfn(pmd)))categories |= PAGE_IS_PFNZERO;} else if (is_swap_pmd(pmd)) {categories |= PAGE_IS_SWAPPED;if (!pmd_swp_uffd_wp(pmd))categories |= PAGE_IS_WRITTEN;}return categories;+}
I guess THPs can't be file-backed currently, but can we somehow mark this assumption so it can be easily found if the capability arrives?
Yeah, THPs cannot be file backed. Lets not care for some feature which may not arrive in several years or eternity.
Yes, it might not arrive. But please add at least a comment, so that it is clearly visible that lack if PAGE_IS_FILE here is intentional.
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+#ifdef CONFIG_HUGETLB_PAGE +static unsigned long pagemap_hugetlb_category(pte_t pte) +{
unsigned long categories = PAGE_IS_HUGE;if (pte_present(pte)) {categories |= PAGE_IS_PRESENT;if (!huge_pte_uffd_wp(pte))categories |= PAGE_IS_WRITTEN;if (!PageAnon(pte_page(pte)))categories |= PAGE_IS_FILE;if (is_zero_pfn(pte_pfn(pte)))categories |= PAGE_IS_PFNZERO;} else if (is_swap_pte(pte)) {categories |= PAGE_IS_SWAPPED;if (!pte_swp_uffd_wp_any(pte))categories |= PAGE_IS_WRITTEN;}BTW, can a HugeTLB page be file-backed and swapped out?
Accourding to pagemap_hugetlb_range(), file-backed HugeTLB page cannot be swapped.
Here too a comment that leaving out this case is intentional would be useful.
[...]
walk_start = p.arg.start;for (; walk_start < p.arg.end; walk_start = p.arg.walk_end) {
[...[
ret = mmap_read_lock_killable(mm);if (ret)break;ret = walk_page_range(mm, walk_start, p.arg.end,&pagemap_scan_ops, &p);mmap_read_unlock(mm);
[...]
if (ret != -ENOSPC || p.arg.vec_len - 1 == 0 ||p.found_pages == p.arg.max_pages)break;The second condition is equivalent to `p.arg.vec_len == 1`, but why is that an ending condition? Isn't the last entry enough to gather one more range? (The walk could have returned -ENOSPC due to buffer full and after flushing it could continue with the last free entry.)
Now we are walking the entire range walk_page_range(). We don't break loop when we get -ENOSPC as this error may only mean that the temporary buffer is full. So we need check if max pages have been found or output buffer is full or ret is 0 or any other error. When p.arg.vec_len = 1 is end condition as the last entry is in cur. As we have walked over the entire range, cur must be full after which the walk returned.
So current condition is necessary. I've double checked it. I'll change it to `p.arg.vec_len == 1`.
If we have walked the whole range, then the loop will end anyway due to `walk_start < walk_end` not held in the `for()`'s condition.
[...]
+/*
- struct pm_scan_arg - Pagemap ioctl argument
- @size: Size of the structure
- @flags: Flags for the IOCTL
- @start: Starting address of the region
- @end: Ending address of the region
- @walk_end Address where the scan stopped (written by kernel).
walk_end == end informs that the scan completed on entire range.Can we ensure this holds also for the tagged pointers?
No, we cannot.
So this need explanation in the comment here. (Though I'd still like to know how the address tags are supposed to be used from someone that knows them.)
Best Regards Michał Mirosław
On 8/11/23 6:32 PM, Michał Mirosław wrote:
On Fri, Aug 11, 2023 at 05:02:44PM +0500, Muhammad Usama Anjum wrote:
On 8/10/23 10:32 PM, Michał Mirosław wrote:
On Wed, 9 Aug 2023 at 08:16, Muhammad Usama Anjum usama.anjum@collabora.com wrote:
[...]
--- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c
[...]
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static unsigned long pagemap_thp_category(pmd_t pmd) +{
unsigned long categories = PAGE_IS_HUGE;if (pmd_present(pmd)) {categories |= PAGE_IS_PRESENT;if (!pmd_uffd_wp(pmd))categories |= PAGE_IS_WRITTEN;if (is_zero_pfn(pmd_pfn(pmd)))categories |= PAGE_IS_PFNZERO;} else if (is_swap_pmd(pmd)) {categories |= PAGE_IS_SWAPPED;if (!pmd_swp_uffd_wp(pmd))categories |= PAGE_IS_WRITTEN;}return categories;+}
I guess THPs can't be file-backed currently, but can we somehow mark this assumption so it can be easily found if the capability arrives?
Yeah, THPs cannot be file backed. Lets not care for some feature which may not arrive in several years or eternity.
Yes, it might not arrive. But please add at least a comment, so that it is clearly visible that lack if PAGE_IS_FILE here is intentional.
I'll add a comment.
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+#ifdef CONFIG_HUGETLB_PAGE +static unsigned long pagemap_hugetlb_category(pte_t pte) +{
unsigned long categories = PAGE_IS_HUGE;if (pte_present(pte)) {categories |= PAGE_IS_PRESENT;if (!huge_pte_uffd_wp(pte))categories |= PAGE_IS_WRITTEN;if (!PageAnon(pte_page(pte)))categories |= PAGE_IS_FILE;if (is_zero_pfn(pte_pfn(pte)))categories |= PAGE_IS_PFNZERO;} else if (is_swap_pte(pte)) {categories |= PAGE_IS_SWAPPED;if (!pte_swp_uffd_wp_any(pte))categories |= PAGE_IS_WRITTEN;}BTW, can a HugeTLB page be file-backed and swapped out?
Accourding to pagemap_hugetlb_range(), file-backed HugeTLB page cannot be swapped.
Here too a comment that leaving out this case is intentional would be useful.
Sure.
[...]
walk_start = p.arg.start;for (; walk_start < p.arg.end; walk_start = p.arg.walk_end) {[...[
ret = mmap_read_lock_killable(mm);if (ret)break;ret = walk_page_range(mm, walk_start, p.arg.end,&pagemap_scan_ops, &p);mmap_read_unlock(mm);[...]
if (ret != -ENOSPC || p.arg.vec_len - 1 == 0 ||p.found_pages == p.arg.max_pages)break;The second condition is equivalent to `p.arg.vec_len == 1`, but why is that an ending condition? Isn't the last entry enough to gather one more range? (The walk could have returned -ENOSPC due to buffer full and after flushing it could continue with the last free entry.)
Now we are walking the entire range walk_page_range(). We don't break loop when we get -ENOSPC as this error may only mean that the temporary buffer is full. So we need check if max pages have been found or output buffer is full or ret is 0 or any other error. When p.arg.vec_len = 1 is end condition as the last entry is in cur. As we have walked over the entire range, cur must be full after which the walk returned.
So current condition is necessary. I've double checked it. I'll change it to `p.arg.vec_len == 1`.
If we have walked the whole range, then the loop will end anyway due to `walk_start < walk_end` not held in the `for()`'s condition.
Sorry, for not explaining to-the-point. Why would we walk the entire range when we should recognize that the output buffer is full and break the loop?
I've test cases written for this case. If I remove `p.arg.vec_len == 1` check, there is infinite loop for walking. So we are doing correct thing here.
[...]
+/*
- struct pm_scan_arg - Pagemap ioctl argument
- @size: Size of the structure
- @flags: Flags for the IOCTL
- @start: Starting address of the region
- @end: Ending address of the region
- @walk_end Address where the scan stopped (written by kernel).
walk_end == end informs that the scan completed on entire range.Can we ensure this holds also for the tagged pointers?
No, we cannot.
So this need explanation in the comment here. (Though I'd still like to know how the address tags are supposed to be used from someone that knows them.)
I've looked at some documentations (presentations/talks) about tags. Tags is more like userspace feature. Kernel should just ignore them for our use case. I'll add comment.
Best Regards Michał Mirosław
On Fri, Aug 11, 2023 at 08:30:16PM +0500, Muhammad Usama Anjum wrote:
On 8/11/23 6:32 PM, Michał Mirosław wrote:
On Fri, Aug 11, 2023 at 05:02:44PM +0500, Muhammad Usama Anjum wrote:
Now we are walking the entire range walk_page_range(). We don't break loop when we get -ENOSPC as this error may only mean that the temporary buffer is full. So we need check if max pages have been found or output buffer is full or ret is 0 or any other error. When p.arg.vec_len = 1 is end condition as the last entry is in cur. As we have walked over the entire range, cur must be full after which the walk returned.
So current condition is necessary. I've double checked it. I'll change it to `p.arg.vec_len == 1`.
If we have walked the whole range, then the loop will end anyway due to `walk_start < walk_end` not held in the `for()`'s condition.
Sorry, for not explaining to-the-point. Why would we walk the entire range when we should recognize that the output buffer is full and break the loop?
I've test cases written for this case. If I remove `p.arg.vec_len == 1` check, there is infinite loop for walking. So we are doing correct thing here.
It seems there is a bug somewhere then. I'll take a look at v29.
[...]
+/*
- struct pm_scan_arg - Pagemap ioctl argument
- @size: Size of the structure
- @flags: Flags for the IOCTL
- @start: Starting address of the region
- @end: Ending address of the region
- @walk_end Address where the scan stopped (written by kernel).
walk_end == end informs that the scan completed on entire range.Can we ensure this holds also for the tagged pointers?
No, we cannot.
So this need explanation in the comment here. (Though I'd still like to know how the address tags are supposed to be used from someone that knows them.)
I've looked at some documentations (presentations/talks) about tags. Tags is more like userspace feature. Kernel should just ignore them for our use case. I'll add comment.
Kernel does ignore them when reading, but what about returning a tagged pointer? How that should work? In case of `walk_end` we can safely copy the tag from `end` or `start` when we return exactly on of those. But what about other addresses? When fed back as `start` any tag will work, so the question is only what to do with pointers in the middle? We can clear those of course - this should be mentioned in the doc - so userspace always gets a predictable value (note: 'predictable' does not require treating `start` and `end` the same way as addresses between them, just that what happens is well defined). (I think making `walk_end` == `end` work regardless of pointer tagging will make userspace happier, but I guess doc will also make it workable. And I'm repeating myself. ;-)
Best Regards Michał Mirosław
On Tue, Aug 8, 2023 at 11:16 PM Muhammad Usama Anjum usama.anjum@collabora.com wrote:
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear the info about page table entries. The following operations are supported in this ioctl:
- Get the information if the pages have Async Write-Protection enabled (``PAGE_IS_WPALLOWED``), have been written to (``PAGE_IS_WRITTEN``), file mapped (``PAGE_IS_FILE``), present (``PAGE_IS_PRESENT``), swapped (``PAGE_IS_SWAPPED``) or page has pfn zero (``PAGE_IS_PFNZERO``).
- Find pages which have been written to and/or write protect (atomic ``PM_SCAN_WP_MATCHING + PM_SCAN_CHECK_WPASYNC``) the pages atomically. The (``PM_SCAN_WP_MATCHING``) is used to WP the matched pages. The (``PM_SCAN_CHECK_WPASYNC``) aborts the operation if non-Async-Write-Protected pages are found. Get is automatically performed if output buffer is specified.
This IOCTL can be extended to get information about more PTE bits. The entire address range passed by user [start, end) is scanned until either the user provided buffer is full or max_pages have been found.
Reviewed-by: Andrei Vagin avagin@gmail.com Reviewed-by: Michał Mirosław mirq-linux@rere.qmqm.pl Signed-off-by: Michał Mirosław mirq-linux@rere.qmqm.pl Signed-off-by: Muhammad Usama Anjum usama.anjum@collabora.com
Changes in v28:
- Fix walk_end one last time after doing through testing
Changes in v27:
- Add PAGE_IS_HUGE
- Iterate until temporary buffer is full to do less iterations
- Don't check if PAGE_IS_FILE if no mask needs it as it is very expensive to check per pte
- bring is_interesting_page() outside pagemap_scan_output() to remove the horrible return value check
- Replace memcpy() with direct copy
- rename end_addr to walk_end_addr in pagemap_scan_private
- Abort walk if fatal_signal_pending()
Changes in v26: Changes made by Usama:
- Fix the wrong breaking of loop if page isn't interesting, skip intsead
- Untag the address and save them into struct
- Round off the end address to next page
- Correct the partial hugetlb page handling and returning the error
- Rename PAGE_IS_WPASYNC to PAGE_IS_WPALLOWED
- Return walk ending address in walk_end instead of returning in start as there is potential of replacing the memory tag
Changes by Michał:
- the API:
a. return ranges as {begin, end} instead of {begin + len}; b. rename match "flags" to 'page categories' everywhere - this makes it easier to differentiate the ioctl()s categorisation of pages from struct page flags; c. change {required + excluded} to {inverted + required}. This was rejected before, but I'd like to illustrate the difference. Old interface can be translated to the new by: categories_inverted = excluded_mask categories_mask = required_mask | excluded_mask categories_anyof_mask = anyof_mask The new way allows filtering by: A & (B | !C) categories_inverted = C categories_mask = A categories_anyof_mask = B | C e. allow no-op calls 2. the implementation: a. gather the page-categorising and write-protecting code in one place; b. optimization: add whole-vma skipping for WP usecase; c. extracted output limiting code to pagemap_scan_output(); d. extracted range coalescing to pagemap_scan_push_range(); e. extracted THP entry handling to pagemap_scan_thp_entry(); f. added a shortcut for non-WP hugetlb scan; avoids conditional locking; g. extracted scan buffer handling code out of do_pagemap_scan(); h. rework output code to always try to write pending ranges; if EFAULT is generated it always overwrites the original error code; (the case of SIGKILL is needlessly trying to write the output now, but this should be rare case and ignoring it makes the code not needing a goto) 3.Change no-GET operation condition from `arg.return_mask == 0` to `arg.vec == NULL`. This will allow issuing the ioctl with return_mask == 0 to gather matching ranges when the exact category is not interesting. (Anticipated for CRIU scanning a large sparse anonymous mapping).
Changes in v25:
- Do proper filtering on hole as well (hole got missed earlier)
Changes in v24:
- Place WP markers in case of hole as well
Changes in v23:
- Set vec_buf_index to 0 only when vec_buf_index is set
- Return -EFAULT instead of -EINVAL if vec is NULL
- Correctly return the walk ending address to the page granularity
Changes in v22:
- Interface change to return walk ending address to user:
- Replace [start start + len) with [start, end)
- Return the ending address of the address walk in start
Changes in v21:
- Abort walk instead of returning error if WP is to be performed on partial hugetlb
- Changed the data types of some variables in pagemap_scan_private to long
Changes in v20:
- Correct PAGE_IS_FILE and add PAGE_IS_PFNZERO
Changes in v19:
- Interface changes such as renaming, return mask and WP can be used with any flags specified in masks
- Internal code changes
Changes in v18:
- Rebased on top of next-20230613
- ptep_get() updates
- remove pmd_trans_unstable() and add ACTION_AGAIN
- Review updates (Micheal)
Changes in v17:
- Rebased on next-20230606
- Made make_uffd_wp_*_pte() better and minor changes
Changes in v16:
- Fixed a corner case where kernel writes beyond user buffer by one element
- Bring back exclusive PM_SCAN_OP_WP
- Cosmetic changes
Changes in v15:
- Build fix:
- Use generic tlb flush function in pagemap_scan_pmd_entry() instead of using x86 specific flush function in do_pagemap_scan()
- Remove #ifdef from pagemap_scan_hugetlb_entry()
- Use mm instead of undefined vma->vm_mm
Changes in v14:
- Fix build error caused by #ifdef added at last minute in some configs
Changes in v13:
- Review updates
- mmap_read_lock_killable() instead of mmap_read_lock()
- Replace uffd_wp_range() with helpers which increases performance drastically for OP_WP operations by reducing the number of tlb flushing etc
- Add MMU_NOTIFY_PROTECTION_VMA notification for the memory range
Changes in v12:
- Add hugetlb support to cover all memory types
- Merge "userfaultfd: Define dummy uffd_wp_range()" with this patch
- Review updates to the code
Changes in v11:
- Find written pages in a better way
- Fix a corner case (thanks Paul)
- Improve the code/comments
- remove ENGAGE_WP + ! GET operation
- shorten the commit message in favour of moving documentation to pagemap.rst
Changes in v10:
- move changes in tools/include/uapi/linux/fs.h to separate patch
- update commit message
Change in v8:
- Correct is_pte_uffd_wp()
- Improve readability and error checks
- Remove some un-needed code
Changes in v7:
- Rebase on top of latest next
- Fix some corner cases
- Base soft-dirty on the uffd wp async
- Update the terminologies
- Optimize the memory usage inside the ioctl
fs/proc/task_mmu.c | 678 ++++++++++++++++++++++++++++++++++++++++ include/linux/hugetlb.h | 1 + include/uapi/linux/fs.h | 59 ++++ mm/hugetlb.c | 2 +- 4 files changed, 739 insertions(+), 1 deletion(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index c1e6531cb02ae..0e219a44e97cd 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -19,6 +19,8 @@ #include <linux/shmem_fs.h> #include <linux/uaccess.h> #include <linux/pkeys.h> +#include <linux/minmax.h> +#include <linux/overflow.h>
#include <asm/elf.h> #include <asm/tlb.h> @@ -1749,11 +1751,687 @@ static int pagemap_release(struct inode *inode, struct file *file) return 0; }
+#define PM_SCAN_CATEGORIES (PAGE_IS_WPALLOWED | PAGE_IS_WRITTEN | \
PAGE_IS_FILE | PAGE_IS_PRESENT | \PAGE_IS_SWAPPED | PAGE_IS_PFNZERO | \PAGE_IS_HUGE)+#define PM_SCAN_FLAGS (PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC)
+#define MASKS_OF_INTEREST(a) (a.category_inverted | a.category_mask | \
a.category_anyof_mask | a.return_mask)+struct pagemap_scan_private {
struct pm_scan_arg arg;unsigned long masks_of_interest, cur_vma_category;struct page_region *vec_buf, cur_buf;
I think we can remove cur_buf. Imho, it makes code a bit more readable. Here is a quick poc patch: https://gist.github.com/avagin/2e465e7c362c515ec84d72a201a28de4
unsigned long vec_buf_len, vec_buf_index, found_pages, walk_end_addr;struct page_region __user *vec_out;+};
...
+#ifdef CONFIG_HUGETLB_PAGE +static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
unsigned long start, unsigned long end,struct mm_walk *walk)+{
struct pagemap_scan_private *p = walk->private;struct vm_area_struct *vma = walk->vma;unsigned long categories;spinlock_t *ptl;int ret = 0;pte_t pte;if (~p->arg.flags & PM_SCAN_WP_MATCHING) {/* Go the short route when not write-protecting pages. */pte = huge_ptep_get(ptep);categories = p->cur_vma_category | pagemap_hugetlb_category(pte);if (!pagemap_scan_is_interesting_page(categories, p))return 0;return pagemap_scan_output(categories, p, start, &end);}i_mmap_lock_write(vma->vm_file->f_mapping);ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep);pte = huge_ptep_get(ptep);categories = p->cur_vma_category | pagemap_hugetlb_category(pte);if (!pagemap_scan_is_interesting_page(categories, p))goto out_unlock;ret = pagemap_scan_output(categories, p, start, &end);if (start == end)goto out_unlock;if (~categories & PAGE_IS_WRITTEN)goto out_unlock;if (end != start + HPAGE_SIZE) {/* Partial HugeTLB page WP isn't possible. */pagemap_scan_backout_range(p, start, end, start);ret = -EINVAL;
Will this error be returned from ioctl? If the answer is yet, it looks wrong to me.
goto out_unlock;}make_uffd_wp_huge_pte(vma, start, ptep, pte);flush_hugetlb_tlb_range(vma, start, end);+out_unlock:
spin_unlock(ptl);i_mmap_unlock_write(vma->vm_file->f_mapping);return ret;+}
....
+static int pagemap_scan_get_args(struct pm_scan_arg *arg,
unsigned long uarg)+{
if (copy_from_user(arg, (void __user *)uarg, sizeof(*arg)))return -EFAULT;if (arg->size != sizeof(struct pm_scan_arg))return -EINVAL;/* Validate requested features */if (arg->flags & ~PM_SCAN_FLAGS)return -EINVAL;if ((arg->category_inverted | arg->category_mask |arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES)return -EINVAL;arg->start = untagged_addr((unsigned long)arg->start);arg->end = untagged_addr((unsigned long)arg->end);arg->vec = untagged_addr((unsigned long)arg->vec);/* Validate memory pointers */if (!IS_ALIGNED(arg->start, PAGE_SIZE))return -EINVAL;if (!access_ok((void __user *)arg->start, arg->end - arg->start))return -EFAULT;if (!arg->vec && arg->vec_len)return -EFAULT;
It looks more like EINVAL.
if (arg->vec && !access_ok((void __user *)arg->vec,arg->vec_len * sizeof(struct page_region)))return -EFAULT;/* Fixup default values */arg->end = ALIGN(arg->end, PAGE_SIZE);if (!arg->max_pages)arg->max_pages = ULONG_MAX;return 0;+}
Thanks, Andrei
On 8/11/23 12:07 AM, Andrei Vagin wrote:
On Tue, Aug 8, 2023 at 11:16 PM Muhammad Usama Anjum usama.anjum@collabora.com wrote:
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear the info about page table entries. The following operations are supported in this ioctl:
- Get the information if the pages have Async Write-Protection enabled (``PAGE_IS_WPALLOWED``), have been written to (``PAGE_IS_WRITTEN``), file mapped (``PAGE_IS_FILE``), present (``PAGE_IS_PRESENT``), swapped (``PAGE_IS_SWAPPED``) or page has pfn zero (``PAGE_IS_PFNZERO``).
- Find pages which have been written to and/or write protect (atomic ``PM_SCAN_WP_MATCHING + PM_SCAN_CHECK_WPASYNC``) the pages atomically. The (``PM_SCAN_WP_MATCHING``) is used to WP the matched pages. The (``PM_SCAN_CHECK_WPASYNC``) aborts the operation if non-Async-Write-Protected pages are found. Get is automatically performed if output buffer is specified.
This IOCTL can be extended to get information about more PTE bits. The entire address range passed by user [start, end) is scanned until either the user provided buffer is full or max_pages have been found.
Reviewed-by: Andrei Vagin avagin@gmail.com Reviewed-by: Michał Mirosław mirq-linux@rere.qmqm.pl Signed-off-by: Michał Mirosław mirq-linux@rere.qmqm.pl Signed-off-by: Muhammad Usama Anjum usama.anjum@collabora.com
Changes in v28:
- Fix walk_end one last time after doing through testing
Changes in v27:
- Add PAGE_IS_HUGE
- Iterate until temporary buffer is full to do less iterations
- Don't check if PAGE_IS_FILE if no mask needs it as it is very expensive to check per pte
- bring is_interesting_page() outside pagemap_scan_output() to remove the horrible return value check
- Replace memcpy() with direct copy
- rename end_addr to walk_end_addr in pagemap_scan_private
- Abort walk if fatal_signal_pending()
Changes in v26: Changes made by Usama:
- Fix the wrong breaking of loop if page isn't interesting, skip intsead
- Untag the address and save them into struct
- Round off the end address to next page
- Correct the partial hugetlb page handling and returning the error
- Rename PAGE_IS_WPASYNC to PAGE_IS_WPALLOWED
- Return walk ending address in walk_end instead of returning in start as there is potential of replacing the memory tag
Changes by Michał:
- the API:
a. return ranges as {begin, end} instead of {begin + len}; b. rename match "flags" to 'page categories' everywhere - this makes it easier to differentiate the ioctl()s categorisation of pages from struct page flags; c. change {required + excluded} to {inverted + required}. This was rejected before, but I'd like to illustrate the difference. Old interface can be translated to the new by: categories_inverted = excluded_mask categories_mask = required_mask | excluded_mask categories_anyof_mask = anyof_mask The new way allows filtering by: A & (B | !C) categories_inverted = C categories_mask = A categories_anyof_mask = B | C e. allow no-op calls 2. the implementation: a. gather the page-categorising and write-protecting code in one place; b. optimization: add whole-vma skipping for WP usecase; c. extracted output limiting code to pagemap_scan_output(); d. extracted range coalescing to pagemap_scan_push_range(); e. extracted THP entry handling to pagemap_scan_thp_entry(); f. added a shortcut for non-WP hugetlb scan; avoids conditional locking; g. extracted scan buffer handling code out of do_pagemap_scan(); h. rework output code to always try to write pending ranges; if EFAULT is generated it always overwrites the original error code; (the case of SIGKILL is needlessly trying to write the output now, but this should be rare case and ignoring it makes the code not needing a goto) 3.Change no-GET operation condition from `arg.return_mask == 0` to `arg.vec == NULL`. This will allow issuing the ioctl with return_mask == 0 to gather matching ranges when the exact category is not interesting. (Anticipated for CRIU scanning a large sparse anonymous mapping).
Changes in v25:
- Do proper filtering on hole as well (hole got missed earlier)
Changes in v24:
- Place WP markers in case of hole as well
Changes in v23:
- Set vec_buf_index to 0 only when vec_buf_index is set
- Return -EFAULT instead of -EINVAL if vec is NULL
- Correctly return the walk ending address to the page granularity
Changes in v22:
- Interface change to return walk ending address to user:
- Replace [start start + len) with [start, end)
- Return the ending address of the address walk in start
Changes in v21:
- Abort walk instead of returning error if WP is to be performed on partial hugetlb
- Changed the data types of some variables in pagemap_scan_private to long
Changes in v20:
- Correct PAGE_IS_FILE and add PAGE_IS_PFNZERO
Changes in v19:
- Interface changes such as renaming, return mask and WP can be used with any flags specified in masks
- Internal code changes
Changes in v18:
- Rebased on top of next-20230613
- ptep_get() updates
- remove pmd_trans_unstable() and add ACTION_AGAIN
- Review updates (Micheal)
Changes in v17:
- Rebased on next-20230606
- Made make_uffd_wp_*_pte() better and minor changes
Changes in v16:
- Fixed a corner case where kernel writes beyond user buffer by one element
- Bring back exclusive PM_SCAN_OP_WP
- Cosmetic changes
Changes in v15:
- Build fix:
- Use generic tlb flush function in pagemap_scan_pmd_entry() instead of using x86 specific flush function in do_pagemap_scan()
- Remove #ifdef from pagemap_scan_hugetlb_entry()
- Use mm instead of undefined vma->vm_mm
Changes in v14:
- Fix build error caused by #ifdef added at last minute in some configs
Changes in v13:
- Review updates
- mmap_read_lock_killable() instead of mmap_read_lock()
- Replace uffd_wp_range() with helpers which increases performance drastically for OP_WP operations by reducing the number of tlb flushing etc
- Add MMU_NOTIFY_PROTECTION_VMA notification for the memory range
Changes in v12:
- Add hugetlb support to cover all memory types
- Merge "userfaultfd: Define dummy uffd_wp_range()" with this patch
- Review updates to the code
Changes in v11:
- Find written pages in a better way
- Fix a corner case (thanks Paul)
- Improve the code/comments
- remove ENGAGE_WP + ! GET operation
- shorten the commit message in favour of moving documentation to pagemap.rst
Changes in v10:
- move changes in tools/include/uapi/linux/fs.h to separate patch
- update commit message
Change in v8:
- Correct is_pte_uffd_wp()
- Improve readability and error checks
- Remove some un-needed code
Changes in v7:
- Rebase on top of latest next
- Fix some corner cases
- Base soft-dirty on the uffd wp async
- Update the terminologies
- Optimize the memory usage inside the ioctl
fs/proc/task_mmu.c | 678 ++++++++++++++++++++++++++++++++++++++++ include/linux/hugetlb.h | 1 + include/uapi/linux/fs.h | 59 ++++ mm/hugetlb.c | 2 +- 4 files changed, 739 insertions(+), 1 deletion(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index c1e6531cb02ae..0e219a44e97cd 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -19,6 +19,8 @@ #include <linux/shmem_fs.h> #include <linux/uaccess.h> #include <linux/pkeys.h> +#include <linux/minmax.h> +#include <linux/overflow.h>
#include <asm/elf.h> #include <asm/tlb.h> @@ -1749,11 +1751,687 @@ static int pagemap_release(struct inode *inode, struct file *file) return 0; }
+#define PM_SCAN_CATEGORIES (PAGE_IS_WPALLOWED | PAGE_IS_WRITTEN | \
PAGE_IS_FILE | PAGE_IS_PRESENT | \PAGE_IS_SWAPPED | PAGE_IS_PFNZERO | \PAGE_IS_HUGE)+#define PM_SCAN_FLAGS (PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC)
+#define MASKS_OF_INTEREST(a) (a.category_inverted | a.category_mask | \
a.category_anyof_mask | a.return_mask)+struct pagemap_scan_private {
struct pm_scan_arg arg;unsigned long masks_of_interest, cur_vma_category;struct page_region *vec_buf, cur_buf;I think we can remove cur_buf. Imho, it makes code a bit more readable. Here is a quick poc patch: https://gist.github.com/avagin/2e465e7c362c515ec84d72a201a28de4
I thought ohhh how can this be removed initially. But considering that we have moved to walking full range until temporary buffer is full, removing cur_buf is possible. You have proved with your POC as well. Thank you for doing it. I've updated it after testing and simplified it further.
unsigned long vec_buf_len, vec_buf_index, found_pages, walk_end_addr;struct page_region __user *vec_out;+};
...
+#ifdef CONFIG_HUGETLB_PAGE +static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
unsigned long start, unsigned long end,struct mm_walk *walk)+{
struct pagemap_scan_private *p = walk->private;struct vm_area_struct *vma = walk->vma;unsigned long categories;spinlock_t *ptl;int ret = 0;pte_t pte;if (~p->arg.flags & PM_SCAN_WP_MATCHING) {/* Go the short route when not write-protecting pages. */pte = huge_ptep_get(ptep);categories = p->cur_vma_category | pagemap_hugetlb_category(pte);if (!pagemap_scan_is_interesting_page(categories, p))return 0;return pagemap_scan_output(categories, p, start, &end);}i_mmap_lock_write(vma->vm_file->f_mapping);ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep);pte = huge_ptep_get(ptep);categories = p->cur_vma_category | pagemap_hugetlb_category(pte);if (!pagemap_scan_is_interesting_page(categories, p))goto out_unlock;ret = pagemap_scan_output(categories, p, start, &end);if (start == end)goto out_unlock;if (~categories & PAGE_IS_WRITTEN)goto out_unlock;if (end != start + HPAGE_SIZE) {/* Partial HugeTLB page WP isn't possible. */pagemap_scan_backout_range(p, start, end, start);ret = -EINVAL;Will this error be returned from ioctl? If the answer is yet, it looks wrong to me.
Sorry, we missed it in previous revisions. I'll return 0 here and walk_end will indicate to user that we have not walked the entire range.
goto out_unlock;}make_uffd_wp_huge_pte(vma, start, ptep, pte);flush_hugetlb_tlb_range(vma, start, end);+out_unlock:
spin_unlock(ptl);i_mmap_unlock_write(vma->vm_file->f_mapping);return ret;+}
....
+static int pagemap_scan_get_args(struct pm_scan_arg *arg,
unsigned long uarg)+{
if (copy_from_user(arg, (void __user *)uarg, sizeof(*arg)))return -EFAULT;if (arg->size != sizeof(struct pm_scan_arg))return -EINVAL;/* Validate requested features */if (arg->flags & ~PM_SCAN_FLAGS)return -EINVAL;if ((arg->category_inverted | arg->category_mask |arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES)return -EINVAL;arg->start = untagged_addr((unsigned long)arg->start);arg->end = untagged_addr((unsigned long)arg->end);arg->vec = untagged_addr((unsigned long)arg->vec);/* Validate memory pointers */if (!IS_ALIGNED(arg->start, PAGE_SIZE))return -EINVAL;if (!access_ok((void __user *)arg->start, arg->end - arg->start))return -EFAULT;if (!arg->vec && arg->vec_len)return -EFAULT;It looks more like EINVAL.
Updated for next revision.
if (arg->vec && !access_ok((void __user *)arg->vec,arg->vec_len * sizeof(struct page_region)))return -EFAULT;/* Fixup default values */arg->end = ALIGN(arg->end, PAGE_SIZE);if (!arg->max_pages)arg->max_pages = ULONG_MAX;return 0;+}
Thanks, Andrei
Adding fast code paths to handle specifically only get and/or clear operation of PAGE_IS_WRITTEN, increases its performance by 0-35%. The results of some test cases are given below:
Test-case-1 t1 = (Get + WP) time t2 = WP time t1 t2 Without this patch: 140-170mcs 90-115mcs With this patch: 110mcs 80mcs Worst case diff: 35% faster 30% faster
Test-case-2 t3 = atomic Get and WP t3 Without this patch: 120-140mcs With this patch: 100-110mcs Worst case diff: 21% faster
Signed-off-by: Muhammad Usama Anjum usama.anjum@collabora.com --- The test to measure the performance can be found: https://is.gd/FtSKcD 8 8192 3 1 0 and 8 8192 3 1 1 arguments have been used to produce the above mentioned results. --- fs/proc/task_mmu.c | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 0e219a44e97cd..e336ec0151185 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -2107,6 +2107,43 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, return 0; }
+ if (!p->vec_out) { + /* Fast path for performing exclusive WP */ + for (addr = start; addr != end; pte++, addr += PAGE_SIZE) { + if (pte_uffd_wp(ptep_get(pte))) + continue; + make_uffd_wp_pte(vma, addr, pte); + if (!flush) { + start = addr; + flush = true; + } + } + goto flush_and_return; + } + + if (!p->arg.category_anyof_mask && !p->arg.category_inverted && + p->arg.category_mask == PAGE_IS_WRITTEN && + p->arg.return_mask == PAGE_IS_WRITTEN) { + for (addr = start; addr < end; pte++, addr += PAGE_SIZE) { + unsigned long next = addr + PAGE_SIZE; + + if (pte_uffd_wp(ptep_get(pte))) + continue; + ret = pagemap_scan_output(p->cur_vma_category | PAGE_IS_WRITTEN, + p, addr, &next); + if (next == addr) + break; + if (~p->arg.flags & PM_SCAN_WP_MATCHING) + continue; + make_uffd_wp_pte(vma, addr, pte); + if (!flush) { + start = addr; + flush = true; + } + } + goto flush_and_return; + } + for (addr = start; addr != end; pte++, addr += PAGE_SIZE) { unsigned long categories = p->cur_vma_category | pagemap_page_category(p, vma, addr, ptep_get(pte)); @@ -2131,6 +2168,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, } }
+flush_and_return: if (flush) flush_tlb_range(vma, start, addr);
New IOCTL and macros has been added in the kernel sources. Update the tools header file as well.
Signed-off-by: Muhammad Usama Anjum usama.anjum@collabora.com --- Changes in v27: - Comment updates in fs.h
Changes in v26: - Update according to tools/include/uapi/linux/fs.h
Changes in v21: - Update tools/include/uapi/linux/fs.h
Changes in v19: - Update fs.h according to precious patch --- tools/include/uapi/linux/fs.h | 59 +++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+)
diff --git a/tools/include/uapi/linux/fs.h b/tools/include/uapi/linux/fs.h index b7b56871029c5..1c9d38af1015e 100644 --- a/tools/include/uapi/linux/fs.h +++ b/tools/include/uapi/linux/fs.h @@ -305,4 +305,63 @@ typedef int __bitwise __kernel_rwf_t; #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ RWF_APPEND)
+/* Pagemap ioctl */ +#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg) + +/* Bits are set in flags of the page_region and masks in pm_scan_args */ +#define PAGE_IS_WPALLOWED (1 << 0) +#define PAGE_IS_WRITTEN (1 << 1) +#define PAGE_IS_FILE (1 << 2) +#define PAGE_IS_PRESENT (1 << 3) +#define PAGE_IS_SWAPPED (1 << 4) +#define PAGE_IS_PFNZERO (1 << 5) +#define PAGE_IS_HUGE (1 << 6) + +/* + * struct page_region - Page region with flags + * @start: Start of the region + * @end: End of the region (exclusive) + * @categories: PAGE_IS_* category bitmask for the region + */ +struct page_region { + __u64 start; + __u64 end; + __u64 categories; +}; + +/* Flags for PAGEMAP_SCAN ioctl */ +#define PM_SCAN_WP_MATCHING (1 << 0) /* Write protect the pages matched. */ +#define PM_SCAN_CHECK_WPASYNC (1 << 1) /* Abort the scan when a non-WP-enabled page is found. */ + +/* + * struct pm_scan_arg - Pagemap ioctl argument + * @size: Size of the structure + * @flags: Flags for the IOCTL + * @start: Starting address of the region + * @end: Ending address of the region + * @walk_end Address where the scan stopped (written by kernel). + * walk_end == end informs that the scan completed on entire range. + * @vec: Address of page_region struct array for output + * @vec_len: Length of the page_region struct array + * @max_pages: Optional limit for number of returned pages (0 = disabled) + * @category_inverted: PAGE_IS_* categories which values match if 0 instead of 1 + * @category_mask: Skip pages for which any category doesn't match + * @category_anyof_mask: Skip pages for which no category matches + * @return_mask: PAGE_IS_* categories that are to be reported in `page_region`s returned + */ +struct pm_scan_arg { + __u64 size; + __u64 flags; + __u64 start; + __u64 end; + __u64 walk_end; + __u64 vec; + __u64 vec_len; + __u64 max_pages; + __u64 category_inverted; + __u64 category_mask; + __u64 category_anyof_mask; + __u64 return_mask; +}; + #endif /* _UAPI_LINUX_FS_H */
Add some explanation and method to use write-protection and written-to on memory range.
Signed-off-by: Muhammad Usama Anjum usama.anjum@collabora.com --- Changes in v26: - Update documentation according to latest revision
Changes in v21: - Update according to changed interface
Changes in v20: - Add new flag in documentation
Changes in v19: - Improved documentatiom with punctuation marks
Changes in v16: - Update the documentation
Changes in v11: - Add more documentation --- Documentation/admin-guide/mm/pagemap.rst | 64 ++++++++++++++++++++++++ 1 file changed, 64 insertions(+)
diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst index c8f380271cad8..ea18e4575902a 100644 --- a/Documentation/admin-guide/mm/pagemap.rst +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -227,3 +227,67 @@ Before Linux 3.11 pagemap bits 55-60 were used for "page-shift" (which is always 12 at most architectures). Since Linux 3.11 their meaning changes after first clear of soft-dirty bits. Since Linux 4.2 they are used for flags unconditionally. + +Pagemap Scan IOCTL +================== + +The ``PAGEMAP_SCAN`` IOCTL on the pagemap file can be used to get or optionally +clear the info about page table entries. The following operations are supported +in this IOCTL: +- Get the information if the pages have Async Write-Protection enabled + (``PAGE_IS_WPALLOWED``), have been written to (``PAGE_IS_WRITTEN``), file mapped + (``PAGE_IS_FILE``), present (``PAGE_IS_PRESENT``), swapped (``PAGE_IS_SWAPPED``) + or page has pfn zero (``PAGE_IS_PFNZERO``). +- Find pages which have been written to and/or write protect + (atomic ``PM_SCAN_WP_MATCHING + PM_SCAN_CHECK_WPASYNC``) the pages atomically. + The (``PM_SCAN_WP_MATCHING``) is used to WP the matched pages. The + (``PM_SCAN_CHECK_WPASYNC``) aborts the operation if non-Async-Write-Protected + pages are found. + +The ``struct pm_scan_arg`` is used as the argument of the IOCTL. + 1. The size of the ``struct pm_scan_arg`` must be specified in the ``size`` + field. This field will be helpful in recognizing the structure if extensions + are done later. + 2. The flags can be specified in the ``flags`` field. The ``PM_SCAN_WP_MATCHING`` + and ``PM_SCAN_CHECK_WPASYNC`` are the only added flags at this time. The get + operation is optionally performed depending upon if the output buffer is + provided or not. + 3. The range is specified through ``start`` and ``end``. + 4. The output buffer of ``struct page_region`` array and size is specified in + ``vec`` and ``vec_len``. + 5. The optional maximum requested pages are specified in the ``max_pages``. + 6. The masks are specified in ``category_mask``, ``category_anyof_mask``, + ``category_inverted`` and ``return_mask``. + 1. To find if ``PAGE_IS_WRITTEN`` flag is set for pages which have + ``PAGE_IS_FILE`` set and ``PAGE_IS_SWAPPED`` unset, ``category_mask`` + is set to ``PAGE_IS_FILE | PAGE_IS_SWAPPED``, ``category_inverted`` is + set to ``PAGE_IS_SWAPPED`` and ``return_mask`` is set to ``PAGE_IS_WRITTEN``. + The output buffer in ``vec`` and length must be specified in ``vec_len``. + 2. To find pages which have either ``PAGE_IS_FILE`` or ``PAGE_IS_SWAPPED`` + set, ``category_anyof_mask`` is set to ``PAGE_IS_FILE | PAGE_IS_SWAPPED``. + 3. To find written pages and engage write protect, ``PAGE_IS_WRITTEN`` is + specified in ``category_mask`` and ``return_mask``. In addition to + specifying the output buffer in ``vec`` and length in ``vec_len``, the + ``PM_SCAN_WP_MATCHING`` is specified in ``flags`` to perform write protect + on the range as well. + +The ``PAGE_IS_WRITTEN`` flag can be considered as the better and correct +alternative of soft-dirty flag. It doesn't get affected by housekeeping chores +(VMA merging) of the kernel and hence the user can find the true soft-dirty pages +only. This IOCTL adds the atomic way to find which pages have been written and +write protect those pages again. This kind of operation is needed to efficiently +find out which pages have changed in the memory. + +To get information about which pages have been written to or optionally write +protect the pages, following must be performed first in order: + 1. The userfaultfd file descriptor is created with ``userfaultfd`` syscall. + 2. The ``UFFD_FEATURE_WP_UNPOPULATED`` and ``UFFD_FEATURE_WP_ASYNC`` features + are set by ``UFFDIO_API`` IOCTL. + 3. The memory range is registered with ``UFFDIO_REGISTER_MODE_WP`` mode + through ``UFFDIO_REGISTER`` IOCTL. + 4. Then any part of the registered memory or the whole memory region must + be write protected using ``PAGEMAP_SCAN`` IOCTL with flag ``PM_SCAN_OP_WP`` + or the ``UFFDIO_WRITEPROTECT`` IOCTL can be used. Both of these perform the + same operation. The former is better in terms of performance. + 5. Now the ``PAGEMAP_SCAN`` IOCTL can be used to either just find pages which + have been written to and/or optionally write protect the pages as well.
On Wed, 9 Aug 2023 at 08:16, Muhammad Usama Anjum usama.anjum@collabora.com wrote:
Add some explanation and method to use write-protection and written-to on memory range.
[...]
--- a/Documentation/admin-guide/mm/pagemap.rst +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -227,3 +227,67 @@ Before Linux 3.11 pagemap bits 55-60 were used for "page-shift" (which is always 12 at most architectures). Since Linux 3.11 their meaning changes after first clear of soft-dirty bits. Since Linux 4.2 they are used for flags unconditionally.
+Pagemap Scan IOCTL +==================
+The ``PAGEMAP_SCAN`` IOCTL on the pagemap file can be used to get or optionally +clear the info about page table entries. The following operations are supported +in this IOCTL: +- Get the information if the pages have Async Write-Protection enabled
- (``PAGE_IS_WPALLOWED``), have been written to (``PAGE_IS_WRITTEN``), file mapped
- (``PAGE_IS_FILE``), present (``PAGE_IS_PRESENT``), swapped (``PAGE_IS_SWAPPED``)
- or page has pfn zero (``PAGE_IS_PFNZERO``).
A recent addition -- PAGE_IS_HUGE -- is missing.
BTW, it could be easier to understand if the page categories were separated from the operation description and listed so that each has its own line and maybe a longer description where needed.
+- Find pages which have been written to and/or write protect
- (atomic ``PM_SCAN_WP_MATCHING + PM_SCAN_CHECK_WPASYNC``) the pages atomically.
- The (``PM_SCAN_WP_MATCHING``) is used to WP the matched pages. The
- (``PM_SCAN_CHECK_WPASYNC``) aborts the operation if non-Async-Write-Protected
- pages are found.
The operation the IOCTL does now is: "scan the process page tables and report memory ranges matching provided criteria '. Flags extend the operation: "PM_SCAN_WP_MATCHING write protects the memory reported" (it does it atomically, but this is just an optimization, isn't it? A process could gather the ranges, WP them, and then copy.) "PM_SCAN_CHECK_WPASYNC" aborts the scan early if a non-WP-able matching page is found.
+The ``struct pm_scan_arg`` is used as the argument of the IOCTL.
- The size of the ``struct pm_scan_arg`` must be specified in the ``size``
- field. This field will be helpful in recognizing the structure if extensions
- are done later.
- The flags can be specified in the ``flags`` field. The ``PM_SCAN_WP_MATCHING``
- and ``PM_SCAN_CHECK_WPASYNC`` are the only added flags at this time. The get
- operation is optionally performed depending upon if the output buffer is
- provided or not.
- The range is specified through ``start`` and ``end``.
- The output buffer of ``struct page_region`` array and size is specified in
- ``vec`` and ``vec_len``.
- The optional maximum requested pages are specified in the ``max_pages``.
- The masks are specified in ``category_mask``, ``category_anyof_mask``,
- ``category_inverted`` and ``return_mask``.
- To find if ``PAGE_IS_WRITTEN`` flag is set for pages which have
``PAGE_IS_FILE`` set and ``PAGE_IS_SWAPPED`` unset, ``category_mask``is set to ``PAGE_IS_FILE | PAGE_IS_SWAPPED``, ``category_inverted`` isset to ``PAGE_IS_SWAPPED`` and ``return_mask`` is set to ``PAGE_IS_WRITTEN``.The output buffer in ``vec`` and length must be specified in ``vec_len``.
- To find pages which have either ``PAGE_IS_FILE`` or ``PAGE_IS_SWAPPED``
set, ``category_anyof_mask`` is set to ``PAGE_IS_FILE | PAGE_IS_SWAPPED``.
- To find written pages and engage write protect, ``PAGE_IS_WRITTEN`` is
specified in ``category_mask`` and ``return_mask``. In addition tospecifying the output buffer in ``vec`` and length in ``vec_len``, the``PM_SCAN_WP_MATCHING`` is specified in ``flags`` to perform write protecton the range as well.
Could this be rewritten as examples? E.g.:
Finding dirty file-backed pages:
struct pm_scan_arg arg = { .size = sizeof(arg), .flags = 0, ... .category_mask = ..., .return_mask = ... }; ssize_t n = ioctl(..., &arg);
Find dirty pages and write protect them in the same call:
arg = { ... }; do { ... ioctl(...) } while(...);
(The code snippets heavily commented.)
+The ``PAGE_IS_WRITTEN`` flag can be considered as the better and correct
"as a better-performing alternative"
+alternative of soft-dirty flag. It doesn't get affected by housekeeping chores +(VMA merging) of the kernel and hence the user can find the true soft-dirty pages +only.
This is still an optimization, e.g. in THP case there might be too many pages reported?
- This IOCTL adds the atomic way to find which pages have been written and
+write protect those pages again. This kind of operation is needed to efficiently +find out which pages have changed in the memory.
This repeats the description of PM_SCAN_WP_MATCHING -- I suggest removing this part.
+To get information about which pages have been written to or optionally write +protect the pages, following must be performed first in order:
"PAGE_IS_WRITTEN" category is used with uffd write protect-enabled ranges to implement memory dirty tracking in userspace:
- The userfaultfd file descriptor is created with ``userfaultfd`` syscall.
- The ``UFFD_FEATURE_WP_UNPOPULATED`` and ``UFFD_FEATURE_WP_ASYNC`` features
- are set by ``UFFDIO_API`` IOCTL.
- The memory range is registered with ``UFFDIO_REGISTER_MODE_WP`` mode
- through ``UFFDIO_REGISTER`` IOCTL.
- Then any part of the registered memory or the whole memory region must
- be write protected using ``PAGEMAP_SCAN`` IOCTL with flag ``PM_SCAN_OP_WP``
- or the ``UFFDIO_WRITEPROTECT`` IOCTL can be used. Both of these perform the
- same operation. The former is better in terms of performance.
I guess that the UFFD performance could be fixed? But this part refers to the old PM_SCAN_OP_WP, so an updated example is needed.
- Now the ``PAGEMAP_SCAN`` IOCTL can be used to either just find pages which
- have been written to and/or optionally write protect the pages as well.
"find the pages written to since they were last write protected", but this sounds contradicting: we look for pages that were WP but written anyway. (IOW: marking write-protected is an implementation detail - the ioctl is to find pages that changed since they were last marked.) Maybe we should call the operation "marking CLEAN" or alike?
Best Regards Michał Mirosław
On 8/11/23 12:26 AM, Michał Mirosław wrote:
On Wed, 9 Aug 2023 at 08:16, Muhammad Usama Anjum usama.anjum@collabora.com wrote:
Add some explanation and method to use write-protection and written-to on memory range.
[...]
--- a/Documentation/admin-guide/mm/pagemap.rst +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -227,3 +227,67 @@ Before Linux 3.11 pagemap bits 55-60 were used for "page-shift" (which is always 12 at most architectures). Since Linux 3.11 their meaning changes after first clear of soft-dirty bits. Since Linux 4.2 they are used for flags unconditionally.
+Pagemap Scan IOCTL +==================
+The ``PAGEMAP_SCAN`` IOCTL on the pagemap file can be used to get or optionally +clear the info about page table entries. The following operations are supported +in this IOCTL: +- Get the information if the pages have Async Write-Protection enabled
- (``PAGE_IS_WPALLOWED``), have been written to (``PAGE_IS_WRITTEN``), file mapped
- (``PAGE_IS_FILE``), present (``PAGE_IS_PRESENT``), swapped (``PAGE_IS_SWAPPED``)
- or page has pfn zero (``PAGE_IS_PFNZERO``).
A recent addition -- PAGE_IS_HUGE -- is missing.
BTW, it could be easier to understand if the page categories were separated from the operation description and listed so that each has its own line and maybe a longer description where needed.
I've made 90% of changes you have asked in documentation.
+- Find pages which have been written to and/or write protect
- (atomic ``PM_SCAN_WP_MATCHING + PM_SCAN_CHECK_WPASYNC``) the pages atomically.
- The (``PM_SCAN_WP_MATCHING``) is used to WP the matched pages. The
- (``PM_SCAN_CHECK_WPASYNC``) aborts the operation if non-Async-Write-Protected
- pages are found.
The operation the IOCTL does now is: "scan the process page tables and report memory ranges matching provided criteria '. Flags extend the operation: "PM_SCAN_WP_MATCHING write protects the memory reported" (it does it atomically, but this is just an optimization, isn't it? A process could gather the ranges, WP them, and then copy.) "PM_SCAN_CHECK_WPASYNC" aborts the scan early if a non-WP-able matching page is found.
+The ``struct pm_scan_arg`` is used as the argument of the IOCTL.
- The size of the ``struct pm_scan_arg`` must be specified in the ``size``
- field. This field will be helpful in recognizing the structure if extensions
- are done later.
- The flags can be specified in the ``flags`` field. The ``PM_SCAN_WP_MATCHING``
- and ``PM_SCAN_CHECK_WPASYNC`` are the only added flags at this time. The get
- operation is optionally performed depending upon if the output buffer is
- provided or not.
- The range is specified through ``start`` and ``end``.
- The output buffer of ``struct page_region`` array and size is specified in
- ``vec`` and ``vec_len``.
- The optional maximum requested pages are specified in the ``max_pages``.
- The masks are specified in ``category_mask``, ``category_anyof_mask``,
- ``category_inverted`` and ``return_mask``.
- To find if ``PAGE_IS_WRITTEN`` flag is set for pages which have
``PAGE_IS_FILE`` set and ``PAGE_IS_SWAPPED`` unset, ``category_mask``is set to ``PAGE_IS_FILE | PAGE_IS_SWAPPED``, ``category_inverted`` isset to ``PAGE_IS_SWAPPED`` and ``return_mask`` is set to ``PAGE_IS_WRITTEN``.The output buffer in ``vec`` and length must be specified in ``vec_len``.
- To find pages which have either ``PAGE_IS_FILE`` or ``PAGE_IS_SWAPPED``
set, ``category_anyof_mask`` is set to ``PAGE_IS_FILE | PAGE_IS_SWAPPED``.
- To find written pages and engage write protect, ``PAGE_IS_WRITTEN`` is
specified in ``category_mask`` and ``return_mask``. In addition tospecifying the output buffer in ``vec`` and length in ``vec_len``, the``PM_SCAN_WP_MATCHING`` is specified in ``flags`` to perform write protecton the range as well.Could this be rewritten as examples? E.g.:
Finding dirty file-backed pages:
struct pm_scan_arg arg = { .size = sizeof(arg), .flags = 0, ... .category_mask = ..., .return_mask = ... }; ssize_t n = ioctl(..., &arg);
Find dirty pages and write protect them in the same call:
arg = { ... }; do { ... ioctl(...) } while(...);
(The code snippets heavily commented.)
+The ``PAGE_IS_WRITTEN`` flag can be considered as the better and correct
"as a better-performing alternative"
+alternative of soft-dirty flag. It doesn't get affected by housekeeping chores +(VMA merging) of the kernel and hence the user can find the true soft-dirty pages +only.
This is still an optimization, e.g. in THP case there might be too many pages reported?
- This IOCTL adds the atomic way to find which pages have been written and
+write protect those pages again. This kind of operation is needed to efficiently +find out which pages have changed in the memory.
This repeats the description of PM_SCAN_WP_MATCHING -- I suggest removing this part.
+To get information about which pages have been written to or optionally write +protect the pages, following must be performed first in order:
"PAGE_IS_WRITTEN" category is used with uffd write protect-enabled ranges to implement memory dirty tracking in userspace:
- The userfaultfd file descriptor is created with ``userfaultfd`` syscall.
- The ``UFFD_FEATURE_WP_UNPOPULATED`` and ``UFFD_FEATURE_WP_ASYNC`` features
- are set by ``UFFDIO_API`` IOCTL.
- The memory range is registered with ``UFFDIO_REGISTER_MODE_WP`` mode
- through ``UFFDIO_REGISTER`` IOCTL.
- Then any part of the registered memory or the whole memory region must
- be write protected using ``PAGEMAP_SCAN`` IOCTL with flag ``PM_SCAN_OP_WP``
- or the ``UFFDIO_WRITEPROTECT`` IOCTL can be used. Both of these perform the
- same operation. The former is better in terms of performance.
I guess that the UFFD performance could be fixed? But this part refers to the old PM_SCAN_OP_WP, so an updated example is needed.
- Now the ``PAGEMAP_SCAN`` IOCTL can be used to either just find pages which
- have been written to and/or optionally write protect the pages as well.
"find the pages written to since they were last write protected", but this sounds contradicting: we look for pages that were WP but written anyway. (IOW: marking write-protected is an implementation detail - the ioctl is to find pages that changed since they were last marked.) Maybe we should call the operation "marking CLEAN" or alike?
Best Regards Michał Mirosław
Add pagemap ioctl tests. Add several different types of tests to judge the correction of the interface.
Signed-off-by: Muhammad Usama Anjum usama.anjum@collabora.com --- Changes in v28: - Add walk_end tests
Changes in v27: - Open the pagemap file with O_RDONLY - Use UFFD_USER_MODE_ONLY flag for userfaultfd() syscall
Changes in v26: - Updates
Changes in v19: - Incorporated interface changes and update tests
Changes in v18: - Rebase on top of 20230613 (Resolve conflict in Makefile) - Add temp files to .gitignore
Changes in v17: - Rebase on top of next-20230525
Changes in v16: - Added yet more tests which is a randomization test case to catch the corner cases - Add reset by exclusive PM_SCAN_OP_WP as well
Changes in v13: - Update tests and rebase Makefile
Changes in v12: - Updates and add more memory type tests
Changes in v11: - Rebase on top of next-20230216 and update tests
Chages in v7: - Add and update all test cases
Changes in v6: - Rename variables
Changes in v4: - Updated all the tests to conform to new IOCTL
Changes in v3: - Add another test to do sanity of flags
Changes in v2: - Update the tests to use the ioctl interface instead of syscall --- TAP version 13 1..108 ok 1 sanity_tests_sd Zero range size is valid ok 2 sanity_tests_sd output buffer must be specified with size ok 3 sanity_tests_sd output buffer can be 0 ok 4 sanity_tests_sd wrong flag specified ok 5 sanity_tests_sd flag has extra bits specified ok 6 sanity_tests_sd no selection mask is specified ok 7 sanity_tests_sd no return mask is specified ok 8 sanity_tests_sd wrong return mask specified ok 9 sanity_tests_sd mixture of correct and wrong flag ok 10 sanity_tests_sd PAGEMAP_BITS_ALL can be specified with PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC ok 11 sanity_tests_sd Clear area with larger vec size ok 12 sanity_tests_sd Repeated pattern of written and non-written pages ok 13 sanity_tests_sd Repeated pattern of written and non-written pages in parts 3 2 2 ok 14 sanity_tests_sd Repeated pattern of written and non-written pages max_pages ok 15 sanity_tests_sd only get 2 written pages and clear them as well ok 16 sanity_tests_sd Two regions ok 17 sanity_tests_sd Smaller max_pages ok 18 Smaller vec ok 19 Walk_end: Same start and end address ok 20 Walk_end: Same start and end with WP ok 21 Walk_end: Same start and end with 0 output buffer ok 22 Walk_end: Big vec ok 23 Walk_end: vec of minimum length ok 24 Walk_end: Max pages specified ok 25 Walk_end: Half max pages ok 26 Walk_end: 1 max page ok 27 Walk_end: max pages ok 28 Walk_end sparse: Big vec ok 29 Walk_end sparse: vec of minimum length ok 30 Walk_end sparse: Max pages specified ok 31 Walk_end sparse: Max pages specified ok 32 Walk_end sparse: Max pages specified ok 33 Walk_endsparse : Half max pages ok 34 Walk_end: 1 max page ok 35 Page testing: all new pages must not be written (dirty) ok 36 Page testing: all pages must be written (dirty) ok 37 Page testing: all pages dirty other than first and the last one ok 38 Page testing: PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC ok 39 Page testing: only middle page dirty ok 40 Page testing: only two middle pages dirty ok 41 Large Page testing: all new pages must not be written (dirty) ok 42 Large Page testing: all pages must be written (dirty) ok 43 Large Page testing: all pages dirty other than first and the last one ok 44 Large Page testing: PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC ok 45 Large Page testing: only middle page dirty ok 46 Large Page testing: only two middle pages dirty ok 47 Huge page testing: all new pages must not be written (dirty) ok 48 Huge page testing: all pages must be written (dirty) ok 49 Huge page testing: all pages dirty other than first and the last one ok 50 Huge page testing: PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC ok 51 Huge page testing: only middle page dirty ok 52 Huge page testing: only two middle pages dirty ok 53 Hugetlb shmem testing: all new pages must not be written (dirty) ok 54 Hugetlb shmem testing: all pages must be written (dirty) ok 55 Hugetlb shmem testing: all pages dirty other than first and the last one ok 56 Hugetlb shmem testing: PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC ok 57 Hugetlb shmem testing: only middle page dirty ok 58 Hugetlb shmem testing: only two middle pages dirty ok 59 Hugetlb mem testing: all new pages must not be written (dirty) ok 60 Hugetlb mem testing: all pages must be written (dirty) ok 61 Hugetlb mem testing: all pages dirty other than first and the last one ok 62 Hugetlb mem testing: PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC ok 63 Hugetlb mem testing: only middle page dirty ok 64 Hugetlb mem testing: only two middle pages dirty ok 65 File memory testing: all new pages must not be written (dirty) ok 66 File memory testing: all pages must be written (dirty) ok 67 File memory testing: all pages dirty other than first and the last one ok 68 File memory testing: PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC ok 69 File memory testing: only middle page dirty ok 70 File memory testing: only two middle pages dirty ok 71 File anonymous memory testing: all new pages must not be written (dirty) ok 72 File anonymous memory testing: all pages must be written (dirty) ok 73 File anonymous memory testing: all pages dirty other than first and the last one ok 74 File anonymous memory testing: PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC ok 75 File anonymous memory testing: only middle page dirty ok 76 File anonymous memory testing: only two middle pages dirty ok 77 hpage_unit_tests all new huge page must not be written (dirty) ok 78 hpage_unit_tests all the huge page must not be written ok 79 hpage_unit_tests all the huge page must be written and clear ok 80 hpage_unit_tests only middle page written ok 81 hpage_unit_tests clear first half of huge page ok 82 hpage_unit_tests clear first half of huge page with limited buffer ok 83 hpage_unit_tests clear second half huge page ok 84 hpage_unit_tests get half huge page ok 85 hpage_unit_tests get half huge page ok 86 Test test_simple ok 87 mprotect_tests Both pages written ok 88 mprotect_tests Both pages are not written (dirty) ok 89 mprotect_tests Both pages written after remap and mprotect ok 90 mprotect_tests Clear and make the pages written ok 91 transact_test count 192 ok 92 transact_test count 0 ok 93 transact_test Extra pages 1 (0.0%), extra thread faults 1. ok 94 sanity_tests WP op can be specified with !PAGE_IS_WRITTEN ok 95 sanity_tests required_mask specified ok 96 sanity_tests anyof_mask specified ok 97 sanity_tests excluded_mask specified ok 98 sanity_tests required_mask and anyof_mask specified ok 99 sanity_tests Get sd and present pages with anyof_mask ok 100 sanity_tests Get all the pages with required_mask ok 101 sanity_tests Get sd and present pages with required_mask and anyof_mask ok 102 sanity_tests Don't get sd pages ok 103 sanity_tests Don't get present pages ok 104 sanity_tests Find written present pages with return mask ok 105 sanity_tests Memory mapped file ok 106 sanity_tests Read/write to memory ok 107 unmapped_region_tests Get status of pages ok 108 userfaultfd_tests all new pages must not be written (dirty) # Totals: pass:108 fail:0 xfail:0 xpass:0 skip:0 error:0 --- tools/testing/selftests/mm/.gitignore | 2 + tools/testing/selftests/mm/Makefile | 3 +- tools/testing/selftests/mm/config | 1 + tools/testing/selftests/mm/pagemap_ioctl.c | 1658 ++++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 4 + 5 files changed, 1667 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/mm/pagemap_ioctl.c
diff --git a/tools/testing/selftests/mm/.gitignore b/tools/testing/selftests/mm/.gitignore index cdc9ce4426b95..cc920c79ff1c3 100644 --- a/tools/testing/selftests/mm/.gitignore +++ b/tools/testing/selftests/mm/.gitignore @@ -18,6 +18,8 @@ mremap_dontunmap mremap_test on-fault-limit transhuge-stress +pagemap_ioctl +*.tmp* protection_keys protection_keys_32 protection_keys_64 diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile index b7fce9073279e..1dac4e8477e48 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -33,7 +33,7 @@ endif MAKEFLAGS += --no-builtin-rules
CFLAGS = -Wall -I $(top_srcdir) $(EXTRA_CFLAGS) $(KHDR_INCLUDES) -LDLIBS = -lrt -lpthread +LDLIBS = -lrt -lpthread -lm
TEST_GEN_PROGS = cow TEST_GEN_PROGS += compaction_test @@ -60,6 +60,7 @@ TEST_GEN_PROGS += mrelease_test TEST_GEN_PROGS += mremap_dontunmap TEST_GEN_PROGS += mremap_test TEST_GEN_PROGS += on-fault-limit +TEST_GEN_PROGS += pagemap_ioctl TEST_GEN_PROGS += thuge-gen TEST_GEN_PROGS += transhuge-stress TEST_GEN_PROGS += uffd-stress diff --git a/tools/testing/selftests/mm/config b/tools/testing/selftests/mm/config index be087c4bc3961..4309916f629e3 100644 --- a/tools/testing/selftests/mm/config +++ b/tools/testing/selftests/mm/config @@ -1,5 +1,6 @@ CONFIG_SYSVIPC=y CONFIG_USERFAULTFD=y +CONFIG_PTE_MARKER_UFFD_WP=y CONFIG_TEST_VMALLOC=m CONFIG_DEVICE_PRIVATE=y CONFIG_TEST_HMM=m diff --git a/tools/testing/selftests/mm/pagemap_ioctl.c b/tools/testing/selftests/mm/pagemap_ioctl.c new file mode 100644 index 0000000000000..9313beddb23d9 --- /dev/null +++ b/tools/testing/selftests/mm/pagemap_ioctl.c @@ -0,0 +1,1658 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include <stdio.h> +#include <fcntl.h> +#include <string.h> +#include <sys/mman.h> +#include <errno.h> +#include <malloc.h> +#include "vm_util.h" +#include "../kselftest.h" +#include <linux/types.h> +#include <linux/memfd.h> +#include <linux/userfaultfd.h> +#include <linux/fs.h> +#include <sys/ioctl.h> +#include <sys/stat.h> +#include <math.h> +#include <asm/unistd.h> +#include <pthread.h> +#include <sys/resource.h> +#include <assert.h> +#include <sys/ipc.h> +#include <sys/shm.h> + +#define PAGEMAP_BITS_ALL (PAGE_IS_WPALLOWED | PAGE_IS_WRITTEN | \ + PAGE_IS_FILE | PAGE_IS_PRESENT | \ + PAGE_IS_SWAPPED | PAGE_IS_PFNZERO | \ + PAGE_IS_HUGE) +#define PAGEMAP_NON_WRITTEN_BITS (PAGE_IS_WPALLOWED | PAGE_IS_FILE | \ + PAGE_IS_PRESENT | PAGE_IS_SWAPPED | \ + PAGE_IS_PFNZERO | PAGE_IS_HUGE) + +#define TEST_ITERATIONS 100 +#define PAGEMAP "/proc/self/pagemap" +int pagemap_fd; +int uffd; +int page_size; +int hpage_size; + +#define LEN(region) ((region.end - region.start)/page_size) + +static long pagemap_ioctl(void *start, int len, void *vec, int vec_len, int flag, + int max_pages, long required_mask, long anyof_mask, long excluded_mask, + long return_mask) +{ + struct pm_scan_arg arg; + + arg.start = (uintptr_t)start; + arg.end = (uintptr_t)(start + len); + arg.vec = (uintptr_t)vec; + arg.vec_len = vec_len; + arg.flags = flag; + arg.size = sizeof(struct pm_scan_arg); + arg.max_pages = max_pages; + arg.category_mask = required_mask; + arg.category_anyof_mask = anyof_mask; + arg.category_inverted = excluded_mask; + arg.return_mask = return_mask; + + return ioctl(pagemap_fd, PAGEMAP_SCAN, &arg); +} + +static long pagemap_ioc(void *start, int len, void *vec, int vec_len, int flag, + int max_pages, long required_mask, long anyof_mask, long excluded_mask, + long return_mask, long *walk_end) +{ + struct pm_scan_arg arg; + int ret; + + arg.start = (uintptr_t)start; + arg.end = (uintptr_t)(start + len); + arg.vec = (uintptr_t)vec; + arg.vec_len = vec_len; + arg.flags = flag; + arg.size = sizeof(struct pm_scan_arg); + arg.max_pages = max_pages; + arg.category_mask = required_mask; + arg.category_anyof_mask = anyof_mask; + arg.category_inverted = excluded_mask; + arg.return_mask = return_mask; + + ret = ioctl(pagemap_fd, PAGEMAP_SCAN, &arg); + + if (walk_end) + *walk_end = arg.walk_end; + + return ret; +} + + +int init_uffd(void) +{ + struct uffdio_api uffdio_api; + + uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); + if (uffd == -1) + ksft_exit_fail_msg("uffd syscall failed\n"); + + uffdio_api.api = UFFD_API; + uffdio_api.features = UFFD_FEATURE_WP_UNPOPULATED | UFFD_FEATURE_WP_ASYNC | + UFFD_FEATURE_WP_HUGETLBFS_SHMEM; + if (ioctl(uffd, UFFDIO_API, &uffdio_api)) + ksft_exit_fail_msg("UFFDIO_API\n"); + + if (!(uffdio_api.api & UFFDIO_REGISTER_MODE_WP) || + !(uffdio_api.features & UFFD_FEATURE_WP_UNPOPULATED) || + !(uffdio_api.features & UFFD_FEATURE_WP_ASYNC) || + !(uffdio_api.features & UFFD_FEATURE_WP_HUGETLBFS_SHMEM)) + ksft_exit_fail_msg("UFFDIO_API error %llu\n", uffdio_api.api); + + return 0; +} + +int wp_init(void *lpBaseAddress, int dwRegionSize) +{ + struct uffdio_register uffdio_register; + struct uffdio_writeprotect wp; + + uffdio_register.range.start = (unsigned long)lpBaseAddress; + uffdio_register.range.len = dwRegionSize; + uffdio_register.mode = UFFDIO_REGISTER_MODE_WP; + if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register)) + ksft_exit_fail_msg("ioctl(UFFDIO_REGISTER) %d %s\n", errno, strerror(errno)); + + if (!(uffdio_register.ioctls & UFFDIO_WRITEPROTECT)) + ksft_exit_fail_msg("ioctl set is incorrect\n"); + + wp.range.start = (unsigned long)lpBaseAddress; + wp.range.len = dwRegionSize; + wp.mode = UFFDIO_WRITEPROTECT_MODE_WP; + + if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp)) + ksft_exit_fail_msg("ioctl(UFFDIO_WRITEPROTECT)\n"); + + return 0; +} + +int wp_free(void *lpBaseAddress, int dwRegionSize) +{ + struct uffdio_register uffdio_register; + + uffdio_register.range.start = (unsigned long)lpBaseAddress; + uffdio_register.range.len = dwRegionSize; + uffdio_register.mode = UFFDIO_REGISTER_MODE_WP; + if (ioctl(uffd, UFFDIO_UNREGISTER, &uffdio_register.range)) + ksft_exit_fail_msg("ioctl unregister failure\n"); + return 0; +} + +int wp_addr_range(void *lpBaseAddress, int dwRegionSize) +{ + struct uffdio_writeprotect wp; + + if (rand() % 2) { + wp.range.start = (unsigned long)lpBaseAddress; + wp.range.len = dwRegionSize; + wp.mode = UFFDIO_WRITEPROTECT_MODE_WP; + + if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp)) + ksft_exit_fail_msg("ioctl(UFFDIO_WRITEPROTECT)\n"); + } else { + if (pagemap_ioctl(lpBaseAddress, dwRegionSize, NULL, 0, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) < 0) + ksft_exit_fail_msg("error %d %d %s\n", 1, errno, strerror(errno)); + } + + return 0; +} + +void *gethugetlb_mem(int size, int *shmid) +{ + char *mem; + + if (shmid) { + *shmid = shmget(2, size, SHM_HUGETLB | IPC_CREAT | SHM_R | SHM_W); + if (*shmid < 0) + return NULL; + + mem = shmat(*shmid, 0, 0); + if (mem == (char *)-1) { + shmctl(*shmid, IPC_RMID, NULL); + ksft_exit_fail_msg("Shared memory attach failure\n"); + } + } else { + mem = mmap(NULL, size, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_HUGETLB | MAP_PRIVATE, -1, 0); + if (mem == MAP_FAILED) + return NULL; + } + + return mem; +} + +int userfaultfd_tests(void) +{ + int mem_size, vec_size, written, num_pages = 16; + char *mem, *vec; + + mem_size = num_pages * page_size; + mem = mmap(NULL, mem_size, PROT_NONE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + + wp_init(mem, mem_size); + + /* Change protection of pages differently */ + mprotect(mem, mem_size/8, PROT_READ|PROT_WRITE); + mprotect(mem + 1 * mem_size/8, mem_size/8, PROT_READ); + mprotect(mem + 2 * mem_size/8, mem_size/8, PROT_READ|PROT_WRITE); + mprotect(mem + 3 * mem_size/8, mem_size/8, PROT_READ); + mprotect(mem + 4 * mem_size/8, mem_size/8, PROT_READ|PROT_WRITE); + mprotect(mem + 5 * mem_size/8, mem_size/8, PROT_NONE); + mprotect(mem + 6 * mem_size/8, mem_size/8, PROT_READ|PROT_WRITE); + mprotect(mem + 7 * mem_size/8, mem_size/8, PROT_READ); + + wp_addr_range(mem + (mem_size/16), mem_size - 2 * (mem_size/8)); + wp_addr_range(mem, mem_size); + + vec_size = mem_size/page_size; + vec = malloc(sizeof(struct page_region) * vec_size); + + written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + vec_size - 2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written == 0, "%s all new pages must not be written (dirty)\n", __func__); + + wp_free(mem, mem_size); + munmap(mem, mem_size); + free(vec); + return 0; +} + +int get_reads(struct page_region *vec, int vec_size) +{ + int i, sum = 0; + + for (i = 0; i < vec_size; i++) + sum += LEN(vec[i]); + + return sum; +} + +int sanity_tests_sd(void) +{ + int mem_size, vec_size, ret, ret2, ret3, i, num_pages = 10, total_pages = 0; + int total_writes, total_reads, reads, count; + struct page_region *vec, *vec2; + char *mem, *m[2]; + long walk_end; + + vec_size = 100; + mem_size = num_pages * page_size; + + vec = malloc(sizeof(struct page_region) * vec_size); + if (!vec) + ksft_exit_fail_msg("error nomem\n"); + + vec2 = malloc(sizeof(struct page_region) * vec_size); + if (!vec2) + ksft_exit_fail_msg("error nomem\n"); + + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + /* 1. wrong operation */ + ksft_test_result(pagemap_ioctl(mem, 0, vec, vec_size, 0, + 0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) == 0, + "%s Zero range size is valid\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, NULL, vec_size, 0, + 0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) < 0, + "%s output buffer must be specified with size\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, 0, 0, + 0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) == 0, + "%s output buffer can be 0\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, -1, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) < 0, + "%s wrong flag specified\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC | 0xFF, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) < 0, + "%s flag has extra bits specified\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, + 0, 0, 0, 0, PAGE_IS_WRITTEN) >= 0, + "%s no selection mask is specified\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, + 0, PAGE_IS_WRITTEN, PAGE_IS_WRITTEN, 0, 0) == 0, + "%s no return mask is specified\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, + 0, PAGE_IS_WRITTEN, 0, 0, 0x1000) < 0, + "%s wrong return mask specified\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 0, 0xFFF, PAGE_IS_WRITTEN, 0, PAGE_IS_WRITTEN) < 0, + "%s mixture of correct and wrong flag\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 0, 0, 0, PAGEMAP_BITS_ALL, PAGE_IS_WRITTEN) >= 0, + "%s PAGEMAP_BITS_ALL can be specified with PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC\n", + __func__); + + /* 2. Clear area with larger vec size */ + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + ksft_test_result(ret >= 0, "%s Clear area with larger vec size\n", __func__); + + /* 3. Repeated pattern of written and non-written pages */ + for (i = 0; i < mem_size; i += 2 * page_size) + mem[i]++; + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, PAGE_IS_WRITTEN, 0, + 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == mem_size/(page_size * 2), + "%s Repeated pattern of written and non-written pages\n", __func__); + + /* 4. Repeated pattern of written and non-written pages in parts */ + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + num_pages/2 - 2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ret2 = pagemap_ioctl(mem, mem_size, vec, 2, 0, 0, PAGE_IS_WRITTEN, 0, 0, + PAGE_IS_WRITTEN); + if (ret2 < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno)); + + ret3 = pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret3 < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret3, errno, strerror(errno)); + + ksft_test_result((ret + ret3) == num_pages/2 && ret2 == 2, + "%s Repeated pattern of written and non-written pages in parts %d %d %d\n", + __func__, ret, ret3, ret2); + + /* 5. Repeated pattern of written and non-written pages max_pages */ + for (i = 0; i < mem_size; i += 2 * page_size) + mem[i]++; + mem[(mem_size/page_size - 1) * page_size]++; + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + num_pages/2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ret2 = pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret2 < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno)); + + ksft_test_result(ret == num_pages/2 && ret2 == 1, + "%s Repeated pattern of written and non-written pages max_pages\n", + __func__); + + /* 6. only get 2 dirty pages and clear them as well */ + vec_size = mem_size/page_size; + memset(mem, -1, mem_size); + + /* get and clear second and third pages */ + ret = pagemap_ioctl(mem + page_size, 2 * page_size, vec, 1, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ret2 = pagemap_ioctl(mem, mem_size, vec2, vec_size, 0, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret2 < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno)); + + ksft_test_result(ret == 1 && LEN(vec[0]) == 2 && + vec[0].start == (uintptr_t)(mem + page_size) && + ret2 == 2 && LEN(vec2[0]) == 1 && vec2[0].start == (uintptr_t)mem && + LEN(vec2[1]) == vec_size - 3 && + vec2[1].start == (uintptr_t)(mem + 3 * page_size), + "%s only get 2 written pages and clear them as well\n", __func__); + + wp_free(mem, mem_size); + munmap(mem, mem_size); + + /* 7. Two regions */ + m[0] = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (m[0] == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + m[1] = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (m[1] == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + + wp_init(m[0], mem_size); + wp_init(m[1], mem_size); + wp_addr_range(m[0], mem_size); + wp_addr_range(m[1], mem_size); + + memset(m[0], 'a', mem_size); + memset(m[1], 'b', mem_size); + + wp_addr_range(m[0], mem_size); + + ret = pagemap_ioctl(m[1], mem_size, vec, 1, 0, 0, PAGE_IS_WRITTEN, 0, 0, + PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && LEN(vec[0]) == mem_size/page_size, + "%s Two regions\n", __func__); + + wp_free(m[0], mem_size); + wp_free(m[1], mem_size); + munmap(m[0], mem_size); + munmap(m[1], mem_size); + + free(vec); + free(vec2); + + /* 8. Smaller vec */ + mem_size = 1050 * page_size; + vec_size = mem_size/(page_size*2); + + vec = malloc(sizeof(struct page_region) * vec_size); + if (!vec) + ksft_exit_fail_msg("error nomem\n"); + + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + for (i = 0; i < mem_size/page_size; i += 2) + mem[i * page_size]++; + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + mem_size/(page_size*5), PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + total_pages += ret; + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + mem_size/(page_size*5), PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + total_pages += ret; + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + mem_size/(page_size*5), PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + total_pages += ret; + + ksft_test_result(total_pages == mem_size/(page_size*2), "%s Smaller max_pages\n", __func__); + + free(vec); + wp_free(mem, mem_size); + munmap(mem, mem_size); + total_pages = 0; + + /* 9. Smaller vec */ + mem_size = 10000 * page_size; + vec_size = 50; + + vec = malloc(sizeof(struct page_region) * vec_size); + if (!vec) + ksft_exit_fail_msg("error nomem\n"); + + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + for (count = 0; count < TEST_ITERATIONS; count++) { + total_writes = total_reads = 0; + walk_end = (long)mem; + + for (i = 0; i < mem_size; i += page_size) { + if (rand() % 2) { + mem[i]++; + total_writes++; + } + } + + while (total_reads < total_writes) { + ret = pagemap_ioc((void *)walk_end, mem_size-(walk_end - (long)mem), vec, + vec_size, PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + if (ret > vec_size) + break; + + reads = get_reads(vec, ret); + total_reads += reads; + } + + if (total_reads != total_writes) + break; + } + + ksft_test_result(count == TEST_ITERATIONS, "Smaller vec\n"); + + free(vec); + wp_free(mem, mem_size); + munmap(mem, mem_size); + + /* 10. Walk_end tester */ + vec_size = 1000; + mem_size = vec_size * page_size; + + vec = malloc(sizeof(struct page_region) * vec_size); + if (!vec) + ksft_exit_fail_msg("error nomem\n"); + + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + memset(mem, 0, mem_size); + + ret = pagemap_ioc(mem, 0, vec, vec_size, 0, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == 0 && walk_end == (long)mem, + "Walk_end: Same start and end address\n"); + + ret = pagemap_ioc(mem, 0, vec, vec_size, PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == 0 && walk_end == (long)mem, + "Walk_end: Same start and end with WP\n"); + + ret = pagemap_ioc(mem, 0, vec, 0, PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == 0 && walk_end == (long)mem, + "Walk_end: Same start and end with 0 output buffer \n"); + + ret = pagemap_ioc(mem, mem_size, vec, vec_size, 0, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == 1 && walk_end == (long)(mem + mem_size), + "Walk_end: Big vec\n"); + + ret = pagemap_ioc(mem, mem_size, vec, 1, 0, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == 1 && walk_end == (long)(mem + mem_size), + "Walk_end: vec of minimum length\n"); + + ret = pagemap_ioc(mem, mem_size, vec, 1, 0, + vec_size, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == 1 && walk_end == (long)(mem + mem_size), + "Walk_end: Max pages specified\n"); + + ret = pagemap_ioc(mem, mem_size, vec, vec_size, 0, + vec_size/2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == 1 && walk_end == (long)(mem + mem_size/2), + "Walk_end: Half max pages\n"); + + ret = pagemap_ioc(mem, mem_size, vec, vec_size, 0, + 1, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == 1 && walk_end == (long)(mem + page_size), + "Walk_end: 1 max page\n"); + + ret = pagemap_ioc(mem, mem_size, vec, vec_size, 0, + -1, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == 1 && walk_end == (long)(mem + mem_size), + "Walk_end: max pages\n"); + + wp_addr_range(mem, mem_size); + for (i = 0; i < mem_size; i += 2 * page_size) + mem[i]++; + + ret = pagemap_ioc(mem, mem_size, vec, vec_size, 0, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == vec_size/2 && walk_end == (long)(mem + mem_size), + "Walk_end sparse: Big vec\n"); + + ret = pagemap_ioc(mem, mem_size, vec, 1, 0, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == 1 && walk_end == (long)(mem + page_size * 2), + "Walk_end sparse: vec of minimum length\n"); + + ret = pagemap_ioc(mem, mem_size, vec, 1, 0, + vec_size, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == 1 && walk_end == (long)(mem + page_size * 2), + "Walk_end sparse: Max pages specified\n"); + + ret = pagemap_ioc(mem, mem_size, vec, vec_size/2, 0, + vec_size, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == vec_size/2 && walk_end == (long)(mem + mem_size), + "Walk_end sparse: Max pages specified\n"); + + ret = pagemap_ioc(mem, mem_size, vec, vec_size, 0, + vec_size, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == vec_size/2 && walk_end == (long)(mem + mem_size), + "Walk_end sparse: Max pages specified\n"); + + ret = pagemap_ioc(mem, mem_size, vec, vec_size, 0, + vec_size/2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == vec_size/2 && walk_end == (long)(mem + mem_size), + "Walk_endsparse : Half max pages\n"); + + ret = pagemap_ioc(mem, mem_size, vec, vec_size, 0, + 1, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN, &walk_end); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + ksft_test_result(ret == 1 && walk_end == (long)(mem + page_size * 2), + "Walk_end: 1 max page\n"); + + free(vec); + wp_free(mem, mem_size); + munmap(mem, mem_size); + + return 0; +} + +int base_tests(char *prefix, char *mem, int mem_size, int skip) +{ + int vec_size, written; + struct page_region *vec, *vec2; + + if (skip) { + ksft_test_result_skip("%s all new pages must not be written (dirty)\n", prefix); + ksft_test_result_skip("%s all pages must be written (dirty)\n", prefix); + ksft_test_result_skip("%s all pages dirty other than first and the last one\n", + prefix); + ksft_test_result_skip("%s PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC\n", prefix); + ksft_test_result_skip("%s only middle page dirty\n", prefix); + ksft_test_result_skip("%s only two middle pages dirty\n", prefix); + return 0; + } + + vec_size = mem_size/page_size; + vec = malloc(sizeof(struct page_region) * vec_size); + vec2 = malloc(sizeof(struct page_region) * vec_size); + + /* 1. all new pages must be not be written (dirty) */ + written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + vec_size - 2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written == 0, "%s all new pages must not be written (dirty)\n", prefix); + + /* 2. all pages must be written */ + memset(mem, -1, mem_size); + + written = pagemap_ioctl(mem, mem_size, vec, 1, 0, 0, PAGE_IS_WRITTEN, 0, 0, + PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written == 1 && LEN(vec[0]) == mem_size/page_size, + "%s all pages must be written (dirty)\n", prefix); + + /* 3. all pages dirty other than first and the last one */ + written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + memset(mem + page_size, 0, mem_size - (2 * page_size)); + + written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written == 1 && LEN(vec[0]) >= vec_size - 2 && LEN(vec[0]) <= vec_size, + "%s all pages dirty other than first and the last one\n", prefix); + + written = pagemap_ioctl(mem, mem_size, vec, 1, 0, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written == 0, + "%s PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC\n", prefix); + + /* 4. only middle page dirty */ + written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + mem[vec_size/2 * page_size]++; + + written = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, PAGE_IS_WRITTEN, + 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written == 1 && LEN(vec[0]) >= 1, + "%s only middle page dirty\n", prefix); + + /* 5. only two middle pages dirty and walk over only middle pages */ + written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + mem[vec_size/2 * page_size]++; + mem[(vec_size/2 + 1) * page_size]++; + + written = pagemap_ioctl(&mem[vec_size/2 * page_size], 2 * page_size, vec, 1, 0, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written == 1 && vec[0].start == (uintptr_t)(&mem[vec_size/2 * page_size]) + && LEN(vec[0]) == 2, + "%s only two middle pages dirty\n", prefix); + + free(vec); + free(vec2); + return 0; +} + +void *gethugepage(int map_size) +{ + int ret; + char *map; + + map = memalign(hpage_size, map_size); + if (!map) + ksft_exit_fail_msg("memalign failed %d %s\n", errno, strerror(errno)); + + ret = madvise(map, map_size, MADV_HUGEPAGE); + if (ret) + return NULL; + + memset(map, 0, map_size); + + return map; +} + +int hpage_unit_tests(void) +{ + char *map; + int ret, ret2; + size_t num_pages = 10; + int map_size = hpage_size * num_pages; + int vec_size = map_size/page_size; + struct page_region *vec, *vec2; + + vec = malloc(sizeof(struct page_region) * vec_size); + vec2 = malloc(sizeof(struct page_region) * vec_size); + if (!vec || !vec2) + ksft_exit_fail_msg("malloc failed\n"); + + map = gethugepage(map_size); + if (map) { + wp_init(map, map_size); + wp_addr_range(map, map_size); + + /* 1. all new huge page must not be written (dirty) */ + ret = pagemap_ioctl(map, map_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 0, "%s all new huge page must not be written (dirty)\n", + __func__); + + /* 2. all the huge page must not be written */ + ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 0, "%s all the huge page must not be written\n", __func__); + + /* 3. all the huge page must be written and clear dirty as well */ + memset(map, -1, map_size); + ret = pagemap_ioctl(map, map_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && vec[0].start == (uintptr_t)map && + LEN(vec[0]) == vec_size && vec[0].categories == PAGE_IS_WRITTEN, + "%s all the huge page must be written and clear\n", __func__); + + /* 4. only middle page written */ + wp_free(map, map_size); + free(map); + map = gethugepage(map_size); + wp_init(map, map_size); + wp_addr_range(map, map_size); + map[vec_size/2 * page_size]++; + + ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && LEN(vec[0]) > 0, + "%s only middle page written\n", __func__); + + wp_free(map, map_size); + free(map); + } else { + ksft_test_result_skip("%s all new huge page must be written\n", __func__); + ksft_test_result_skip("%s all the huge page must not be written\n", __func__); + ksft_test_result_skip("%s all the huge page must be written and clear\n", __func__); + ksft_test_result_skip("%s only middle page written\n", __func__); + } + + /* 5. clear first half of huge page */ + map = gethugepage(map_size); + if (map) { + wp_init(map, map_size); + wp_addr_range(map, map_size); + + memset(map, 0, map_size); + + wp_addr_range(map, map_size/2); + + ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && LEN(vec[0]) == vec_size/2 && + vec[0].start == (uintptr_t)(map + map_size/2), + "%s clear first half of huge page\n", __func__); + wp_free(map, map_size); + free(map); + } else { + ksft_test_result_skip("%s clear first half of huge page\n", __func__); + } + + /* 6. clear first half of huge page with limited buffer */ + map = gethugepage(map_size); + if (map) { + wp_init(map, map_size); + wp_addr_range(map, map_size); + + memset(map, 0, map_size); + + ret = pagemap_ioctl(map, map_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + vec_size/2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && LEN(vec[0]) == vec_size/2 && + vec[0].start == (uintptr_t)(map + map_size/2), + "%s clear first half of huge page with limited buffer\n", + __func__); + wp_free(map, map_size); + free(map); + } else { + ksft_test_result_skip("%s clear first half of huge page with limited buffer\n", + __func__); + } + + /* 7. clear second half of huge page */ + map = gethugepage(map_size); + if (map) { + wp_init(map, map_size); + wp_addr_range(map, map_size); + + memset(map, -1, map_size); + + ret = pagemap_ioctl(map + map_size/2, map_size/2, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, vec_size/2, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && LEN(vec[0]) == vec_size/2, + "%s clear second half huge page\n", __func__); + wp_free(map, map_size); + free(map); + } else { + ksft_test_result_skip("%s clear second half huge page\n", __func__); + } + + /* 8. get half huge page */ + map = gethugepage(map_size); + if (map) { + wp_init(map, map_size); + wp_addr_range(map, map_size); + + memset(map, -1, map_size); + usleep(100); + + ret = pagemap_ioctl(map, map_size, vec, 1, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + hpage_size/(2*page_size), PAGE_IS_WRITTEN, 0, 0, + PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && LEN(vec[0]) == hpage_size/(2*page_size), + "%s get half huge page\n", __func__); + + ret2 = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret2 < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno)); + + ksft_test_result(ret2 == 1 && LEN(vec[0]) == (map_size - hpage_size/2)/page_size, + "%s get half huge page\n", __func__); + + wp_free(map, map_size); + free(map); + } else { + ksft_test_result_skip("%s get half huge page\n", __func__); + ksft_test_result_skip("%s get half huge page\n", __func__); + } + + free(vec); + free(vec2); + return 0; +} + +int unmapped_region_tests(void) +{ + void *start = (void *)0x10000000; + int written, len = 0x00040000; + int vec_size = len / page_size; + struct page_region *vec = malloc(sizeof(struct page_region) * vec_size); + + /* 1. Get written pages */ + written = pagemap_ioctl(start, len, vec, vec_size, 0, 0, + PAGEMAP_NON_WRITTEN_BITS, 0, 0, PAGEMAP_NON_WRITTEN_BITS); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written >= 0, "%s Get status of pages\n", __func__); + + free(vec); + return 0; +} + +static void test_simple(void) +{ + int i; + char *map; + struct page_region vec; + + map = aligned_alloc(page_size, page_size); + if (!map) + ksft_exit_fail_msg("aligned_alloc failed\n"); + + wp_init(map, page_size); + wp_addr_range(map, page_size); + + for (i = 0 ; i < TEST_ITERATIONS; i++) { + if (pagemap_ioctl(map, page_size, &vec, 1, 0, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) == 1) { + ksft_print_msg("written bit was 1, but should be 0 (i=%d)\n", i); + break; + } + + wp_addr_range(map, page_size); + /* Write something to the page to get the written bit enabled on the page */ + map[0]++; + + if (pagemap_ioctl(map, page_size, &vec, 1, 0, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) == 0) { + ksft_print_msg("written bit was 0, but should be 1 (i=%d)\n", i); + break; + } + + wp_addr_range(map, page_size); + } + wp_free(map, page_size); + free(map); + + ksft_test_result(i == TEST_ITERATIONS, "Test %s\n", __func__); +} + +int sanity_tests(void) +{ + int mem_size, vec_size, ret, fd, i, buf_size; + struct page_region *vec; + char *mem, *fmem; + struct stat sbuf; + char *tmp_buf; + + /* 1. wrong operation */ + mem_size = 10 * page_size; + vec_size = mem_size / page_size; + + vec = malloc(sizeof(struct page_region) * vec_size); + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED || vec == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, + 0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) >= 0, + "%s WP op can be specified with !PAGE_IS_WRITTEN\n", __func__); + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, + PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) >= 0, + "%s required_mask specified\n", __func__); + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, + 0, PAGEMAP_BITS_ALL, 0, PAGEMAP_BITS_ALL) >= 0, + "%s anyof_mask specified\n", __func__); + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, + 0, 0, PAGEMAP_BITS_ALL, PAGEMAP_BITS_ALL) >= 0, + "%s excluded_mask specified\n", __func__); + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, + PAGEMAP_BITS_ALL, PAGEMAP_BITS_ALL, 0, + PAGEMAP_BITS_ALL) >= 0, + "%s required_mask and anyof_mask specified\n", __func__); + wp_free(mem, mem_size); + munmap(mem, mem_size); + + /* 2. Get sd and present pages with anyof_mask */ + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + memset(mem, 0, mem_size); + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, + 0, PAGEMAP_BITS_ALL, 0, PAGEMAP_BITS_ALL); + ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && LEN(vec[0]) == vec_size && + (vec[0].categories & (PAGE_IS_WRITTEN | PAGE_IS_PRESENT)) == + (PAGE_IS_WRITTEN | PAGE_IS_PRESENT), + "%s Get sd and present pages with anyof_mask\n", __func__); + + /* 3. Get sd and present pages with required_mask */ + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, + PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL); + ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && LEN(vec[0]) == vec_size && + (vec[0].categories & (PAGE_IS_WRITTEN | PAGE_IS_PRESENT)) == + (PAGE_IS_WRITTEN | PAGE_IS_PRESENT), + "%s Get all the pages with required_mask\n", __func__); + + /* 4. Get sd and present pages with required_mask and anyof_mask */ + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, + PAGE_IS_WRITTEN, PAGE_IS_PRESENT, 0, PAGEMAP_BITS_ALL); + ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && LEN(vec[0]) == vec_size && + (vec[0].categories & (PAGE_IS_WRITTEN | PAGE_IS_PRESENT)) == + (PAGE_IS_WRITTEN | PAGE_IS_PRESENT), + "%s Get sd and present pages with required_mask and anyof_mask\n", + __func__); + + /* 5. Don't get sd pages */ + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, + PAGE_IS_WRITTEN, 0, PAGE_IS_WRITTEN, PAGEMAP_BITS_ALL); + ksft_test_result(ret == 0, "%s Don't get sd pages\n", __func__); + + /* 6. Don't get present pages */ + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, + PAGE_IS_PRESENT, 0, PAGE_IS_PRESENT, PAGEMAP_BITS_ALL); + ksft_test_result(ret == 0, "%s Don't get present pages\n", __func__); + + wp_free(mem, mem_size); + munmap(mem, mem_size); + + /* 8. Find written present pages with return mask */ + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + memset(mem, 0, mem_size); + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC, 0, + 0, PAGEMAP_BITS_ALL, 0, PAGE_IS_WRITTEN); + ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && LEN(vec[0]) == vec_size && + vec[0].categories == PAGE_IS_WRITTEN, + "%s Find written present pages with return mask\n", __func__); + wp_free(mem, mem_size); + munmap(mem, mem_size); + + /* 9. Memory mapped file */ + fd = open(__FILE__, O_RDONLY); + if (fd < 0) + ksft_exit_fail_msg("%s Memory mapped file\n"); + + ret = stat(__FILE__, &sbuf); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + fmem = mmap(NULL, sbuf.st_size, PROT_READ, MAP_PRIVATE, fd, 0); + if (fmem == MAP_FAILED) + ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno)); + + tmp_buf = malloc(sbuf.st_size); + memcpy(tmp_buf, fmem, sbuf.st_size); + + ret = pagemap_ioctl(fmem, sbuf.st_size, vec, vec_size, 0, 0, + 0, PAGEMAP_NON_WRITTEN_BITS, 0, PAGEMAP_NON_WRITTEN_BITS); + + ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)fmem && + LEN(vec[0]) == ceilf((float)sbuf.st_size/page_size) && + (vec[0].categories & PAGE_IS_FILE), + "%s Memory mapped file\n", __func__); + + munmap(fmem, sbuf.st_size); + close(fd); + + /* 10. Create and read/write to a memory mapped file */ + buf_size = page_size * 10; + + fd = open(__FILE__".tmp2", O_RDWR | O_CREAT, 0666); + if (fd < 0) + ksft_exit_fail_msg("Read/write to memory: %s\n", + strerror(errno)); + + for (i = 0; i < buf_size; i++) + if (write(fd, "c", 1) < 0) + ksft_exit_fail_msg("Create and read/write to a memory mapped file\n"); + + fmem = mmap(NULL, buf_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0); + if (fmem == MAP_FAILED) + ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno)); + + wp_init(fmem, buf_size); + wp_addr_range(fmem, buf_size); + + for (i = 0; i < buf_size; i++) + fmem[i] = 'z'; + + msync(fmem, buf_size, MS_SYNC); + + ret = pagemap_ioctl(fmem, buf_size, vec, vec_size, 0, 0, + PAGE_IS_WRITTEN, PAGE_IS_PRESENT | PAGE_IS_SWAPPED | PAGE_IS_FILE, 0, + PAGEMAP_BITS_ALL); + + ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)fmem && + LEN(vec[0]) == (buf_size/page_size) && + (vec[0].categories & PAGE_IS_WRITTEN), + "%s Read/write to memory\n", __func__); + + wp_free(fmem, buf_size); + munmap(fmem, buf_size); + close(fd); + + free(vec); + return 0; +} + +int mprotect_tests(void) +{ + int ret; + char *mem, *mem2; + struct page_region vec; + int pagemap_fd = open("/proc/self/pagemap", O_RDONLY); + + if (pagemap_fd < 0) { + fprintf(stderr, "open() failed\n"); + exit(1); + } + + /* 1. Map two pages */ + mem = mmap(0, 2 * page_size, PROT_READ|PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + wp_init(mem, 2 * page_size); + wp_addr_range(mem, 2 * page_size); + + /* Populate both pages. */ + memset(mem, 1, 2 * page_size); + + ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, 0, 0, PAGE_IS_WRITTEN, + 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && LEN(vec) == 2, "%s Both pages written\n", __func__); + + /* 2. Start tracking */ + wp_addr_range(mem, 2 * page_size); + + ksft_test_result(pagemap_ioctl(mem, 2 * page_size, &vec, 1, 0, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) == 0, + "%s Both pages are not written (dirty)\n", __func__); + + /* 3. Remap the second page */ + mem2 = mmap(mem + page_size, page_size, PROT_READ|PROT_WRITE, + MAP_PRIVATE|MAP_ANON|MAP_FIXED, -1, 0); + if (mem2 == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + wp_init(mem2, page_size); + wp_addr_range(mem2, page_size); + + /* Protect + unprotect. */ + mprotect(mem, page_size, PROT_NONE); + mprotect(mem, 2 * page_size, PROT_READ); + mprotect(mem, 2 * page_size, PROT_READ|PROT_WRITE); + + /* Modify both pages. */ + memset(mem, 2, 2 * page_size); + + /* Protect + unprotect. */ + mprotect(mem, page_size, PROT_NONE); + mprotect(mem, page_size, PROT_READ); + mprotect(mem, page_size, PROT_READ|PROT_WRITE); + + ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, 0, 0, PAGE_IS_WRITTEN, + 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && LEN(vec) == 2, + "%s Both pages written after remap and mprotect\n", __func__); + + /* 4. Clear and make the pages written */ + wp_addr_range(mem, 2 * page_size); + + memset(mem, 'A', 2 * page_size); + + ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, 0, 0, PAGE_IS_WRITTEN, + 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && LEN(vec) == 2, + "%s Clear and make the pages written\n", __func__); + + wp_free(mem, 2 * page_size); + munmap(mem, 2 * page_size); + return 0; +} + +/* transact test */ +static const unsigned int nthreads = 6, pages_per_thread = 32, access_per_thread = 8; +static pthread_barrier_t start_barrier, end_barrier; +static unsigned int extra_thread_faults; +static unsigned int iter_count = 1000; +static volatile int finish; + +static ssize_t get_dirty_pages_reset(char *mem, unsigned int count, + int reset, int page_size) +{ + struct pm_scan_arg arg = {0}; + struct page_region rgns[256]; + int i, j, cnt, ret; + + arg.size = sizeof(struct pm_scan_arg); + arg.start = (uintptr_t)mem; + arg.max_pages = count; + arg.end = (uintptr_t)(mem + count * page_size); + arg.vec = (uintptr_t)rgns; + arg.vec_len = sizeof(rgns) / sizeof(*rgns); + if (reset) + arg.flags |= PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC; + arg.category_mask = PAGE_IS_WRITTEN; + arg.return_mask = PAGE_IS_WRITTEN; + + ret = ioctl(pagemap_fd, PAGEMAP_SCAN, &arg); + if (ret < 0) + ksft_exit_fail_msg("ioctl failed\n"); + + cnt = 0; + for (i = 0; i < ret; ++i) { + if (rgns[i].categories != PAGE_IS_WRITTEN) + ksft_exit_fail_msg("wrong flags\n"); + + for (j = 0; j < LEN(rgns[i]); ++j) + cnt++; + } + + return cnt; +} + +void *thread_proc(void *mem) +{ + int *m = mem; + long curr_faults, faults; + struct rusage r; + unsigned int i; + int ret; + + if (getrusage(RUSAGE_THREAD, &r)) + ksft_exit_fail_msg("getrusage\n"); + + curr_faults = r.ru_minflt; + + while (!finish) { + ret = pthread_barrier_wait(&start_barrier); + if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD) + ksft_exit_fail_msg("pthread_barrier_wait\n"); + + for (i = 0; i < access_per_thread; ++i) + __atomic_add_fetch(m + i * (0x1000 / sizeof(*m)), 1, __ATOMIC_SEQ_CST); + + ret = pthread_barrier_wait(&end_barrier); + if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD) + ksft_exit_fail_msg("pthread_barrier_wait\n"); + + if (getrusage(RUSAGE_THREAD, &r)) + ksft_exit_fail_msg("getrusage\n"); + + faults = r.ru_minflt - curr_faults; + if (faults < access_per_thread) + ksft_exit_fail_msg("faults < access_per_thread"); + + __atomic_add_fetch(&extra_thread_faults, faults - access_per_thread, + __ATOMIC_SEQ_CST); + curr_faults = r.ru_minflt; + } + + return NULL; +} + +static void transact_test(int page_size) +{ + unsigned int i, count, extra_pages; + pthread_t th; + char *mem; + int ret, c; + + if (pthread_barrier_init(&start_barrier, NULL, nthreads + 1)) + ksft_exit_fail_msg("pthread_barrier_init\n"); + + if (pthread_barrier_init(&end_barrier, NULL, nthreads + 1)) + ksft_exit_fail_msg("pthread_barrier_init\n"); + + mem = mmap(NULL, 0x1000 * nthreads * pages_per_thread, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("Error mmap %s.\n", strerror(errno)); + + wp_init(mem, 0x1000 * nthreads * pages_per_thread); + wp_addr_range(mem, 0x1000 * nthreads * pages_per_thread); + + memset(mem, 0, 0x1000 * nthreads * pages_per_thread); + + count = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, page_size); + ksft_test_result(count > 0, "%s count %d\n", __func__, count); + count = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, page_size); + ksft_test_result(count == 0, "%s count %d\n", __func__, count); + + finish = 0; + for (i = 0; i < nthreads; ++i) + pthread_create(&th, NULL, thread_proc, mem + 0x1000 * i * pages_per_thread); + + extra_pages = 0; + for (i = 0; i < iter_count; ++i) { + count = 0; + + ret = pthread_barrier_wait(&start_barrier); + if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD) + ksft_exit_fail_msg("pthread_barrier_wait\n"); + + count = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, + page_size); + + ret = pthread_barrier_wait(&end_barrier); + if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD) + ksft_exit_fail_msg("pthread_barrier_wait\n"); + + if (count > nthreads * access_per_thread) + ksft_exit_fail_msg("Too big count %d expected %d, iter %d\n", + count, nthreads * access_per_thread, i); + + c = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, page_size); + count += c; + + if (c > nthreads * access_per_thread) { + ksft_test_result_fail(" %s count > nthreads\n", __func__); + return; + } + + if (count != nthreads * access_per_thread) { + /* + * The purpose of the test is to make sure that no page updates are lost + * when the page updates and read-resetting soft dirty flags are performed + * in parallel. However, it is possible that the application will get the + * soft dirty flags twice on the two consecutive read-resets. This seems + * unavoidable as soft dirty flag is handled in software through page faults + * in kernel. While the updating the flags is supposed to be synchronized + * between page fault handling and read-reset, it is possible that + * read-reset happens after page fault PTE update but before the application + * re-executes write instruction. So read-reset gets the flag, clears write + * access and application gets page fault again for the same write. + */ + if (count < nthreads * access_per_thread) { + ksft_test_result_fail("Lost update, iter %d, %d vs %d.\n", i, count, + nthreads * access_per_thread); + return; + } + + extra_pages += count - nthreads * access_per_thread; + } + } + + pthread_barrier_wait(&start_barrier); + finish = 1; + pthread_barrier_wait(&end_barrier); + + ksft_test_result_pass("%s Extra pages %u (%.1lf%%), extra thread faults %d.\n", __func__, + extra_pages, + 100.0 * extra_pages / (iter_count * nthreads * access_per_thread), + extra_thread_faults); +} + +int main(void) +{ + int mem_size, shmid, buf_size, fd, i, ret; + char *mem, *map, *fmem; + struct stat sbuf; + + ksft_print_header(); + ksft_set_plan(108); + + page_size = getpagesize(); + hpage_size = read_pmd_pagesize(); + + pagemap_fd = open(PAGEMAP, O_RDONLY); + if (pagemap_fd < 0) + return -EINVAL; + + if (init_uffd()) + ksft_exit_fail_msg("uffd init failed\n"); + + /* + * Written (dirty) PTE bit tests + */ + + /* 1. Sanity testing */ + sanity_tests_sd(); + + /* 2. Normal page testing */ + mem_size = 10 * page_size; + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + base_tests("Page testing:", mem, mem_size, 0); + + wp_free(mem, mem_size); + munmap(mem, mem_size); + + /* 3. Large page testing */ + mem_size = 512 * 10 * page_size; + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + base_tests("Large Page testing:", mem, mem_size, 0); + + wp_free(mem, mem_size); + munmap(mem, mem_size); + + /* 4. Huge page testing */ + map = gethugepage(hpage_size); + if (map) { + wp_init(map, hpage_size); + wp_addr_range(map, hpage_size); + base_tests("Huge page testing:", map, hpage_size, 0); + wp_free(map, hpage_size); + free(map); + } else { + base_tests("Huge page testing:", NULL, 0, 1); + } + + /* 5. Hugetlb page testing */ + mem_size = 2*1024*1024; + mem = gethugetlb_mem(mem_size, &shmid); + if (mem) { + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + base_tests("Hugetlb shmem testing:", mem, mem_size, 0); + + wp_free(mem, mem_size); + shmctl(shmid, IPC_RMID, NULL); + } else { + base_tests("Hugetlb shmem testing:", NULL, 0, 1); + } + + /* 6. Hugetlb page testing */ + mem = gethugetlb_mem(mem_size, NULL); + if (mem) { + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + base_tests("Hugetlb mem testing:", mem, mem_size, 0); + + wp_free(mem, mem_size); + } else { + base_tests("Hugetlb mem testing:", NULL, 0, 1); + } + + /* 7. file memory testing */ + buf_size = page_size * 10; + + fd = open(__FILE__".tmp0", O_RDWR | O_CREAT, 0777); + if (fd < 0) + ksft_exit_fail_msg("Create and read/write to a memory mapped file: %s\n", + strerror(errno)); + + for (i = 0; i < buf_size; i++) + if (write(fd, "c", 1) < 0) + ksft_exit_fail_msg("Create and read/write to a memory mapped file\n"); + + ret = stat(__FILE__".tmp0", &sbuf); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + fmem = mmap(NULL, sbuf.st_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0); + if (fmem == MAP_FAILED) + ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno)); + + wp_init(fmem, sbuf.st_size); + wp_addr_range(fmem, sbuf.st_size); + + base_tests("File memory testing:", fmem, sbuf.st_size, 0); + + wp_free(fmem, sbuf.st_size); + munmap(fmem, sbuf.st_size); + close(fd); + + /* 8. file memory testing */ + buf_size = page_size * 10; + + fd = memfd_create(__FILE__".tmp00", MFD_NOEXEC_SEAL); + if (fd < 0) + ksft_exit_fail_msg("Create and read/write to a memory mapped file: %s\n", + strerror(errno)); + + if (ftruncate(fd, buf_size)) + ksft_exit_fail_msg("Error ftruncate\n"); + + for (i = 0; i < buf_size; i++) + if (write(fd, "c", 1) < 0) + ksft_exit_fail_msg("Create and read/write to a memory mapped file\n"); + + fmem = mmap(NULL, buf_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0); + if (fmem == MAP_FAILED) + ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno)); + + wp_init(fmem, buf_size); + wp_addr_range(fmem, buf_size); + + base_tests("File anonymous memory testing:", fmem, buf_size, 0); + + wp_free(fmem, buf_size); + munmap(fmem, buf_size); + close(fd); + + /* 9. Huge page tests */ + hpage_unit_tests(); + + /* 10. Iterative test */ + test_simple(); + + /* 11. Mprotect test */ + mprotect_tests(); + + /* 12. Transact test */ + transact_test(page_size); + + /* + * Other PTE bit tests + */ + + /* 1. Sanity testing */ + sanity_tests(); + + /* 2. Unmapped address test */ + unmapped_region_tests(); + + /* 3. Userfaultfd tests */ + userfaultfd_tests(); + + close(pagemap_fd); + return ksft_exit_pass(); +} diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index 9666c0c171ab8..cca4dd8ee45d3 100755 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -56,6 +56,8 @@ separated by spaces: memory protection key tests - soft_dirty test soft dirty page bit semantics +- pagemap + test pagemap_scan IOCTL - cow test copy-on-write semantics example: ./run_vmtests.sh -t "hmm mmap ksm" @@ -324,6 +326,8 @@ fi
CATEGORY="soft_dirty" run_test ./soft-dirty
+CATEGORY="pagemap" run_test ./pagemap_ioctl + # COW tests CATEGORY="cow" run_test ./cow
linux-kselftest-mirror@lists.linaro.org
-
Andrei Vagin -
Michał Mirosław -
Michał Mirosław -
Muhammad Usama Anjum