On Mon, 10 Feb 2020 16:15:30 -0800 John Hubbard jhubbard@nvidia.com wrote:
Add tracking of pages that were pinned via FOLL_PIN. This tracking is implemented via overloading of page->_refcount: pins are added by adding GUP_PIN_COUNTING_BIAS (1024) to the refcount. This provides a fuzzy indication of pinning, and it can have false positives (and that's OK). Please see the pre-existing Documentation/core-api/pin_user_pages.rst for details.
As mentioned in pin_user_pages.rst, callers who effectively set FOLL_PIN (typically via pin_user_pages*()) are required to ultimately free such pages via unpin_user_page().
Please also note the limitation, discussed in pin_user_pages.rst under the "TODO: for 1GB and larger huge pages" section. (That limitation will be removed in a following patch.)
The effect of a FOLL_PIN flag is similar to that of FOLL_GET, and may be thought of as "FOLL_GET for DIO and/or RDMA use".
Pages that have been pinned via FOLL_PIN are identifiable via a new function call:
bool page_maybe_dma_pinned(struct page *page);
What to do in response to encountering such a page, is left to later patchsets. There is discussion about this in [1], [2], [3], and [4].
This also changes a BUG_ON(), to a WARN_ON(), in follow_page_mask().
[1] Some slow progress on get_user_pages() (Apr 2, 2019): https://lwn.net/Articles/784574/ [2] DMA and get_user_pages() (LPC: Dec 12, 2018): https://lwn.net/Articles/774411/ [3] The trouble with get_user_pages() (Apr 30, 2018): https://lwn.net/Articles/753027/ [4] LWN kernel index: get_user_pages(): https://lwn.net/Kernel/Index/#Memory_management-get_user_pages
Reviewed-by: Jan Kara jack@suse.cz Acked-by: Kirill A. Shutemov kirill.shutemov@linux.intel.com Suggested-by: Jan Kara jack@suse.cz Suggested-by: Jérôme Glisse jglisse@redhat.com Signed-off-by: John Hubbard jhubbard@nvidia.com
Documentation/core-api/pin_user_pages.rst | 6 +- include/linux/mm.h | 82 +++++-- mm/gup.c | 254 +++++++++++++++++----- mm/huge_memory.c | 29 ++- mm/hugetlb.c | 54 +++-- 5 files changed, 334 insertions(+), 91 deletions(-)
Hi John,
I'm seeing a regression bisected back to this commit (3faa52c03f44 mm/gup: track FOLL_PIN pages). I've attached some vfio-pci test code that reproduces this by mmap'ing a page of MMIO space of a device and then tries to map that through the IOMMU, so this should be attempting a gup/pin of a PFNMAP page. Previously this failed gracefully (-EFAULT), but now results in:
BUG: unable to handle page fault for address: ffffae5cbfe5e938 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: 0000 [#1] SMP NOPTI CPU: 18 PID: 3365 Comm: vfio-pci-dma-ma Tainted: G OE 5.6.0+ #6 Hardware name: AMD Corporation Diesel/Diesel, BIOS TDL100CB 03/17/2020 RIP: 0010:get_pfnblock_flags_mask+0x22/0x70 Code: c3 0f 1f 80 00 00 00 00 0f 1f 44 00 00 48 8b 05 bc e1 d9 01 48 89 f7 49 89 c8 48 c1 ef 0f 48 85 c0 74 48 48 89 f1 48 c1 e9 17 <48> 8b 04 c8 48 85 c0 74 0b 40 0f b6 ff 48 c1 e7 04 48 01 f8 48 c1 RSP: 0018:ffffb55289b3fcc8 EFLAGS: 00010216 RAX: ffff9e5cbff50000 RBX: 0000000000000001 RCX: 000001fffffe1d27 RDX: 0000000000000002 RSI: ffffff0e93acd633 RDI: 0001fffffe1d2759 RBP: ffffb55289b3fd88 R08: 0000000000000007 R09: ffff9e48a52476a8 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 R13: 0000000000000000 R14: 0000000000000001 R15: ffff9e48ab358cc0 FS: 00007f4ef7269740(0000) GS:ffff9e48afa80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffae5cbfe5e938 CR3: 0000000c61eda000 CR4: 00000000003406e0 Call Trace: __gup_longterm_locked+0x274/0x620 vaddr_get_pfn+0x74/0x110 [vfio_iommu_type1] vfio_pin_pages_remote+0x6e/0x370 [vfio_iommu_type1] vfio_iommu_type1_ioctl+0x8e5/0xaac [vfio_iommu_type1] ksys_ioctl+0x86/0xc0 __x64_sys_ioctl+0x16/0x20 do_syscall_64+0x5b/0x1f0 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7f4ef6d7d307 Code: 44 00 00 48 8b 05 69 1b 2d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 39 1b 2d 00 f7 d8 64 89 01 48 RSP: 002b:00007fff76ada738 EFLAGS: 00000213 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4ef6d7d307 RDX: 00007fff76ada760 RSI: 0000000000003b71 RDI: 0000000000000003 RBP: 00007fff76ada930 R08: 0000000000000005 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000213 R12: 0000000000400950 R13: 00007fff76adaa10 R14: 0000000000000000 R15: 0000000000000000 Modules linked in: vfio_pci(OE) vfio_virqfd(OE) vfio_iommu_type1(OE) vfio(OE) amd64_edac_mod edac_mce_amd kvm_amd kvm rfkill sunrpc ipmi_ssif vfat irqbypass fat ipmi_si crct10dif_pclmul crc32_pclmul sp5100_tco ghash_clmulni_intel ipmi_devintf pcspkr joydev ccp i2c_piix4 k10temp ipmi_msghandler pinctrl_amd acpi_cpufreq ip_tables nouveau ast video mxm_wmi drm_vram_helper wmi drm_ttm_helper i2c_algo_bit drm_kms_helper cec ttm drm i40e e1000e crc32c_intel CR2: ffffae5cbfe5e938 ---[ end trace a384ab7cc8e37d46 ]--- RIP: 0010:get_pfnblock_flags_mask+0x22/0x70 Code: c3 0f 1f 80 00 00 00 00 0f 1f 44 00 00 48 8b 05 bc e1 d9 01 48 89 f7 49 89 c8 48 c1 ef 0f 48 85 c0 74 48 48 89 f1 48 c1 e9 17 <48> 8b 04 c8 48 85 c0 74 0b 40 0f b6 ff 48 c1 e7 04 48 01 f8 48 c1 RSP: 0018:ffffb55289b3fcc8 EFLAGS: 00010216 RAX: ffff9e5cbff50000 RBX: 0000000000000001 RCX: 000001fffffe1d27 RDX: 0000000000000002 RSI: ffffff0e93acd633 RDI: 0001fffffe1d2759 RBP: ffffb55289b3fd88 R08: 0000000000000007 R09: ffff9e48a52476a8 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 R13: 0000000000000000 R14: 0000000000000001 R15: ffff9e48ab358cc0 FS: 00007f4ef7269740(0000) GS:ffff9e48afa80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffae5cbfe5e938 CR3: 0000000c61eda000 CR4: 00000000003406e0
Thanks, Alex