From: "Kirill A. Shutemov" kirill.shutemov@linux.intel.com
commit 7aef4172c7957d7e65fc172be4c99becaef855d4 upstream.
With new refcounting we are going to see THP tail pages mapped with PTE. Generic fast GUP rely on page_cache_get_speculative() to obtain reference on page. page_cache_get_speculative() always fails on tail pages, because ->_count on tail pages is always zero.
Let's handle tail pages in gup_pte_range().
New split_huge_page() will rely on migration entries to freeze page's counts. Recheck PTE value after page_cache_get_speculative() on head page should be enough to serialize against split.
Signed-off-by: Kirill A. Shutemov kirill.shutemov@linux.intel.com Tested-by: Sasha Levin sasha.levin@oracle.com Tested-by: Aneesh Kumar K.V aneesh.kumar@linux.vnet.ibm.com Acked-by: Jerome Marchand jmarchan@redhat.com Acked-by: Vlastimil Babka vbabka@suse.cz Cc: Andrea Arcangeli aarcange@redhat.com Cc: Hugh Dickins hughd@google.com Cc: Dave Hansen dave.hansen@intel.com Cc: Mel Gorman mgorman@suse.de Cc: Rik van Riel riel@redhat.com Cc: Naoya Horiguchi n-horiguchi@ah.jp.nec.com Cc: Steve Capper steve.capper@linaro.org Cc: Johannes Weiner hannes@cmpxchg.org Cc: Michal Hocko mhocko@suse.cz Cc: Christoph Lameter cl@linux.com Cc: David Rientjes rientjes@google.com Signed-off-by: Andrew Morton akpm@linux-foundation.org Signed-off-by: Linus Torvalds torvalds@linux-foundation.org Signed-off-by: Ajay Kaher akaher@vmware.com --- mm/gup.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c index 2cd3b31..45c544b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1070,7 +1070,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, * for an example see gup_get_pte in arch/x86/mm/gup.c */ pte_t pte = READ_ONCE(*ptep); - struct page *page; + struct page *head, *page;
/* * Similar to the PMD case below, NUMA hinting must take slow @@ -1082,15 +1082,17 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,
VM_BUG_ON(!pfn_valid(pte_pfn(pte))); page = pte_page(pte); + head = compound_head(page);
- if (!page_cache_get_speculative(page)) + if (!page_cache_get_speculative(head)) goto pte_unmap;
if (unlikely(pte_val(pte) != pte_val(*ptep))) { - put_page(page); + put_page(head); goto pte_unmap; }
+ VM_BUG_ON_PAGE(compound_head(page) != head, page); pages[*nr] = page; (*nr)++;