From: Ignacio Moreno Gonzalez Ignacio.MorenoGonzalez@kuka.com
commit c4608d1bf7c6 ("mm: mmap: map MAP_STACK to VM_NOHUGEPAGE") maps the mmap option MAP_STACK to VM_NOHUGEPAGE. This is also done if CONFIG_TRANSPARENT_HUGETABLES is not defined. But in that case, the VM_NOHUGEPAGE does not make sense.
I discovered this issue when trying to use the tool CRIU to checkpoint and restore a container. Our running kernel is compiled without CONFIG_TRANSPARENT_HUGETABLES. CRIU parses the output of /proc/<pid>/smaps and saves the "nh" flag. When trying to restore the container, CRIU fails to restore the "nh" mappings, since madvise() MADV_NOHUGEPAGE always returns an error because CONFIG_TRANSPARENT_HUGEPAGE is not defined.
Fixes: c4608d1bf7c6 ("mm: mmap: map MAP_STACK to VM_NOHUGEPAGE") Cc: stable@vger.kernel.org Reviewed-by: Lorenzo Stoakes lorenzo.stoakes@oracle.com Reviewed-by: Yang Shi yang@os.amperecomputing.com Reviewed-by: Liam R. Howlett Liam.Howlett@oracle.com Signed-off-by: Ignacio Moreno Gonzalez Ignacio.MorenoGonzalez@kuka.com --- I discovered this issue when trying to use the tool CRIU to checkpoint and restore a container. Our running kernel is compiled without CONFIG_TRANSPARENT_HUGEPAGE. CRIU parses the output of /proc/<pid>/smaps and saves the "nh" flag. When trying to restore the container, CRIU fails to restore the "nh" mappings, since madvise() MADV_NOHUGEPAGE always returns an error because CONFIG_TRANSPARENT_HUGEPAGE is not defined.
The mapping MAP_STACK -> VM_NOHUGEPAGE was introduced by commit c4608d1bf7c6 ("mm: mmap: map MAP_STACK to VM_NOHUGEPAGE") in order to fix a regression introduced by commit efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries"). The change introducing the regression (efa7df3e3bb5) was limited to THP kernels, but its fix (c4608d1bf7c6) is applied without checking if THP is set.
The mapping MAP_STACK -> VM_NOHUGEPAGE should only be applied if THP is enabled. --- Changes in v4: - Correct typo CONFIG_TRANSPARENT_HUGETABLES -> CONFIG_TRANSPARENT_HUGEPAGE - Copy description from cover letter to commit description - Link to v3: https://lore.kernel.org/r/20250507-map-map_stack-to-vm_nohugepage-only-if-th...
Changes in v3: - Exclude non-stable patch (for huge_mm.h) from this series to avoid mixing stable and non-stable patches, as suggested by Andrew. - Extend description in cover letter. - Link to v2: https://lore.kernel.org/r/20250506-map-map_stack-to-vm_nohugepage-only-if-th...
Changes in v2: - [Patch 1/2] Use '#ifdef' instead of '#if defined(...)' - [Patch 1/2] Add 'Fixes: c4608d1bf7c6...' - Create [Patch 2/2]
- Link to v1: https://lore.kernel.org/r/20250502-map-map_stack-to-vm_nohugepage-only-if-th... --- include/linux/mman.h | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/include/linux/mman.h b/include/linux/mman.h index bce214fece16b9af3791a2baaecd6063d0481938..f4c6346a8fcd29b08d43f7cd9158c3eddc3383e1 100644 --- a/include/linux/mman.h +++ b/include/linux/mman.h @@ -155,7 +155,9 @@ calc_vm_flag_bits(struct file *file, unsigned long flags) return _calc_vm_trans(flags, MAP_GROWSDOWN, VM_GROWSDOWN ) | _calc_vm_trans(flags, MAP_LOCKED, VM_LOCKED ) | _calc_vm_trans(flags, MAP_SYNC, VM_SYNC ) | +#ifdef CONFIG_TRANSPARENT_HUGEPAGE _calc_vm_trans(flags, MAP_STACK, VM_NOHUGEPAGE) | +#endif arch_calc_vm_flag_bits(file, flags); }
--- base-commit: fc96b232f8e7c0a6c282f47726b2ff6a5fb341d2 change-id: 20250428-map-map_stack-to-vm_nohugepage-only-if-thp-is-enabled-ce40a1de095d
Best regards,