David Hildenbrand 3523a37e65 mm: provide vm_normal_(page|folio)_pmd() with CONFIG_PGTABLE_HAS_HUGE_LEAVES ... Patch series "mm: replace follow_page() by folio_walk". Looking into a way of moving the last folio_likely_mapped_shared() call in add_folio_for_migration() under the PTL, I found myself removing follow_page(). This paves the way for cleaning up all the FOLL_, follow_* terminology to just be called "GUP" nowadays. The new page table walker will lookup a mapped folio and return to the caller with the PTL held, such that the folio cannot get unmapped concurrently. Callers can then conditionally decide whether they really want to take a short-term folio reference or whether the can simply unlock the PTL and be done with it. folio_walk is similar to page_vma_mapped_walk(), except that we don't know the folio we want to walk to and that we are only walking to exactly one PTE/PMD/PUD. folio_walk provides access to the pte/pmd/pud (and the referenced folio page because things like KSM need that), however, as part of this series no page table modifications are performed by users. We might be able to convert some other walk_page_range() users that really only walk to one address, such as DAMON with damon_mkold_ops/damon_young_ops. It might make sense to extend folio_walk in the future to optionally fault in a folio (if applicable), such that we can replace some get_user_pages() users that really only want to lookup a single page/folio under PTL without unconditionally grabbing a folio reference. I have plans to extend the approach to a range walker that will try batching various page table entries (not just folio pages) to be a better replace for walk_page_range() -- and users will be able to opt in which type of page table entries they want to process -- but that will require more work and more thoughts. KSM seems to work just fine (ksm_functional_tests selftests) and move_pages seems to work (migration selftest). I tested the leaf implementation excessively using various hugetlb sizes (64K, 2M, 32M, 1G) on arm64 using move_pages and did some more testing on x86-64. Cross compiled on a bunch of architectures. This patch (of 11): We want to make use of vm_normal_page_pmd() in generic page table walking code where we might walk hugetlb folios that are mapped by PMDs even without CONFIG_TRANSPARENT_HUGEPAGE. So let's expose vm_normal_page_pmd() + vm_normal_folio_pmd() with CONFIG_PGTABLE_HAS_HUGE_LEAVES. Link: https://lkml.kernel.org/r/20240802155524.517137-1-david@redhat.com Link: https://lkml.kernel.org/r/20240802155524.517137-2-david@redhat.com Signed-off-by: David Hildenbrand <david@redhat.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Matthew Wilcox <willy@infradead.org> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>		2024-09-01 20:25:59 -07:00
..
damon	mm/damon/lru_sort: adjust local variable to dynamic allocation	2024-09-01 20:25:45 -07:00
kasan	kasan: fix bad call to unpoison_slab_object	2024-06-24 20:52:09 -07:00
kfence	mm: remove CONFIG_MEMCG_KMEM	2024-07-10 12:14:54 -07:00
kmsan	kmsan: do not pass NULL pointers as 0	2024-07-03 19:30:26 -07:00
backing-dev.c	writeback: support retrieving per group debug writeback stats of bdi	2024-05-05 17:53:51 -07:00
balloon_compaction.c	mm: remove MIGRATE_SYNC_NO_COPY mode	2024-07-03 19:30:00 -07:00
bootmem_info.c	bootmem: use kmemleak_free_part_phys in put_page_bootmem	2023-10-25 16:47:13 -07:00
cma_debug.c	mm/cma_debug: show complete cma name in debugfs directories	2022-09-11 20:25:50 -07:00
cma_sysfs.c	mm/cma: add sysfs file 'release_pages_success'	2024-02-22 10:24:57 -08:00
cma.c	mm/cma: change the addition of totalcma_pages in the cma_init_reserved_mem	2024-09-01 20:25:56 -07:00
cma.h	mm/cma: add sysfs file 'release_pages_success'	2024-02-22 10:24:57 -08:00
compaction.c	sysctl: treewide: constify the ctl_table argument of proc_handlers	2024-07-24 20:59:29 +02:00
debug_page_alloc.c	mm: page_alloc: consolidate free page accounting	2024-04-25 20:56:04 -07:00
debug_page_ref.c
debug_vm_pgtable.c	mm/debug_vm_pgtable: drop RANDOM_ORVALUE trick	2024-06-15 10:43:08 -07:00
debug.c	mm/debug: print only page mapcount (excluding folio entire mapcount) in __dump_folio()	2024-05-05 17:53:31 -07:00
dmapool_test.c	mm/dmapool: add MODULE_DESCRIPTION()	2024-07-03 19:29:58 -07:00
dmapool.c	mm/mempool/dmapool: remove CONFIG_DEBUG_SLAB ifdefs	2023-12-05 11:17:58 +01:00
early_ioremap.c	mm/early_ioremap.c: improve the execution efficiency of early_ioremap_setup()	2023-06-09 16:25:56 -07:00
execmem.c	mm/execmem, arch: convert remaining overrides of module_alloc to execmem	2024-05-14 00:31:43 -07:00
fadvise.c	mm: remove unnecessary pagevec includes	2023-06-23 16:59:31 -07:00
fail_page_alloc.c	mm, page_alloc: put should_fail_alloc_page() back behing CONFIG_FAIL_PAGE_ALLOC	2024-07-17 21:05:18 -07:00
failslab.c	mm, slab: put should_failslab() back behind CONFIG_SHOULD_FAILSLAB	2024-07-17 21:05:18 -07:00
filemap.c	- 875fa64577da ("mm/hugetlb_vmemmap: fix race with speculative PFN	2024-07-21 17:15:46 -07:00
folio-compat.c	mm: remove page_mapping()	2024-07-03 19:29:59 -07:00
gup_test.c	Merge mm-hotfixes-stable into mm-stable to pick up depended-upon changes.	2023-06-23 16:58:19 -07:00
gup_test.h	mm/gup_test: start/stop/read functionality for PIN LONGTERM test	2022-11-08 17:37:15 -08:00
gup.c	mm/gup: convert to arch_make_folio_accessible()	2024-09-01 20:25:52 -07:00
highmem.c	mm/highmem: make nr_free_highpages() return "unsigned long"	2024-07-03 19:30:06 -07:00
hmm.c	mm: provide mm_struct and address to huge_ptep_get()	2024-07-12 15:52:15 -07:00
huge_memory.c	memory tiering: introduce folio_use_access_time() check	2024-09-01 20:25:47 -07:00
hugetlb_cgroup.c	mm: memcg: don't call propagate_protected_usage() needlessly	2024-09-01 20:25:50 -07:00
hugetlb_vmemmap.c	mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO	2024-09-01 20:25:45 -07:00
hugetlb_vmemmap.h	mm: hugetlb_vmemmap: fix reference to nonexistent file	2023-10-25 16:47:14 -07:00
hugetlb.c	mm/hugetlb: enforce that PMD PT sharing has split PMD PT locks	2024-09-01 20:25:51 -07:00
hwpoison-inject.c	mm/hwpoison: add MODULE_DESCRIPTION()	2024-07-03 19:29:58 -07:00
init-mm.c	mm: Deprecate pasid field	2023-12-12 10:11:32 +01:00
internal.h	mm: move internal core VMA manipulation functions to own file	2024-09-01 20:25:54 -07:00
interval_tree.c
io-mapping.c
ioremap.c	mm: ioremap: remove unneeded ioremap_allowed and iounmap_allowed	2023-08-18 10:12:36 -07:00
Kconfig	mm: turn USE_SPLIT_PTE_PTLOCKS / USE_SPLIT_PTE_PTLOCKS into Kconfig options	2024-09-01 20:25:51 -07:00
Kconfig.debug	mm/slub: unify all sl[au]b parameters with "slab_$param"	2024-01-22 10:31:08 +01:00
khugepaged.c	- 875fa64577da ("mm/hugetlb_vmemmap: fix race with speculative PFN	2024-07-21 17:15:46 -07:00
kmemleak.c	kmemleak: enable tracking for percpu pointers	2024-09-01 20:25:49 -07:00
ksm.c	Random number generator updates for Linux 6.11-rc1.	2024-07-24 10:29:50 -07:00
list_lru.c	mm: list_lru: fix UAF for memory cgroup	2024-08-07 18:33:56 -07:00
maccess.c	mm: Fix copy_from_user_nofault().	2023-04-12 17:36:23 -07:00
madvise.c	Random number generator updates for Linux 6.11-rc1.	2024-07-24 10:29:50 -07:00
Makefile	mm: move internal core VMA manipulation functions to own file	2024-09-01 20:25:54 -07:00
mapping_dirty_helpers.c	mm: fix clean_record_shared_mapping_range kernel-doc	2023-08-24 16:20:30 -07:00
memblock.c	memblock: updates for 6.11-rc1	2024-07-18 14:48:11 -07:00
memcontrol-v1.c	memcg_write_event_control(): fix a user-triggerable oops	2024-08-12 21:58:44 -04:00
memcontrol-v1.h	mm: memcg: gather memcg1-specific fields initialization in memcg1_memcg_init()	2024-07-04 18:05:56 -07:00
memcontrol.c	mm, memcg: cg2 memory{.swap,}.peak write handlers	2024-09-01 20:25:53 -07:00
memfd.c	mm/gup: introduce memfd_pin_folios() for pinning memfd folios	2024-07-12 15:52:09 -07:00
memory_hotplug.c	mm/memory_hotplug: get rid of __ref	2024-09-01 20:25:56 -07:00
memory-failure.c	mm/memory-failure: use raw_spinlock_t in struct memory_failure_cpu	2024-08-15 22:16:14 -07:00
memory-tiers.c	memory tiering: introduce folio_use_access_time() check	2024-09-01 20:25:47 -07:00
memory.c	mm: provide vm_normal_(page|folio)_pmd() with CONFIG_PGTABLE_HAS_HUGE_LEAVES	2024-09-01 20:25:59 -07:00
mempolicy.c	mm: improve code consistency with zonelist_* helper functions	2024-09-01 20:25:55 -07:00
mempool.c	mm: fix xyz_noprof functions calling profiled functions	2024-06-05 19:19:26 -07:00
memremap.c	mm: convert put_devmap_managed_page_refs() to put_devmap_managed_folio_refs()	2024-05-05 17:53:49 -07:00
memtest.c	memtest: use {READ,WRITE}_ONCE in memory scanning	2024-03-13 12:12:21 -07:00
migrate_device.c	mm: extend rmap flags arguments for folio_add_new_anon_rmap	2024-07-03 19:30:18 -07:00
migrate.c	mm: consider CMA pages in watermark check for NUMA balancing target node	2024-09-01 20:25:58 -07:00
mincore.c	mm: provide mm_struct and address to huge_ptep_get()	2024-07-12 15:52:15 -07:00
mlock.c	Random number generator updates for Linux 6.11-rc1.	2024-07-24 10:29:50 -07:00
mm_init.c	alloc_tag: mark pages reserved during CMA activation as not tagged	2024-08-15 22:16:16 -07:00
mm_slot.h	mm: introduce common struct mm_slot	2022-10-03 14:02:43 -07:00
mmap_lock.c	mm: mmap_lock: replace get_memcg_path_buf() with on-stack buffer	2024-07-03 19:30:26 -07:00
mmap.c	mm: move internal core VMA manipulation functions to own file	2024-09-01 20:25:54 -07:00
mmu_gather.c	mm/mmu_gather: improve cond_resched() handling with large folios and expensive page freeing	2024-02-22 15:27:17 -08:00
mmu_notifier.c	mm: move internal core VMA manipulation functions to own file	2024-09-01 20:25:54 -07:00
mmzone.c	mm: improve code consistency with zonelist_* helper functions	2024-09-01 20:25:55 -07:00
mprotect.c	memory tiering: introduce folio_use_access_time() check	2024-09-01 20:25:47 -07:00
mremap.c	mm: remove page_mkclean()	2024-07-03 19:30:17 -07:00
mseal.c	mseal: fix is_madv_discard()	2024-08-15 22:16:13 -07:00
msync.c	mm/msync: use vma_find() instead of vma linked list	2022-09-26 19:46:25 -07:00
nommu.c	mm: vmalloc: implement vrealloc()	2024-09-01 20:25:43 -07:00
oom_kill.c	memory: remove the now superfluous sentinel element from ctl_table array	2024-04-25 20:56:32 -07:00
page_alloc.c	mm: improve code consistency with zonelist_* helper functions	2024-09-01 20:25:55 -07:00
page_counter.c	mm, memcg: cg2 memory{.swap,}.peak write handlers	2024-09-01 20:25:53 -07:00
page_ext.c	mm: don't account memmap per-node	2024-08-15 22:16:14 -07:00
page_idle.c	mm: page_idle: convert page idle to use a folio	2023-01-18 17:12:52 -08:00
page_io.c	mm: ignore data-race in __swap_writepage	2024-07-17 21:05:18 -07:00
page_isolation.c	mm: page_isolation: prepare for hygienic freelists	2024-04-25 20:56:04 -07:00
page_owner.c	mm/page-owner: use gfp_nested_mask() instead of open coded masking	2024-05-19 14:40:44 -07:00
page_poison.c	mm/page_poison: replace kmap_atomic() with kmap_local_page()	2023-12-10 16:51:50 -08:00
page_reporting.c	mm, treewide: rename MAX_ORDER to MAX_PAGE_ORDER	2024-01-08 15:27:15 -08:00
page_reporting.h
page_table_check.c	mm/page_table_check: fix crash on ZONE_DEVICE	2024-06-15 10:43:04 -07:00
page_vma_mapped.c	mm: make page_mapped_in_vma conditional on CONFIG_MEMORY_FAILURE	2024-05-05 17:53:45 -07:00
page-writeback.c	sysctl: treewide: constify the ctl_table argument of proc_handlers	2024-07-24 20:59:29 +02:00
pagewalk.c	mm: remove CONFIG_ARCH_HAS_HUGEPD	2024-07-12 15:52:19 -07:00
percpu-internal.h	mm: remove CONFIG_MEMCG_KMEM	2024-07-10 12:14:54 -07:00
percpu-km.c
percpu-stats.c
percpu-vm.c	percpu: clean up all mappings when pcpu_map_pages() fails	2024-04-25 20:55:49 -07:00
percpu.c	mm: remove CONFIG_MEMCG_KMEM	2024-07-10 12:14:54 -07:00
pgalloc-track.h
pgtable-generic.c	mm: fix race between __split_huge_pmd_locked() and GUP-fast	2024-05-07 10:37:00 -07:00
process_vm_access.c	mm: fix process_vm_rw page counts	2023-12-10 16:51:39 -08:00
ptdump.c	mm: ptdump: add check_wx_pages debugfs attribute	2024-02-22 10:24:47 -08:00
readahead.c	Merge branch 'mm-hotfixes-stable' into mm-stable to pick up "mm: fix	2024-07-06 11:44:41 -07:00
rmap.c	mm/rmap: cleanup partially-mapped handling in __folio_remove_rmap()	2024-09-01 20:25:57 -07:00
rodata_test.c	mm/rodata_test: use PAGE_ALIGNED() helper	2022-10-03 14:03:05 -07:00
secretmem.c	mm/secretmem: use a folio in secretmem_fault()	2023-08-21 13:38:02 -07:00
shmem_quota.c	shmem_quota: build the object file conditionally to the config option	2024-09-01 20:25:45 -07:00
shmem.c	mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()	2024-09-01 20:25:44 -07:00
show_mem.c	lib: add memory allocations report in show_mem()	2024-04-25 20:55:57 -07:00
shrinker_debug.c	mm: shrinker: convert shrinker_rwsem to mutex	2023-10-04 10:32:26 -07:00
shrinker.c	mm: shrinker: use kvzalloc_node() from expand_one_shrinker_info()	2024-01-05 09:58:32 -08:00
shuffle.c	mm/shuffle: convert module_param_call to module_param_cb	2022-10-03 14:03:07 -07:00
shuffle.h	mm, treewide: rename MAX_ORDER to MAX_PAGE_ORDER	2024-01-08 15:27:15 -08:00
slab_common.c	- 875fa64577da ("mm/hugetlb_vmemmap: fix race with speculative PFN	2024-07-21 17:15:46 -07:00
slab.h	- 875fa64577da ("mm/hugetlb_vmemmap: fix race with speculative PFN	2024-07-21 17:15:46 -07:00
slub.c	mm, slub: do not call do_slab_free for kfence object	2024-07-30 11:50:00 +02:00
sparse-vmemmap.c	mm: don't account memmap per-node	2024-08-15 22:16:14 -07:00
sparse.c	mm: don't account memmap per-node	2024-08-15 22:16:14 -07:00
swap_cgroup.c	mm: memcontrol: don't allocate cgroup swap arrays when memcg is disabled	2022-10-03 14:03:36 -07:00
swap_slots.c	mm: swap: update get_swap_pages() to take folio order	2024-04-25 20:56:37 -07:00
swap_state.c	mm: swap: allocate folio only first time in __read_swap_cache_async()	2024-09-01 20:25:57 -07:00
swap.c	mm/swap: remove boilerplate	2024-09-01 20:25:48 -07:00
swap.h	mm: swap: add nr argument in swapcache_prepare and swapcache_clear to support large folios	2024-09-01 20:25:56 -07:00
swapfile.c	mm: swap: add nr argument in swapcache_prepare and swapcache_clear to support large folios	2024-09-01 20:25:56 -07:00
truncate.c	mm: Fix missing folio invalidation calls during truncation	2024-08-24 16:09:16 +02:00
usercopy.c	mm: Fix copy_from_user_nofault().	2023-04-12 17:36:23 -07:00
userfaultfd.c	userfaultfd: move core VMA manipulation logic to mm/userfaultfd.c	2024-09-01 20:25:53 -07:00
util.c	mm: kvmalloc: align kvrealloc() with krealloc()	2024-09-01 20:25:44 -07:00
vma_internal.h	mm: move internal core VMA manipulation functions to own file	2024-09-01 20:25:54 -07:00
vma.c	mm: move internal core VMA manipulation functions to own file	2024-09-01 20:25:54 -07:00
vma.h	mm: move internal core VMA manipulation functions to own file	2024-09-01 20:25:54 -07:00
vmalloc.c	mm: vmalloc: implement vrealloc()	2024-09-01 20:25:43 -07:00
vmpressure.c	eventfd: simplify eventfd_signal()	2023-11-28 14:08:38 +01:00
vmscan.c	mm: create promo_wmark_pages and clean up open-coded sites	2024-09-01 20:25:58 -07:00
vmstat.c	mm: print the promo watermark in zoneinfo	2024-09-01 20:25:59 -07:00
workingset.c	cachestat: do not flush stats in recency check	2024-07-03 22:40:37 -07:00
z3fold.c	mm/z3fold: add __percpu annotation to *unbuddied pointer in struct z3fold_pool	2024-09-01 20:25:56 -07:00
zbud.c	mm: zpool: return pool size in pages	2024-04-25 20:55:48 -07:00
zpool.c	mm: zpool: return pool size in pages	2024-04-25 20:55:48 -07:00
zsmalloc.c	minmax: make generic MIN() and MAX() macros available everywhere	2024-07-28 15:49:18 -07:00
zswap.c	mm: zswap: fix global shrinker error handling logic	2024-09-01 20:25:58 -07:00