Kairui Song 5168a68eb7 mm, swap: avoid over reclaim of full clusters ... When running low on usable slots, cluster allocator will try to reclaim the full clusters aggressively to reclaim HAS_CACHE slots. This guarantees that as long as there are any usable slots, HAS_CACHE or not, the swap device will be usable and workload won't go OOM early. Before the cluster allocator, swap allocator fails easily if device is filled up with reclaimable HAS_CACHE slots. Which can be easily reproduced with following simple program: #include <stdio.h> #include <string.h> #include <linux/mman.h> #include <sys/mman.h> #define SIZE 8192UL * 1024UL * 1024UL int main(int argc, char **argv) { long tmp; char *p = mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); memset(p, 0, SIZE); madvise(p, SIZE, MADV_PAGEOUT); for (unsigned long i = 0; i < SIZE; ++i) tmp += p[i]; getchar(); /* Pause */ return 0; } Setup an 8G non ramdisk swap, the first run of the program will swapout 8G ram successfully. But run same program again after the first run paused, the second run can't swapout all 8G memory as now half of the swap device is pinned by HAS_CACHE. There was a random scan in the old allocator that may reclaim part of the HAS_CACHE by luck, but it's unreliable. The new allocator's added reclaim of full clusters when device is low on usable slots. But when multiple CPUs are seeing the device is low on usable slots at the same time, they ran into a thundering herd problem. This is an observable problem on large machine with mass parallel workload, as full cluster reclaim is slower on large swap device and higher number of CPUs will also make things worse. Testing using a 128G ZRAM on a 48c96t system. When the swap device is very close to full (eg. 124G / 128G), running build linux kernel with make -j96 in a 1G memory cgroup will hung (not a softlockup though) spinning in full cluster reclaim for about ~5min before go OOM. To solve this, split the full reclaim into two parts: - Instead of do a synchronous aggressively reclaim when device is low, do only one aggressively reclaim when device is strictly full with a kworker. This still ensures in worst case the device won't be unusable because of HAS_CACHE slots. - To avoid allocation (especially higher order) suffer from HAS_CACHE filling up clusters and kworker not responsive enough, do one synchronous scan every time the free list is drained, and only scan one cluster. This is kind of similar to the random reclaim before, keeps the full clusters rotated and has a minimal latency. This should provide a fair reclaim strategy suitable for most workloads. Link: https://lkml.kernel.org/r/20241022175512.10398-1-ryncsn@gmail.com Fixes: 2cacbdfdee ("mm: swap: add a adaptive full cluster cache reclaim") Signed-off-by: Kairui Song <kasong@tencent.com> Cc: Barry Song <v-songbaohua@oppo.com> Cc: Chris Li <chrisl@kernel.org> Cc: "Huang, Ying" <ying.huang@intel.com> Cc: Hugh Dickins <hughd@google.com> Cc: Kalesh Singh <kaleshsingh@google.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Yosry Ahmed <yosryahmed@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>		2024-10-30 20:14:11 -07:00
..
damon	mm/damon/tests/sysfs-kunit.h: fix memory leak in damon_sysfs_test_add_targets()	2024-10-17 00:28:08 -07:00
kasan	LoongArch: Set initial pte entry with PAGE_GLOBAL for kernel space	2024-10-21 22:11:19 +08:00
kfence	mm: kfence: fix elapsed time for allocated/freed track	2024-09-26 14:01:44 -07:00
kmsan	kmsan: do not pass NULL pointers as 0	2024-07-03 19:30:26 -07:00
backing-dev.c	writeback: support retrieving per group debug writeback stats of bdi	2024-05-05 17:53:51 -07:00
balloon_compaction.c	mm: remove MIGRATE_SYNC_NO_COPY mode	2024-07-03 19:30:00 -07:00
bootmem_info.c	bootmem: use kmemleak_free_part_phys in put_page_bootmem	2023-10-25 16:47:13 -07:00
cma_debug.c
cma_sysfs.c	mm/cma: add sysfs file 'release_pages_success'	2024-02-22 10:24:57 -08:00
cma.c	mm/cma: add cma_{alloc,free}_folio()	2024-09-03 21:15:36 -07:00
cma.h	mm/cma: add sysfs file 'release_pages_success'	2024-02-22 10:24:57 -08:00
compaction.c	mm:page_alloc: fix the NULL ac->nodemask in __alloc_pages_slowpath()	2024-09-03 21:15:47 -07:00
debug_page_alloc.c	mm: page_alloc: consolidate free page accounting	2024-04-25 20:56:04 -07:00
debug_page_ref.c
debug_vm_pgtable.c	mm/debug_vm_pgtable: Use pxdp_get() for accessing page table entries	2024-09-17 01:07:01 -07:00
debug.c	mm: support only one page_type per page	2024-09-03 21:15:43 -07:00
dmapool_test.c	mm/dmapool: add MODULE_DESCRIPTION()	2024-07-03 19:29:58 -07:00
dmapool.c	mm/mempool/dmapool: remove CONFIG_DEBUG_SLAB ifdefs	2023-12-05 11:17:58 +01:00
early_ioremap.c	mm/early_ioremap.c: improve the execution efficiency of early_ioremap_setup()	2023-06-09 16:25:56 -07:00
execmem.c	mm/execmem, arch: convert remaining overrides of module_alloc to execmem	2024-05-14 00:31:43 -07:00
fadvise.c	introduce fd_file(), convert all accessors to it.	2024-08-12 22:00:43 -04:00
fail_page_alloc.c	fault-inject: improve build for CONFIG_FAULT_INJECTION=n	2024-09-01 20:43:33 -07:00
failslab.c	fault-inject: improve build for CONFIG_FAULT_INJECTION=n	2024-09-01 20:43:33 -07:00
filemap.c	mm/filemap: fix filemap_get_folios_contig THP panic	2024-09-26 14:01:43 -07:00
folio-compat.c	mm: remove putback_lru_page()	2024-09-09 16:38:59 -07:00
gup_test.c	Merge mm-hotfixes-stable into mm-stable to pick up depended-upon changes.	2023-06-23 16:58:19 -07:00
gup_test.h
gup.c	mm/gup: stop leaking pinned pages in low memory conditions	2024-10-30 20:14:10 -07:00
highmem.c	mm/highmem: make nr_free_highpages() return "unsigned long"	2024-07-03 19:30:06 -07:00
hmm.c	mm: provide mm_struct and address to huge_ptep_get()	2024-07-12 15:52:15 -07:00
huge_memory.c	mm: huge_memory: add vma_thp_disabled() and thp_disabled_by_hw()	2024-10-17 00:28:10 -07:00
hugetlb_cgroup.c	mm: memcg: don't call propagate_protected_usage() needlessly	2024-09-01 20:25:50 -07:00
hugetlb_vmemmap.c	mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO	2024-09-01 20:25:45 -07:00
hugetlb_vmemmap.h	mm: hugetlb_vmemmap: fix reference to nonexistent file	2023-10-25 16:47:14 -07:00
hugetlb.c	mm/hugetlb: fix memfd_pin_folios resv_huge_pages leak	2024-09-26 14:01:43 -07:00
hwpoison-inject.c	mm/hwpoison: add MODULE_DESCRIPTION()	2024-07-03 19:29:58 -07:00
init-mm.c	mm: Deprecate pasid field	2023-12-12 10:11:32 +01:00
internal.h	ALong with the usual shower of singleton patches, notable patch series in	2024-09-21 07:29:05 -07:00
interval_tree.c
io-mapping.c
ioremap.c	mm: ioremap: remove unneeded ioremap_allowed and iounmap_allowed	2023-08-18 10:12:36 -07:00
Kconfig	resource: remove dependency on SPARSEMEM from GET_FREE_REGION	2024-10-28 21:40:39 -07:00
Kconfig.debug	slub: Introduce CONFIG_SLUB_RCU_DEBUG	2024-08-27 14:12:51 +02:00
khugepaged.c	mm: khugepaged: fix the incorrect statistics when collapsing large file folios	2024-10-17 00:28:10 -07:00
kmemleak.c	mm/kmemleak: use IS_ERR_PCPU() for pointer in the percpu address space	2024-09-03 21:15:38 -07:00
ksm.c	mm: remove PageSwapCache	2024-09-03 21:15:44 -07:00
list_lru.c	mm: list_lru: fix UAF for memory cgroup	2024-08-07 18:33:56 -07:00
maccess.c	mm: Fix copy_from_user_nofault().	2023-04-12 17:36:23 -07:00
madvise.c	ALong with the usual shower of singleton patches, notable patch series in	2024-09-21 07:29:05 -07:00
Makefile	mm: introduce numa_emulation	2024-09-03 21:15:31 -07:00
mapping_dirty_helpers.c	mm: fix clean_record_shared_mapping_range kernel-doc	2023-08-24 16:20:30 -07:00
memblock.c	memblock: updates for 6.12-rc1	2024-09-25 11:35:19 -07:00
memcontrol-v1.c	struct fd layout change (and conversion to accessor helpers)	2024-09-23 09:35:36 -07:00
memcontrol-v1.h	memcg: cleanup with !CONFIG_MEMCG_V1	2024-09-17 01:07:00 -07:00
memcontrol.c	ALong with the usual shower of singleton patches, notable patch series in	2024-09-21 07:29:05 -07:00
memfd.c	mm/hugetlb: simplify refs in memfd_alloc_folio	2024-09-26 14:01:44 -07:00
memory_hotplug.c	ALong with the usual shower of singleton patches, notable patch series in	2024-09-21 07:29:05 -07:00
memory-failure.c	mm: migrate: add isolate_folio_to_list()	2024-09-03 21:15:59 -07:00
memory-tiers.c	memory tiers: use default_dram_perf_ref_source in log message	2024-09-26 14:01:44 -07:00
memory.c	mm: avoid unconditional one-tick sleep when swapcache_prepare fails	2024-10-28 21:40:41 -07:00
mempolicy.c	mm,memcg: provide per-cgroup counters for NUMA balancing operations	2024-09-03 21:15:36 -07:00
mempool.c	mm: fix xyz_noprof functions calling profiled functions	2024-06-05 19:19:26 -07:00
memremap.c	mm: convert put_devmap_managed_page_refs() to put_devmap_managed_folio_refs()	2024-05-05 17:53:49 -07:00
memtest.c	memtest: use {READ,WRITE}_ONCE in memory scanning	2024-03-13 12:12:21 -07:00
migrate_device.c	mm: remap unused subpages to shared zeropage when splitting isolated thp	2024-09-09 16:39:03 -07:00
migrate.c	mm: avoid VM_BUG_ON when try to map an anon large folio to zero page.	2024-10-30 20:14:10 -07:00
mincore.c	mm: provide mm_struct and address to huge_ptep_get()	2024-07-12 15:52:15 -07:00
mlock.c	Random number generator updates for Linux 6.11-rc1.	2024-07-24 10:29:50 -07:00
mm_init.c	mm: drop CONFIG_HAVE_ARCH_NODEDATA_EXTENSION	2024-09-03 21:15:28 -07:00
mm_slot.h
mmap_lock.c	mm: mmap_lock: replace get_memcg_path_buf() with on-stack buffer	2024-07-03 19:30:26 -07:00
mmap.c	mm: split critical region in remap_file_pages() and invoke LSMs in between	2024-10-28 21:40:41 -07:00
mmu_gather.c	mm/mmu_gather: improve cond_resched() handling with large folios and expensive page freeing	2024-02-22 15:27:17 -08:00
mmu_notifier.c	mm: move internal core VMA manipulation functions to own file	2024-09-01 20:25:54 -07:00
mmzone.c	mm: improve code consistency with zonelist_* helper functions	2024-09-01 20:25:55 -07:00
mprotect.c	mm/mprotect: replace can_modify_mm with can_modify_vma	2024-09-03 21:15:41 -07:00
mremap.c	mm/mremap: fix move_normal_pmd/retract_page_tables race	2024-10-17 00:28:07 -07:00
mseal.c	ALong with the usual shower of singleton patches, notable patch series in	2024-09-21 07:29:05 -07:00
msync.c
nommu.c	mm: remove follow_page()	2024-09-01 20:26:01 -07:00
numa_emulation.c	mm: introduce numa_emulation	2024-09-03 21:15:31 -07:00
numa_memblks.c	mm: numa_clear_kernel_node_hotplug: Add NUMA_NO_NODE check for node id	2024-10-28 21:40:40 -07:00
numa.c	mm: make range-to-target_node lookup facility a part of numa_memblks	2024-09-03 21:15:32 -07:00
oom_kill.c	memory: remove the now superfluous sentinel element from ctl_table array	2024-04-25 20:56:32 -07:00
page_alloc.c	mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves	2024-10-28 21:40:39 -07:00
page_counter.c	mm, memcg: cg2 memory{.swap,}.peak write handlers	2024-09-01 20:25:53 -07:00
page_ext.c	mm: don't account memmap per-node	2024-08-15 22:16:14 -07:00
page_idle.c
page_io.c	mm: fix PSWPIN counter for large folios swap-in	2024-10-30 20:14:11 -07:00
page_isolation.c	mm: remove migration for HugePage in isolate_single_pageblock()	2024-09-03 21:15:40 -07:00
page_owner.c	mm/page-owner: use gfp_nested_mask() instead of open coded masking	2024-05-19 14:40:44 -07:00
page_poison.c	mm/page_poison: replace kmap_atomic() with kmap_local_page()	2023-12-10 16:51:50 -08:00
page_reporting.c	mm, treewide: rename MAX_ORDER to MAX_PAGE_ORDER	2024-01-08 15:27:15 -08:00
page_reporting.h
page_table_check.c	mm/page_table_check: fix crash on ZONE_DEVICE	2024-06-15 10:43:04 -07:00
page_vma_mapped.c	mm: make page_mapped_in_vma conditional on CONFIG_MEMORY_FAILURE	2024-05-05 17:53:45 -07:00
page-writeback.c	ALong with the usual shower of singleton patches, notable patch series in	2024-09-21 07:29:05 -07:00
pagewalk.c	mm/pagewalk: fix usage of pmd_leaf()/pud_leaf() without present check	2024-10-28 21:40:38 -07:00
percpu-internal.h	mm: remove CONFIG_MEMCG_KMEM	2024-07-10 12:14:54 -07:00
percpu-km.c
percpu-stats.c
percpu-vm.c	percpu: clean up all mappings when pcpu_map_pages() fails	2024-04-25 20:55:49 -07:00
percpu.c	percpu: remove pcpu_alloc_size()	2024-09-01 20:26:04 -07:00
pgalloc-track.h
pgtable-generic.c	mm: fix race between __split_huge_pmd_locked() and GUP-fast	2024-05-07 10:37:00 -07:00
process_vm_access.c	mm: fix process_vm_rw page counts	2023-12-10 16:51:39 -08:00
ptdump.c	mm: ptdump: add check_wx_pages debugfs attribute	2024-02-22 10:24:47 -08:00
readahead.c	struct fd layout change (and conversion to accessor helpers)	2024-09-23 09:35:36 -07:00
rmap.c	mm: introduce a pageflag for partially mapped folios	2024-09-09 16:39:04 -07:00
rodata_test.c
secretmem.c	secretmem: disable memfd_secret() if arch cannot set direct map	2024-10-09 12:47:19 -07:00
shmem_quota.c	shmem_quota: build the object file conditionally to the config option	2024-09-01 20:25:45 -07:00
shmem.c	mm: shmem: fix data-race in shmem_getattr()	2024-10-28 21:40:39 -07:00
show_mem.c	mm/show_mem.c: report alloc tags in human readable units	2024-09-17 01:07:00 -07:00
shrinker_debug.c	mm: shrinker: use min() to improve shrinker_debugfs_scan_write()	2024-09-03 21:15:40 -07:00
shrinker.c	mm: shrinker: use kvzalloc_node() from expand_one_shrinker_info()	2024-01-05 09:58:32 -08:00
shuffle.c
shuffle.h	mm, treewide: rename MAX_ORDER to MAX_PAGE_ORDER	2024-01-08 15:27:15 -08:00
slab_common.c	mm, slab: suppress warnings in test_leak_destroy kunit test	2024-10-02 16:28:46 +02:00
slab.h	mm, slab: suppress warnings in test_leak_destroy kunit test	2024-10-02 16:28:46 +02:00
slub.c	mm, slab: suppress warnings in test_leak_destroy kunit test	2024-10-02 16:28:46 +02:00
sparse-vmemmap.c	LoongArch: Set initial pte entry with PAGE_GLOBAL for kernel space	2024-10-21 22:11:19 +08:00
sparse.c	A set of X86 fixes:	2024-09-01 14:43:08 -07:00
swap_cgroup.c	mm: attempt to batch free swap entries for zap_pte_range()	2024-09-03 21:15:33 -07:00
swap_slots.c	mm: swap: update get_swap_pages() to take folio order	2024-04-25 20:56:37 -07:00
swap_state.c	mm: add nr argument in mem_cgroup_swapin_uncharge_swap() helper to support large folios	2024-09-17 01:07:01 -07:00
swap.c	mm: remove isolate_lru_page()	2024-09-09 16:38:59 -07:00
swap.h	mm: fix swap_read_folio_zeromap() for large folios with partial zeromap	2024-09-17 01:07:01 -07:00
swapfile.c	mm, swap: avoid over reclaim of full clusters	2024-10-30 20:14:11 -07:00
truncate.c	mm: Fix missing folio invalidation calls during truncation	2024-08-24 16:09:16 +02:00
usercopy.c	mm: Fix copy_from_user_nofault().	2023-04-12 17:36:23 -07:00
userfaultfd.c	ALong with the usual shower of singleton patches, notable patch series in	2024-09-21 07:29:05 -07:00
util.c	mm: only enforce minimum stack gap size if it's sensible	2024-09-01 20:26:02 -07:00
vma_internal.h	mm: remove duplicated include in vma_internal.h	2024-09-01 20:26:02 -07:00
vma.c	mm/vma: add expand-only VMA merge mode and optimise do_brk_flags()	2024-10-28 21:40:40 -07:00
vma.h	mm/vma: add expand-only VMA merge mode and optimise do_brk_flags()	2024-10-28 21:40:40 -07:00
vmalloc.c	ALong with the usual shower of singleton patches, notable patch series in	2024-09-21 07:29:05 -07:00
vmpressure.c	eventfd: simplify eventfd_signal()	2023-11-28 14:08:38 +01:00
vmscan.c	mm/mglru: only clear kswapd_failures if reclaimable	2024-10-17 00:28:12 -07:00
vmstat.c	mm: split underused THPs	2024-09-09 16:39:04 -07:00
workingset.c	cachestat: do not flush stats in recency check	2024-07-03 22:40:37 -07:00
z3fold.c	mm/z3fold: add __percpu annotation to *unbuddied pointer in struct z3fold_pool	2024-09-01 20:25:56 -07:00
zbud.c	mm: zpool: return pool size in pages	2024-04-25 20:55:48 -07:00
zpool.c	mm: zpool: return pool size in pages	2024-04-25 20:55:48 -07:00
zsmalloc.c	ALong with the usual shower of singleton patches, notable patch series in	2024-09-21 07:29:05 -07:00
zswap.c	mm: zswap: delete comments for "value" member of 'struct zswap_entry'.	2024-10-09 12:47:19 -07:00