Thanks for looking into this problem.
On Sat, 2 May 2020 at 02:28, Andrew Morton akpm@linux-foundation.org wrote:
On Fri, 1 May 2020 18:08:28 +0530 Naresh Kamboju naresh.kamboju@linaro.org wrote:
mkfs -t ext4 invoked oom-killer on i386 kernel running on x86_64 device and started happening on linux -next master branch kernel tag next-20200430 and next-20200501. We did not bisect this problem.
It would be wonderful if you could do so, please. I can't immediately see any MM change in this area which might cause this.
We are planning a bisection soon on this problem.
metadata git branch: master git repo: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git git commit: e4a08b64261ab411b15580c369a3b8fbed28bbc1 git describe: next-20200430 make_kernelversion: 5.7.0-rc3 kernel-config: https://builds.tuxbuild.com/1YrE_XUQ6odA52tSBM919w/kernel.config
Steps to reproduce: (always reproducible)
Reproducibility helps!
oom-killer: gfp_mask=0x101cc0(GFP_USER|__GFP_WRITE), order=0,
[ 34.793430] pagecache_get_page+0xae/0x260
[ 34.897923] active_anon:5366 inactive_anon:2172 isolated_anon:0 [ 34.897923] active_file:4151 inactive_file:212494 isolated_file:0 [ 34.897923] unevictable:0 dirty:16505 writeback:6520 unstable:0
[ 34.987678] Normal free:3948kB min:7732kB low:8640kB high:9548kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:1096kB inactive_file:786400kB unevictable:0kB writepending:65432kB present:884728kB managed:845576kB mlocked:0kB kernel_stack:1112kB pagetables:0kB bounce:0kB free_pcp:2908kB local_pcp:500kB free_cma:0kB
ZONE_NORMAL has a huge amount of clean pagecache stuck on the inactive list, not being reclaimed.
FYI, This issue is already reported here. Now this problem is happening and easily reproducible on i386 and arm beagleboard x15 devices.
mkfs -t ext4 /dev/disk/by-id/ata-SanDisk_SSD_PLUS_120GB_190703A01414 mke2fs 1.43.8 (1-Jan-2018) Discarding device blocks: 4096/29306880 2625536/29306880 9441280/29306880 16257024/29306880 23072768/29306880 done Creating filesystem with 29306880 4k blocks and 7331840 inodes Filesystem UUID: a838d994-0a1e-403a-88d5-444d75aecc5a Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872 Allocating group tables: 0/895 done Writing inode tables: 0/895 done Creating journal (131072 blocks): [ 31.251333] mkfs.ext4 invoked oom-killer: gfp_mask=0x101cc0(GFP_USER|__GFP_WRITE), order=0, oom_score_adj=0 [ 31.261172] CPU: 0 PID: 397 Comm: mkfs.ext4 Not tainted 5.7.0-rc6-next-20200518 #1 [ 31.268771] Hardware name: Generic DRA74X (Flattened Device Tree) [ 31.274904] [<c0411500>] (unwind_backtrace) from [<c040b66c>] (show_stack+0x10/0x14) [ 31.282685] [<c040b66c>] (show_stack) from [<c08b1b14>] (dump_stack+0xc4/0xd8) [ 31.289940] [<c08b1b14>] (dump_stack) from [<c0547bf8>] (dump_header+0x54/0x1ec) [ 31.297367] [<c0547bf8>] (dump_header) from [<c0547008>] (oom_kill_process+0x18c/0x198) [ 31.305405] [<c0547008>] (oom_kill_process) from [<c0547a0c>] (out_of_memory+0x250/0x368) [ 31.313619] [<c0547a0c>] (out_of_memory) from [<c0599d80>] (__alloc_pages_nodemask+0xce8/0x10bc) [ 31.322445] [<c0599d80>] (__alloc_pages_nodemask) from [<c0541bb4>] (pagecache_get_page+0x128/0x358) [ 31.331619] [<c0541bb4>] (pagecache_get_page) from [<c0543a8c>] (grab_cache_page_write_begin+0x18/0x2c) [ 31.341054] [<c0543a8c>] (grab_cache_page_write_begin) from [<c0619fb0>] (block_write_begin+0x20/0xc4) [ 31.350401] [<c0619fb0>] (block_write_begin) from [<c053e718>] (generic_perform_write+0xb8/0x1d8) [ 31.359312] [<c053e718>] (generic_perform_write) from [<c054496c>] (__generic_file_write_iter+0x164/0x1ec) [ 31.369007] [<c054496c>] (__generic_file_write_iter) from [<c061c8a4>] (blkdev_write_iter+0xc8/0x1a4) [ 31.378269] [<c061c8a4>] (blkdev_write_iter) from [<c05d50d0>] (__vfs_write+0x13c/0x1cc) [ 31.386397] [<c05d50d0>] (__vfs_write) from [<c05d81d4>] (vfs_write+0xb0/0x1bc) [ 31.393738] [<c05d81d4>] (vfs_write) from [<c05d85e4>] (ksys_pwrite64+0x60/0x8c) [ 31.401167] [<c05d85e4>] (ksys_pwrite64) from [<c04001a0>] (ret_fast_syscall+0x0/0x4c) [ 31.409115] Exception stack(0xe810dfa8 to 0xe810dff0) [ 31.414185] dfa0: a2000000 0000000d 00000003 b6952008 00400000 00000000 [ 31.422395] dfc0: a2000000 0000000d a2000000 000000b5 00400000 0003b768 b6952008 00da2000 [ 31.430604] dfe0: 00000064 beb891b8 b6f85108 b6e38f2c [ 31.435809] Mem-Info: [ 31.438098] active_anon:5813 inactive_anon:4129 isolated_anon:0 [ 31.438098] active_file:6080 inactive_file:118548 isolated_file:0 [ 31.438098] unevictable:0 dirty:13674 writeback:7440 unstable:0 [ 31.438098] slab_reclaimable:5651 slab_unreclaimable:4566 [ 31.438098] mapped:5585 shmem:4468 pagetables:182 bounce:0 [ 31.438098] free:347556 free_pcp:608 free_cma:57235 [ 31.472362] Node 0 active_anon:23252kB inactive_anon:16516kB active_file:24320kB inactive_file:474192kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:22340kB dirty:54696kB writeback:11196kB shmem:17872kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes [ 31.500943] DMA free:187396kB min:22528kB low:28160kB high:33792kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:4736kB inactive_file:431688kB unevictable:0kB writepending:62020kB present:783360kB managed:668264kB mlocked:0kB kernel_stack:888kB pagetables:0kB bounce:0kB free_pcp:880kB local_pcp:216kB free_cma:163840kB [ 31.531339] lowmem_reserve[]: 0 0 1216 0 [ 31.535289] HighMem free:1203904kB min:512kB low:11592kB high:22672kB reserved_highatomic:0KB active_anon:23252kB inactive_anon:16516kB active_file:19584kB inactive_file:42420kB unevictable:0kB writepending:0kB present:1310720kB managed:1310720kB mlocked:0kB kernel_stack:0kB pagetables:728kB bounce:0kB free_pcp:1584kB local_pcp:1232kB free_cma:65100kB [ 31.566540] lowmem_reserve[]: 0 0 0 0 [ 31.570244] DMA: 87*4kB (UME) 53*8kB (UME) 26*16kB (UE) 6*32kB (UM) 1*64kB (E) 1*128kB (U) 5*256kB (ME) 5*512kB (ME) 4*1024kB (ME) 5*2048kB (M) 1*4096kB (M) 20*8192kB (C) = 187684kB [ 31.586520] HighMem: 2*4kB (MC) 1*8kB (C) 1*16kB (M) 5*32kB (UM) 4*64kB (UMC) 2*128kB (UM) 2*256kB (UM) 1*512kB (C) 2*1024kB (MC) 2*2048kB (MC) 2*4096kB (UC) 145*8192kB (MC) = 1203904kB [ 31.603150] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 31.611637] 129102 total pagecache pages [ 31.615577] 0 pages in swap cache [ 31.618902] Swap cache stats: add 0, delete 0, find 0/0 [ 31.624162] Free swap = 0kB [ 31.627053] Total swap = 0kB [ 31.629955] 523520 pages RAM [ 31.632846] 327680 pages HighMem/MovableOnly [ 31.637128] 28774 pages reserved [ 31.640381] 57344 pages cma reserved [ 31.643971] Tasks state (memory values in pages): [ 31.648691] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 31.657367] [ 183] 0 183 7370 1082 36864 0 0 systemd-journal [ 31.666466] [ 209] 994 209 3742 326 40960 0 0 systemd-timesyn [ 31.675570] [ 217] 0 217 3398 817 32768 0 -1000 systemd-udevd [ 31.684498] [ 230] 993 230 1411 737 32768 0 0 systemd-network [ 31.693598] [ 231] 992 231 1496 712 32768 0 0 systemd-resolve [ 31.702702] [ 236] 996 236 1112 742 24576 0 -900 dbus-daemon [ 31.711454] [ 241] 0 241 1895 1045 36864 0 0 haveged [ 31.719857] [ 242] 0 242 1362 906 28672 0 0 systemd-logind [ 31.728855] [ 243] 0 243 13412 2571 69632 0 0 NetworkManager [ 31.737867] [ 244] 995 244 1197 608 28672 0 0 avahi-daemon [ 31.746707] [ 245] 995 245 1164 59 28672 0 0 avahi-daemon [ 31.755545] [ 246] 0 246 594 332 28672 0 0 atd [ 31.763601] [ 248] 0 248 699 99 24576 0 0 syslogd [ 31.772001] [ 251] 0 251 699 102 24576 0 0 klogd [ 31.780231] [ 252] 0 252 676 365 24576 0 0 crond [ 31.788443] [ 254] 0 254 1172 240 32768 0 0 systemd-hostnam [ 31.797547] [ 264] 65534 264 605 32 24576 0 0 dnsmasq [ 31.805948] [ 265] 0 265 556 357 28672 0 0 agetty [ 31.814262] [ 266] 0 266 1131 613 32768 0 0 login [ 31.822492] [ 268] 998 268 18201 2629 81920 0 0 polkitd [ 31.830895] [ 350] 0 350 1840 1161 32768 0 0 systemd [ 31.839286] [ 351] 0 351 2403 473 36864 0 0 (sd-pam) [ 31.847774] [ 355] 0 355 827 611 24576 0 0 sh [ 31.855742] [ 364] 0 364 7341 1145 53248 0 0 nm-dispatcher [ 31.864667] [ 377] 0 377 711 510 28672 0 0 lava-test-runne [ 31.873770] [ 387] 0 387 711 138 20480 0 0 lava-test-shell [ 31.882869] [ 388] 0 388 711 523 20480 0 0 sh [ 31.890837] [ 397] 0 397 1785 1518 36864 0 0 mkfs.ext4 [ 31.899397] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),global_oom,task_memcg=/,task=polkitd,pid=268,uid=998 [ 31.910012] Out of memory: Killed process 268 (polkitd) total-vm:72804kB, anon-rss:2948kB, file-rss:7568kB, shmem-rss:0kB, UID:998 pgtables:80kB oom_score_adj:0 [ 31.927948] oom_reaper: reaped process 268 (polkitd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [ 31.937461] mkfs.ext4 invoked oom-killer: gfp_mask=0x101cc0(GFP_USER|__GFP_WRITE), order=0, oom_score_adj=0 [ 31.947273] CPU: 1 PID: 397 Comm: mkfs.ext4 Not tainted 5.7.0-rc6-next-20200518 #1 [ 31.954871] Hardware name: Generic DRA74X (Flattened Device Tree) [ 31.961000] [<c0411500>] (unwind_backtrace) from [<c040b66c>] (show_stack+0x10/0x14) [ 31.968778] [<c040b66c>] (show_stack) from [<c08b1b14>] (dump_stack+0xc4/0xd8) [ 31.976032] [<c08b1b14>] (dump_stack) from [<c0547bf8>] (dump_header+0x54/0x1ec) [ 31.983458] [<c0547bf8>] (dump_header) from [<c0547008>] (oom_kill_process+0x18c/0x198) [ 31.991495] [<c0547008>] (oom_kill_process) from [<c0547a0c>] (out_of_memory+0x250/0x368) [ 31.999706] [<c0547a0c>] (out_of_memory) from [<c0599d80>] (__alloc_pages_nodemask+0xce8/0x10bc) [ 32.008532] [<c0599d80>] (__alloc_pages_nodemask) from [<c0541bb4>] (pagecache_get_page+0x128/0x358) [ 32.017704] [<c0541bb4>] (pagecache_get_page) from [<c0543a8c>] (grab_cache_page_write_begin+0x18/0x2c) [ 32.027138] [<c0543a8c>] (grab_cache_page_write_begin) from [<c0619fb0>] (block_write_begin+0x20/0xc4) [ 32.036484] [<c0619fb0>] (block_write_begin) from [<c053e718>] (generic_perform_write+0xb8/0x1d8) [ 32.045395] [<c053e718>] (generic_perform_write) from [<c054496c>] (__generic_file_write_iter+0x164/0x1ec) [ 32.055090] [<c054496c>] (__generic_file_write_iter) from [<c061c8a4>] (blkdev_write_iter+0xc8/0x1a4) [ 32.064350] [<c061c8a4>] (blkdev_write_iter) from [<c05d50d0>] (__vfs_write+0x13c/0x1cc) [ 32.072476] [<c05d50d0>] (__vfs_write) from [<c05d81d4>] (vfs_write+0xb0/0x1bc) [ 32.079814] [<c05d81d4>] (vfs_write) from [<c05d85e4>] (ksys_pwrite64+0x60/0x8c) [ 32.087241] [<c05d85e4>] (ksys_pwrite64) from [<c04001a0>] (ret_fast_syscall+0x0/0x4c) [ 32.095187] Exception stack(0xe810dfa8 to 0xe810dff0) [ 32.100256] dfa0: a2000000 0000000d 00000003 b6952008 00400000 00000000 [ 32.108466] dfc0: a2000000 0000000d a2000000 000000b5 00400000 0003b768 b6952008 00da2000 [ 32.116673] dfe0: 00000064 beb891b8 b6f85108 b6e38f2c [ 32.121786] Mem-Info: [ 32.124070] active_anon:5056 inactive_anon:4129 isolated_anon:0 [ 32.124070] active_file:6289 inactive_file:118790 isolated_file:0 [ 32.124070] unevictable:0 dirty:14118 writeback:6 unstable:0 [ 32.124070] slab_reclaimable:5653 slab_unreclaimable:4209 [ 32.124070] mapped:4839 shmem:4468 pagetables:165 bounce:0 [ 32.124070] free:348249 free_pcp:562 free_cma:57235 [ 32.158031] Node 0 active_anon:20224kB inactive_anon:16516kB active_file:25156kB inactive_file:475160kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:19356kB dirty:56472kB writeback:24kB shmem:17872kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes [ 32.186324] DMA free:186320kB min:22528kB low:28160kB high:33792kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:4736kB inactive_file:433580kB unevictable:0kB writepending:56468kB present:783360kB managed:668264kB mlocked:0kB kernel_stack:888kB pagetables:0kB bounce:0kB free_pcp:420kB local_pcp:220kB free_cma:163840kB [ 32.216693] lowmem_reserve[]: 0 0 1216 0 [ 32.220652] HighMem free:1206676kB min:512kB low:11592kB high:22672kB reserved_highatomic:0KB active_anon:20224kB inactive_anon:16516kB active_file:20420kB inactive_file:41584kB unevictable:0kB writepending:0kB present:1310720kB managed:1310720kB mlocked:0kB kernel_stack:0kB pagetables:660kB bounce:0kB free_pcp:1816kB local_pcp:340kB free_cma:65100kB [ 32.251805] lowmem_reserve[]: 0 0 0 0 [ 32.255482] DMA: 2*4kB (UM) 3*8kB (UME) 1*16kB (U) 1*32kB (M) 0*64kB 1*128kB (U) 5*256kB (ME) 5*512kB (ME) 4*1024kB (ME) 5*2048kB (M) 1*4096kB (M) 20*8192kB (C) = 186320kB [ 32.270871] HighMem: 183*4kB (UMC) 65*8kB (UMC) 21*16kB (M) 11*32kB (UM) 6*64kB (UMC) 3*128kB (UM) 3*256kB (UM) 2*512kB (MC) 2*1024kB (MC) 2*2048kB (MC) 2*4096kB (UC) 145*8192kB (MC) = 1206676kB [ 32.288273] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 32.296751] 129546 total pagecache pages [ 32.300695] 0 pages in swap cache [ 32.304019] Swap cache stats: add 0, delete 0, find 0/0 [ 32.309260] Free swap = 0kB [ 32.312155] Total swap = 0kB [ 32.315045] 523520 pages RAM [ 32.317932] 327680 pages HighMem/MovableOnly [ 32.322221] 28774 pages reserved [ 32.325457] 57344 pages cma reserved [ 32.329043] Tasks state (memory values in pages): [ 32.333771] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 32.342436] [ 183] 0 183 7370 1082 36864 0 0 systemd-journal [ 32.351529] [ 209] 994 209 3742 326 40960 0 0 systemd-timesyn [ 32.360620] [ 217] 0 217 3398 817 32768 0 -1000 systemd-udevd [ 32.369528] [ 230] 993 230 1411 737 32768 0 0 systemd-network [ 32.378620] [ 231] 992 231 1496 712 32768 0 0 systemd-resolve [ 32.387713] [ 236] 996 236 1112 742 24576 0 -900 dbus-daemon [ 32.396456] [ 241] 0 241 1895 1045 36864 0 0 haveged [ 32.404850] [ 242] 0 242 1362 906 28672 0 0 systemd-logind [ 32.413852] [ 243] 0 243 13412 2571 69632 0 0 NetworkManager [ 32.422858] [ 244] 995 244 1197 608 28672 0 0 avahi-daemon [ 32.431687] [ 245] 995 245 1164 59 28672 0 0 avahi-daemon [ 32.440518] [ 246] 0 246 594 332 28672 0 0 atd [ 32.448553] [ 248] 0 248 699 99 24576 0 0 syslogd [ 32.456945] [ 251] 0 251 699 102 24576 0 0 klogd [ 32.465171] [ 252] 0 252 676 365 24576 0 0 crond [ 32.473390] [ 254] 0 254 1172 240 32768 0 0 systemd-hostnam [ 32.482481] [ 264] 65534 264 605 32 24576 0 0 dnsmasq [ 32.490876] [ 265] 0 265 556 357 28672 0 0 agetty [ 32.499175] [ 266] 0 266 1131 613 32768 0 0 login [ 32.507394] [ 350] 0 350 1840 1161 32768 0 0 systemd [ 32.515788] [ 351] 0 351 2403 473 36864 0 0 (sd-pam) [ 32.524268] [ 355] 0 355 827 611 24576 0 0 sh [ 32.532227] [ 364] 0 364 7341 1145 53248 0 0 nm-dispatcher [ 32.541142] [ 377] 0 377 711 510 28672 0 0 lava-test-runne [ 32.550234] [ 387] 0 387 711 138 20480 0 0 lava-test-shell [ 32.559316] [ 388] 0 388 711 523 20480 0 0 sh [ 32.567273] [ 397] 0 397 1785 1518 36864 0 0 mkfs.ext4
ref: https://lkft.validation.linaro.org/scheduler/job/1436647#L4261 https://lkft.validation.linaro.org/scheduler/job/1436562#L1247
-- Linaro LKFT https://lkft.linaro.org