| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Mon, 18 May 2020 19:40:55 +0530
From: Naresh Kamboju <naresh.kamboju@...aro.org>
To: linux-f2fs-devel@...ts.sourceforge.net,
linux-ext4 <linux-ext4@...r.kernel.org>,
linux-block <linux-block@...r.kernel.org>,
Andrew Morton <akpm@...ux-foundation.org>
Cc: open list <linux-kernel@...r.kernel.org>,
Linux-Next Mailing List <linux-next@...r.kernel.org>,
linux-mm <linux-mm@...ck.org>,
Andreas Dilger <adilger.kernel@...ger.ca>,
Jaegeuk Kim <jaegeuk@...nel.org>,
"Theodore Ts'o" <tytso@....edu>, Chao Yu <chao@...nel.org>,
Arnd Bergmann <arnd@...db.de>, Hugh Dickins <hughd@...gle.com>,
Andrea Arcangeli <aarcange@...hat.com>,
Matthew Wilcox <willy@...radead.org>,
Chao Yu <yuchao0@...wei.com>, lkft-triage@...ts.linaro.org
Subject: Re: mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page
Thanks for looking into this problem.
On Sat, 2 May 2020 at 02:28, Andrew Morton <akpm@...ux-foundation.org> wrote:
>
> On Fri, 1 May 2020 18:08:28 +0530 Naresh Kamboju <naresh.kamboju@...aro.org> wrote:
>
> > mkfs -t ext4 invoked oom-killer on i386 kernel running on x86_64 device
> > and started happening on linux -next master branch kernel tag next-20200430
> > and next-20200501. We did not bisect this problem.
>
> It would be wonderful if you could do so, please. I can't immediately see
> any MM change in this area which might cause this.
We are planning a bisection soon on this problem.
>
> > metadata
> > git branch: master
> > git repo: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
> > git commit: e4a08b64261ab411b15580c369a3b8fbed28bbc1
> > git describe: next-20200430
> > make_kernelversion: 5.7.0-rc3
> > kernel-config:
> > https://builds.tuxbuild.com/1YrE_XUQ6odA52tSBM919w/kernel.config
> >
> > Steps to reproduce: (always reproducible)
>
> Reproducibility helps!
>
> > oom-killer: gfp_mask=0x101cc0(GFP_USER|__GFP_WRITE), order=0,
>
> > [ 34.793430] pagecache_get_page+0xae/0x260
>
> > [ 34.897923] active_anon:5366 inactive_anon:2172 isolated_anon:0
> > [ 34.897923] active_file:4151 inactive_file:212494 isolated_file:0
> > [ 34.897923] unevictable:0 dirty:16505 writeback:6520 unstable:0
>
> > [ 34.987678] Normal free:3948kB min:7732kB low:8640kB high:9548kB
> > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB
> > active_file:1096kB inactive_file:786400kB unevictable:0kB
> > writepending:65432kB present:884728kB managed:845576kB mlocked:0kB
> > kernel_stack:1112kB pagetables:0kB bounce:0kB free_pcp:2908kB
> > local_pcp:500kB free_cma:0kB
>
> ZONE_NORMAL has a huge amount of clean pagecache stuck on the
> inactive list, not being reclaimed.
FYI,
This issue is already reported here.
Now this problem is happening and easily reproducible on i386
and arm beagleboard x15 devices.
mkfs -t ext4 /dev/disk/by-id/ata-SanDisk_SSD_PLUS_120GB_190703A01414
mke2fs 1.43.8 (1-Jan-2018)
Discarding device blocks: 4096/29306880
2625536/29306880
9441280/29306880 16257024/29306880
23072768/29306880
done
Creating filesystem with 29306880 4k blocks and 7331840 inodes
Filesystem UUID: a838d994-0a1e-403a-88d5-444d75aecc5a
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: 0/895 done
Writing inode tables: 0/895 done
Creating journal (131072 blocks): [ 31.251333] mkfs.ext4 invoked
oom-killer: gfp_mask=0x101cc0(GFP_USER|__GFP_WRITE), order=0,
oom_score_adj=0
[ 31.261172] CPU: 0 PID: 397 Comm: mkfs.ext4 Not tainted
5.7.0-rc6-next-20200518 #1
[ 31.268771] Hardware name: Generic DRA74X (Flattened Device Tree)
[ 31.274904] [<c0411500>] (unwind_backtrace) from [<c040b66c>]
(show_stack+0x10/0x14)
[ 31.282685] [<c040b66c>] (show_stack) from [<c08b1b14>]
(dump_stack+0xc4/0xd8)
[ 31.289940] [<c08b1b14>] (dump_stack) from [<c0547bf8>]
(dump_header+0x54/0x1ec)
[ 31.297367] [<c0547bf8>] (dump_header) from [<c0547008>]
(oom_kill_process+0x18c/0x198)
[ 31.305405] [<c0547008>] (oom_kill_process) from [<c0547a0c>]
(out_of_memory+0x250/0x368)
[ 31.313619] [<c0547a0c>] (out_of_memory) from [<c0599d80>]
(__alloc_pages_nodemask+0xce8/0x10bc)
[ 31.322445] [<c0599d80>] (__alloc_pages_nodemask) from [<c0541bb4>]
(pagecache_get_page+0x128/0x358)
[ 31.331619] [<c0541bb4>] (pagecache_get_page) from [<c0543a8c>]
(grab_cache_page_write_begin+0x18/0x2c)
[ 31.341054] [<c0543a8c>] (grab_cache_page_write_begin) from
[<c0619fb0>] (block_write_begin+0x20/0xc4)
[ 31.350401] [<c0619fb0>] (block_write_begin) from [<c053e718>]
(generic_perform_write+0xb8/0x1d8)
[ 31.359312] [<c053e718>] (generic_perform_write) from [<c054496c>]
(__generic_file_write_iter+0x164/0x1ec)
[ 31.369007] [<c054496c>] (__generic_file_write_iter) from
[<c061c8a4>] (blkdev_write_iter+0xc8/0x1a4)
[ 31.378269] [<c061c8a4>] (blkdev_write_iter) from [<c05d50d0>]
(__vfs_write+0x13c/0x1cc)
[ 31.386397] [<c05d50d0>] (__vfs_write) from [<c05d81d4>]
(vfs_write+0xb0/0x1bc)
[ 31.393738] [<c05d81d4>] (vfs_write) from [<c05d85e4>]
(ksys_pwrite64+0x60/0x8c)
[ 31.401167] [<c05d85e4>] (ksys_pwrite64) from [<c04001a0>]
(ret_fast_syscall+0x0/0x4c)
[ 31.409115] Exception stack(0xe810dfa8 to 0xe810dff0)
[ 31.414185] dfa0: a2000000 0000000d 00000003
b6952008 00400000 00000000
[ 31.422395] dfc0: a2000000 0000000d a2000000 000000b5 00400000
0003b768 b6952008 00da2000
[ 31.430604] dfe0: 00000064 beb891b8 b6f85108 b6e38f2c
[ 31.435809] Mem-Info:
[ 31.438098] active_anon:5813 inactive_anon:4129 isolated_anon:0
[ 31.438098] active_file:6080 inactive_file:118548 isolated_file:0
[ 31.438098] unevictable:0 dirty:13674 writeback:7440 unstable:0
[ 31.438098] slab_reclaimable:5651 slab_unreclaimable:4566
[ 31.438098] mapped:5585 shmem:4468 pagetables:182 bounce:0
[ 31.438098] free:347556 free_pcp:608 free_cma:57235
[ 31.472362] Node 0 active_anon:23252kB inactive_anon:16516kB
active_file:24320kB inactive_file:474192kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB mapped:22340kB dirty:54696kB
writeback:11196kB shmem:17872kB shmem_thp: 0kB shmem_pmdmapped: 0kB
anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[ 31.500943] DMA free:187396kB min:22528kB low:28160kB high:33792kB
reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB
active_file:4736kB inactive_file:431688kB unevictable:0kB
writepending:62020kB present:783360kB managed:668264kB mlocked:0kB
kernel_stack:888kB pagetables:0kB bounce:0kB free_pcp:880kB
local_pcp:216kB free_cma:163840kB
[ 31.531339] lowmem_reserve[]: 0 0 1216 0
[ 31.535289] HighMem free:1203904kB min:512kB low:11592kB
high:22672kB reserved_highatomic:0KB active_anon:23252kB
inactive_anon:16516kB active_file:19584kB inactive_file:42420kB
unevictable:0kB writepending:0kB present:1310720kB managed:1310720kB
mlocked:0kB kernel_stack:0kB pagetables:728kB bounce:0kB
free_pcp:1584kB local_pcp:1232kB free_cma:65100kB
[ 31.566540] lowmem_reserve[]: 0 0 0 0
[ 31.570244] DMA: 87*4kB (UME) 53*8kB (UME) 26*16kB (UE) 6*32kB (UM)
1*64kB (E) 1*128kB (U) 5*256kB (ME) 5*512kB (ME) 4*1024kB (ME)
5*2048kB (M) 1*4096kB (M) 20*8192kB (C) = 187684kB
[ 31.586520] HighMem: 2*4kB (MC) 1*8kB (C) 1*16kB (M) 5*32kB (UM)
4*64kB (UMC) 2*128kB (UM) 2*256kB (UM) 1*512kB (C) 2*1024kB (MC)
2*2048kB (MC) 2*4096kB (UC) 145*8192kB (MC) = 1203904kB
[ 31.603150] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[ 31.611637] 129102 total pagecache pages
[ 31.615577] 0 pages in swap cache
[ 31.618902] Swap cache stats: add 0, delete 0, find 0/0
[ 31.624162] Free swap = 0kB
[ 31.627053] Total swap = 0kB
[ 31.629955] 523520 pages RAM
[ 31.632846] 327680 pages HighMem/MovableOnly
[ 31.637128] 28774 pages reserved
[ 31.640381] 57344 pages cma reserved
[ 31.643971] Tasks state (memory values in pages):
[ 31.648691] [ pid ] uid tgid total_vm rss pgtables_bytes
swapents oom_score_adj name
[ 31.657367] [ 183] 0 183 7370 1082 36864
0 0 systemd-journal
[ 31.666466] [ 209] 994 209 3742 326 40960
0 0 systemd-timesyn
[ 31.675570] [ 217] 0 217 3398 817 32768
0 -1000 systemd-udevd
[ 31.684498] [ 230] 993 230 1411 737 32768
0 0 systemd-network
[ 31.693598] [ 231] 992 231 1496 712 32768
0 0 systemd-resolve
[ 31.702702] [ 236] 996 236 1112 742 24576
0 -900 dbus-daemon
[ 31.711454] [ 241] 0 241 1895 1045 36864
0 0 haveged
[ 31.719857] [ 242] 0 242 1362 906 28672
0 0 systemd-logind
[ 31.728855] [ 243] 0 243 13412 2571 69632
0 0 NetworkManager
[ 31.737867] [ 244] 995 244 1197 608 28672
0 0 avahi-daemon
[ 31.746707] [ 245] 995 245 1164 59 28672
0 0 avahi-daemon
[ 31.755545] [ 246] 0 246 594 332 28672
0 0 atd
[ 31.763601] [ 248] 0 248 699 99 24576
0 0 syslogd
[ 31.772001] [ 251] 0 251 699 102 24576
0 0 klogd
[ 31.780231] [ 252] 0 252 676 365 24576
0 0 crond
[ 31.788443] [ 254] 0 254 1172 240 32768
0 0 systemd-hostnam
[ 31.797547] [ 264] 65534 264 605 32 24576
0 0 dnsmasq
[ 31.805948] [ 265] 0 265 556 357 28672
0 0 agetty
[ 31.814262] [ 266] 0 266 1131 613 32768
0 0 login
[ 31.822492] [ 268] 998 268 18201 2629 81920
0 0 polkitd
[ 31.830895] [ 350] 0 350 1840 1161 32768
0 0 systemd
[ 31.839286] [ 351] 0 351 2403 473 36864
0 0 (sd-pam)
[ 31.847774] [ 355] 0 355 827 611 24576
0 0 sh
[ 31.855742] [ 364] 0 364 7341 1145 53248
0 0 nm-dispatcher
[ 31.864667] [ 377] 0 377 711 510 28672
0 0 lava-test-runne
[ 31.873770] [ 387] 0 387 711 138 20480
0 0 lava-test-shell
[ 31.882869] [ 388] 0 388 711 523 20480
0 0 sh
[ 31.890837] [ 397] 0 397 1785 1518 36864
0 0 mkfs.ext4
[ 31.899397] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),global_oom,task_memcg=/,task=polkitd,pid=268,uid=998
[ 31.910012] Out of memory: Killed process 268 (polkitd)
total-vm:72804kB, anon-rss:2948kB, file-rss:7568kB, shmem-rss:0kB,
UID:998 pgtables:80kB oom_score_adj:0
[ 31.927948] oom_reaper: reaped process 268 (polkitd), now
anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 31.937461] mkfs.ext4 invoked oom-killer:
gfp_mask=0x101cc0(GFP_USER|__GFP_WRITE), order=0, oom_score_adj=0
[ 31.947273] CPU: 1 PID: 397 Comm: mkfs.ext4 Not tainted
5.7.0-rc6-next-20200518 #1
[ 31.954871] Hardware name: Generic DRA74X (Flattened Device Tree)
[ 31.961000] [<c0411500>] (unwind_backtrace) from [<c040b66c>]
(show_stack+0x10/0x14)
[ 31.968778] [<c040b66c>] (show_stack) from [<c08b1b14>]
(dump_stack+0xc4/0xd8)
[ 31.976032] [<c08b1b14>] (dump_stack) from [<c0547bf8>]
(dump_header+0x54/0x1ec)
[ 31.983458] [<c0547bf8>] (dump_header) from [<c0547008>]
(oom_kill_process+0x18c/0x198)
[ 31.991495] [<c0547008>] (oom_kill_process) from [<c0547a0c>]
(out_of_memory+0x250/0x368)
[ 31.999706] [<c0547a0c>] (out_of_memory) from [<c0599d80>]
(__alloc_pages_nodemask+0xce8/0x10bc)
[ 32.008532] [<c0599d80>] (__alloc_pages_nodemask) from [<c0541bb4>]
(pagecache_get_page+0x128/0x358)
[ 32.017704] [<c0541bb4>] (pagecache_get_page) from [<c0543a8c>]
(grab_cache_page_write_begin+0x18/0x2c)
[ 32.027138] [<c0543a8c>] (grab_cache_page_write_begin) from
[<c0619fb0>] (block_write_begin+0x20/0xc4)
[ 32.036484] [<c0619fb0>] (block_write_begin) from [<c053e718>]
(generic_perform_write+0xb8/0x1d8)
[ 32.045395] [<c053e718>] (generic_perform_write) from [<c054496c>]
(__generic_file_write_iter+0x164/0x1ec)
[ 32.055090] [<c054496c>] (__generic_file_write_iter) from
[<c061c8a4>] (blkdev_write_iter+0xc8/0x1a4)
[ 32.064350] [<c061c8a4>] (blkdev_write_iter) from [<c05d50d0>]
(__vfs_write+0x13c/0x1cc)
[ 32.072476] [<c05d50d0>] (__vfs_write) from [<c05d81d4>]
(vfs_write+0xb0/0x1bc)
[ 32.079814] [<c05d81d4>] (vfs_write) from [<c05d85e4>]
(ksys_pwrite64+0x60/0x8c)
[ 32.087241] [<c05d85e4>] (ksys_pwrite64) from [<c04001a0>]
(ret_fast_syscall+0x0/0x4c)
[ 32.095187] Exception stack(0xe810dfa8 to 0xe810dff0)
[ 32.100256] dfa0: a2000000 0000000d 00000003
b6952008 00400000 00000000
[ 32.108466] dfc0: a2000000 0000000d a2000000 000000b5 00400000
0003b768 b6952008 00da2000
[ 32.116673] dfe0: 00000064 beb891b8 b6f85108 b6e38f2c
[ 32.121786] Mem-Info:
[ 32.124070] active_anon:5056 inactive_anon:4129 isolated_anon:0
[ 32.124070] active_file:6289 inactive_file:118790 isolated_file:0
[ 32.124070] unevictable:0 dirty:14118 writeback:6 unstable:0
[ 32.124070] slab_reclaimable:5653 slab_unreclaimable:4209
[ 32.124070] mapped:4839 shmem:4468 pagetables:165 bounce:0
[ 32.124070] free:348249 free_pcp:562 free_cma:57235
[ 32.158031] Node 0 active_anon:20224kB inactive_anon:16516kB
active_file:25156kB inactive_file:475160kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB mapped:19356kB dirty:56472kB
writeback:24kB shmem:17872kB shmem_thp: 0kB shmem_pmdmapped: 0kB
anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[ 32.186324] DMA free:186320kB min:22528kB low:28160kB high:33792kB
reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB
active_file:4736kB inactive_file:433580kB unevictable:0kB
writepending:56468kB present:783360kB managed:668264kB mlocked:0kB
kernel_stack:888kB pagetables:0kB bounce:0kB free_pcp:420kB
local_pcp:220kB free_cma:163840kB
[ 32.216693] lowmem_reserve[]: 0 0 1216 0
[ 32.220652] HighMem free:1206676kB min:512kB low:11592kB
high:22672kB reserved_highatomic:0KB active_anon:20224kB
inactive_anon:16516kB active_file:20420kB inactive_file:41584kB
unevictable:0kB writepending:0kB present:1310720kB managed:1310720kB
mlocked:0kB kernel_stack:0kB pagetables:660kB bounce:0kB
free_pcp:1816kB local_pcp:340kB free_cma:65100kB
[ 32.251805] lowmem_reserve[]: 0 0 0 0
[ 32.255482] DMA: 2*4kB (UM) 3*8kB (UME) 1*16kB (U) 1*32kB (M)
0*64kB 1*128kB (U) 5*256kB (ME) 5*512kB (ME) 4*1024kB (ME) 5*2048kB
(M) 1*4096kB (M) 20*8192kB (C) = 186320kB
[ 32.270871] HighMem: 183*4kB (UMC) 65*8kB (UMC) 21*16kB (M) 11*32kB
(UM) 6*64kB (UMC) 3*128kB (UM) 3*256kB (UM) 2*512kB (MC) 2*1024kB (MC)
2*2048kB (MC) 2*4096kB (UC) 145*8192kB (MC) = 1206676kB
[ 32.288273] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[ 32.296751] 129546 total pagecache pages
[ 32.300695] 0 pages in swap cache
[ 32.304019] Swap cache stats: add 0, delete 0, find 0/0
[ 32.309260] Free swap = 0kB
[ 32.312155] Total swap = 0kB
[ 32.315045] 523520 pages RAM
[ 32.317932] 327680 pages HighMem/MovableOnly
[ 32.322221] 28774 pages reserved
[ 32.325457] 57344 pages cma reserved
[ 32.329043] Tasks state (memory values in pages):
[ 32.333771] [ pid ] uid tgid total_vm rss pgtables_bytes
swapents oom_score_adj name
[ 32.342436] [ 183] 0 183 7370 1082 36864
0 0 systemd-journal
[ 32.351529] [ 209] 994 209 3742 326 40960
0 0 systemd-timesyn
[ 32.360620] [ 217] 0 217 3398 817 32768
0 -1000 systemd-udevd
[ 32.369528] [ 230] 993 230 1411 737 32768
0 0 systemd-network
[ 32.378620] [ 231] 992 231 1496 712 32768
0 0 systemd-resolve
[ 32.387713] [ 236] 996 236 1112 742 24576
0 -900 dbus-daemon
[ 32.396456] [ 241] 0 241 1895 1045 36864
0 0 haveged
[ 32.404850] [ 242] 0 242 1362 906 28672
0 0 systemd-logind
[ 32.413852] [ 243] 0 243 13412 2571 69632
0 0 NetworkManager
[ 32.422858] [ 244] 995 244 1197 608 28672
0 0 avahi-daemon
[ 32.431687] [ 245] 995 245 1164 59 28672
0 0 avahi-daemon
[ 32.440518] [ 246] 0 246 594 332 28672
0 0 atd
[ 32.448553] [ 248] 0 248 699 99 24576
0 0 syslogd
[ 32.456945] [ 251] 0 251 699 102 24576
0 0 klogd
[ 32.465171] [ 252] 0 252 676 365 24576
0 0 crond
[ 32.473390] [ 254] 0 254 1172 240 32768
0 0 systemd-hostnam
[ 32.482481] [ 264] 65534 264 605 32 24576
0 0 dnsmasq
[ 32.490876] [ 265] 0 265 556 357 28672
0 0 agetty
[ 32.499175] [ 266] 0 266 1131 613 32768
0 0 login
[ 32.507394] [ 350] 0 350 1840 1161 32768
0 0 systemd
[ 32.515788] [ 351] 0 351 2403 473 36864
0 0 (sd-pam)
[ 32.524268] [ 355] 0 355 827 611 24576
0 0 sh
[ 32.532227] [ 364] 0 364 7341 1145 53248
0 0 nm-dispatcher
[ 32.541142] [ 377] 0 377 711 510 28672
0 0 lava-test-runne
[ 32.550234] [ 387] 0 387 711 138 20480
0 0 lava-test-shell
[ 32.559316] [ 388] 0 388 711 523 20480
0 0 sh
[ 32.567273] [ 397] 0 397 1785 1518 36864
0 0 mkfs.ext4
ref:
https://lkft.validation.linaro.org/scheduler/job/1436647#L4261
https://lkft.validation.linaro.org/scheduler/job/1436562#L1247
--
Linaro LKFT
https://lkft.linaro.org
Powered by blists - more mailing lists