lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Sun, 12 Feb 2017 14:47:13 +0100
From:   Paul Menzel <paulepanter@...rs.sourceforge.net>
To:     linux-mm@...ck.org
Cc:     linux-kernel@...r.kernel.org
Subject: Trying to understand OOM killer

Dear Linux folks,


since some time, at Linux 4.8, 4.9, and 4.10-rc6, the OOM kicks in on a
8 GB machine.

```
Feb 12 08:21:50 asrocke350m1 kernel: updatedb.mlocat invoked oom-killer: gfp_mask=0x16040d0(GFP_TEMPORARY|__GFP_COMP|__GFP_NOTRACK), nodemask=
Feb 12 08:21:50 asrocke350m1 kernel: updatedb.mlocat cpuset=/ mems_allowed=0
Feb 12 08:21:50 asrocke350m1 kernel: CPU: 1 PID: 2314 Comm: updatedb.mlocat Tainted: G         C      4.10.0-rc6-686-pae #1 Debian 4.10~rc6-1~
Feb 12 08:21:50 asrocke350m1 kernel: Hardware name: ASROCK E350M1/E350M1, BIOS 4.5-964-gd96669e9db 02/11/2017
Feb 12 08:21:51 asrocke350m1 kernel: Call Trace:
Feb 12 08:21:51 asrocke350m1 kernel:  ? dump_stack+0x55/0x73
Feb 12 08:21:51 asrocke350m1 kernel:  ? dump_header+0x64/0x1ab
Feb 12 08:21:52 asrocke350m1 kernel:  ? ___ratelimit+0x9f/0x100
Feb 12 08:21:52 asrocke350m1 kernel:  ? oom_kill_process+0x221/0x3e0
Feb 12 08:21:52 asrocke350m1 kernel:  ? has_capability_noaudit+0x1a/0x30
Feb 12 08:21:52 asrocke350m1 kernel:  ? oom_badness.part.13+0xd7/0x150
Feb 12 08:21:52 asrocke350m1 kernel:  ? out_of_memory+0xe4/0x290
Feb 12 08:21:52 asrocke350m1 kernel:  ? __alloc_pages_nodemask+0xab8/0xbc0
Feb 12 08:21:52 asrocke350m1 kernel:  ? xfs_init_local_fork+0x8a/0xd0 [xfs]
Feb 12 08:21:52 asrocke350m1 kernel:  ? cache_grow_begin.isra.60+0x75/0x510
Feb 12 08:21:52 asrocke350m1 kernel:  ? xfs_buf_rele+0x43/0x2e0 [xfs]
Feb 12 08:21:52 asrocke350m1 kernel:  ? kmem_cache_alloc+0x1fa/0x530
Feb 12 08:21:52 asrocke350m1 kernel:  ? __d_alloc+0x23/0x180
Feb 12 08:21:52 asrocke350m1 kernel:  ? d_alloc+0x18/0x80
Feb 12 08:21:52 asrocke350m1 kernel:  ? d_alloc_parallel+0x47/0x450
Feb 12 08:21:52 asrocke350m1 kernel:  ? d_splice_alias+0x10d/0x3a0
Feb 12 08:21:53 asrocke350m1 kernel:  ? lockref_get_not_dead+0x8/0x40
Feb 12 08:21:53 asrocke350m1 kernel:  ? unlazy_walk+0xf9/0x1a0
Feb 12 08:21:53 asrocke350m1 kernel:  ? lookup_slow+0x5e/0x140
Feb 12 08:21:53 asrocke350m1 kernel:  ? walk_component+0x1b4/0x350
Feb 12 08:21:53 asrocke350m1 kernel:  ? path_lookupat+0x49/0xe0
Feb 12 08:21:53 asrocke350m1 kernel:  ? filename_lookup+0x99/0x190
Feb 12 08:21:53 asrocke350m1 kernel:  ? __check_object_size+0x9e/0x11c
Feb 12 08:21:53 asrocke350m1 kernel:  ? strncpy_from_user+0x39/0x140
Feb 12 08:21:53 asrocke350m1 kernel:  ? getname_flags+0x55/0x1a0
Feb 12 08:21:53 asrocke350m1 kernel:  ? vfs_fstatat+0x60/0xb0
Feb 12 08:21:53 asrocke350m1 kernel:  ? SyS_lstat64+0x2d/0x50
Feb 12 08:21:53 asrocke350m1 kernel:  ? sys_sync+0x9d/0xa0
Feb 12 08:21:53 asrocke350m1 kernel:  ? SyS_poll+0x6b/0x110
Feb 12 08:21:53 asrocke350m1 kernel:  ? do_fast_syscall_32+0x8a/0x150
Feb 12 08:21:53 asrocke350m1 kernel:  ? entry_SYSENTER_32+0x4e/0x7c
Feb 12 08:21:53 asrocke350m1 kernel: Mem-Info:
Feb 12 08:21:53 asrocke350m1 kernel: active_anon:119893 inactive_anon:17678 isolated_anon:0
                                    active_file:31461 inactive_file:219091 isolated_file:0
                                    unevictable:21 dirty:0 writeback:0 unstable:0
                                    slab_reclaimable:127609 slab_unreclaimable:9519
                                    mapped:63113 shmem:6177 pagetables:1601 bounce:0
                                    free:1381579 free_pcp:583 free_cma:0
Feb 12 08:21:53 asrocke350m1 kernel: Node 0 active_anon:479572kB inactive_anon:70712kB active_file:125844kB inactive_file:876364kB unevictable
Feb 12 08:21:53 asrocke350m1 kernel: DMA free:3840kB min:788kB low:984kB high:1180kB active_anon:0kB inactive_anon:0kB active_file:0kB inactiv
Feb 12 08:21:53 asrocke350m1 kernel: lowmem_reserve[]: 0 763 7663 7663
Feb 12 08:21:53 asrocke350m1 kernel: Normal free:38764kB min:38828kB low:48532kB high:58236kB active_anon:0kB inactive_anon:0kB active_file:16
Feb 12 08:21:53 asrocke350m1 kernel: lowmem_reserve[]: 0 0 55201 55201
Feb 12 08:21:53 asrocke350m1 kernel: HighMem free:5483712kB min:512kB low:88240kB high:175968kB active_anon:479572kB inactive_anon:70712kB act
Feb 12 08:21:54 asrocke350m1 kernel: lowmem_reserve[]: 0 0 0 0
Feb 12 08:21:54 asrocke350m1 kernel: DMA: 0*4kB 42*8kB (UE) 69*16kB (UE) 7*32kB (UE) 10*64kB (UE) 2*128kB (U) 1*256kB (U) 2*512kB (U) 0*1024kB
Feb 12 08:21:54 asrocke350m1 kernel: Normal: 17*4kB (UME) 583*8kB (UME) 1983*16kB (UE) 72*32kB (ME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*
Feb 12 08:21:54 asrocke350m1 kernel: HighMem: 2156*4kB (UM) 1334*8kB (UM) 2760*16kB (UM) 2087*32kB (UM) 1274*64kB (UM) 491*128kB (UM) 234*256k
Feb 12 08:21:54 asrocke350m1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 12 08:21:54 asrocke350m1 kernel: 256720 total pagecache pages
Feb 12 08:21:54 asrocke350m1 kernel: 0 pages in swap cache
Feb 12 08:21:54 asrocke350m1 kernel: Swap cache stats: add 0, delete 0, find 0/0
Feb 12 08:21:54 asrocke350m1 kernel: Free swap  = 4194300kB
Feb 12 08:21:54 asrocke350m1 kernel: Total swap = 4194300kB
Feb 12 08:21:54 asrocke350m1 kernel: 1994197 pages RAM
Feb 12 08:21:54 asrocke350m1 kernel: 1766457 pages HighMem/MovableOnly
Feb 12 08:21:54 asrocke350m1 kernel: 22689 pages reserved
Feb 12 08:21:54 asrocke350m1 kernel: 0 pages hwpoisoned
Feb 12 08:21:54 asrocke350m1 kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Feb 12 08:21:54 asrocke350m1 kernel: [  388]     0   388    18525     8958      32       3        0             0 systemd-journal
Feb 12 08:21:54 asrocke350m1 kernel: [  416]     0   416     5467      391       7       3        0             0 lvmetad
Feb 12 08:21:54 asrocke350m1 kernel: [  429]     0   429     4034     1047       8       3        0         -1000 systemd-udevd
Feb 12 08:21:54 asrocke350m1 kernel: [  597]     0   597      835      573       5       3        0             0 mdadm
Feb 12 08:21:54 asrocke350m1 kernel: [  825]   134   825     4244      992       9       3        0             0 systemd-timesyn
Feb 12 08:21:54 asrocke350m1 kernel: [  835]     0   835     1428      935       6       3        0             0 smartd
Feb 12 08:21:54 asrocke350m1 kernel: [  836]     0   836     1313      719       6       3        0             0 cron
Feb 12 08:21:54 asrocke350m1 kernel: [  838]     0   838     1038      564       6       3        0             0 anacron
Feb 12 08:21:54 asrocke350m1 kernel: [  840]     0   840     5883      751      10       3        0             0 rsyslogd
Feb 12 08:21:54 asrocke350m1 kernel: [  846]     0   846     1108      410       7       3        0             0 irexec
Feb 12 08:21:54 asrocke350m1 kernel: [  851]     0   851     9931     1654      13       3        0             0 accounts-daemon
Feb 12 08:21:54 asrocke350m1 kernel: [  853]   104   853     1699     1110       8       3        0          -900 dbus-daemon
Feb 12 08:21:54 asrocke350m1 kernel: [  867]   105   867     1563      814       6       3        0             0 avahi-daemon
Feb 12 08:21:54 asrocke350m1 kernel: [  868]     0   868    23658     4150      26       3        0             0 NetworkManager
Feb 12 08:21:54 asrocke350m1 kernel: [  869]     0   869     1128      420       6       3        0             0 lircmd
Feb 12 08:21:54 asrocke350m1 kernel: [  870]     0   870      889      568       5       3        0             0 atd
Feb 12 08:21:54 asrocke350m1 kernel: [  878]     0   878     1853     1147       7       3        0             0 systemd-logind
Feb 12 08:21:54 asrocke350m1 kernel: [  879]     0   879    10954     2159      16       3        0             0 ModemManager
Feb 12 08:21:54 asrocke350m1 kernel: [  887]     0   887      558       18       5       3        0             0 minissdpd
Feb 12 08:21:54 asrocke350m1 kernel: [  893]   105   893     1563       73       6       3        0             0 avahi-daemon
Feb 12 08:21:54 asrocke350m1 kernel: [  895]   129   895    11545     3234      17       3        0             0 colord
Feb 12 08:21:54 asrocke350m1 kernel: [  904]     0   904     9909     1981      15       3        0             0 polkitd
Feb 12 08:21:54 asrocke350m1 kernel: [  923]     0   923    94106     6465      51       3        0             0 libvirtd
Feb 12 08:21:54 asrocke350m1 kernel: [  926]     0   926     2616     1283       8       3        0         -1000 sshd
Feb 12 08:21:54 asrocke350m1 kernel: [  961]     0   961    10354     1870      14       3        0             0 gdm3
Feb 12 08:21:54 asrocke350m1 kernel: [  973]     0   973     7965     1994      13       3        0             0 gdm-session-wor
Feb 12 08:21:54 asrocke350m1 kernel: [ 1192]     0  1192     1207      758       7       3        0             0 lircd
Feb 12 08:21:54 asrocke350m1 kernel: [ 1195]     0  1195     1129      431       6       3        0             0 lircd-uinput
Feb 12 08:21:54 asrocke350m1 kernel: [ 1252]   122  1252     8304     7454      21       3        0             0 tor
Feb 12 08:21:55 asrocke350m1 kernel: [ 1255]   101  1255     2813      702       8       3        0             0 exim4
Feb 12 08:21:55 asrocke350m1 kernel: [ 1278]     0  1278     2856     1758       9       3        0             0 wpa_supplicant
Feb 12 08:21:55 asrocke350m1 kernel: [ 1280]     0  1280     2117     1352       8       3        0             0 apache2
Feb 12 08:21:55 asrocke350m1 kernel: [ 1288]    33  1288      811       39       5       3        0             0 htcacheclean
Feb 12 08:21:55 asrocke350m1 kernel: [ 1289]   130  1289     2369     1407       8       3        0             0 systemd
Feb 12 08:21:55 asrocke350m1 kernel: [ 1290]   130  1290     2745      392       9       3        0             0 (sd-pam)
Feb 12 08:21:55 asrocke350m1 kernel: [ 1292]   130  1292     6945     1251      10       3        0             0 gdm-wayland-ses
Feb 12 08:21:55 asrocke350m1 kernel: [ 1294]   130  1294     1590     1002       7       3        0             0 dbus-daemon
Feb 12 08:21:55 asrocke350m1 kernel: [ 1311]   130  1311    19475     3265      23       3        0             0 gnome-session-b
Feb 12 08:21:55 asrocke350m1 kernel: [ 1374]   130  1374   208428    26571     111       3        0             0 gnome-shell
Feb 12 08:21:55 asrocke350m1 kernel: [ 1379]     0  1379    21121     2547      18       3        0             0 upowerd
Feb 12 08:21:55 asrocke350m1 kernel: [ 1405]   130  1405    28029     9140      46       3        0             0 Xwayland
Feb 12 08:21:55 asrocke350m1 kernel: [ 1411]   130  1411    11342     1415      12       3        0             0 at-spi-bus-laun
Feb 12 08:21:55 asrocke350m1 kernel: [ 1416]   130  1416     1564      873       7       3        0             0 dbus-daemon
Feb 12 08:21:55 asrocke350m1 kernel: [ 1418]   130  1418     7592     1544      11       3        0             0 at-spi2-registr
Feb 12 08:21:55 asrocke350m1 kernel: [ 1425]   130  1425   222202     2441      21       3        0             0 pulseaudio
Feb 12 08:21:55 asrocke350m1 kernel: [ 1440]     0  1440    11899     2862      18       3        0             0 packagekitd
Feb 12 08:21:55 asrocke350m1 kernel: [ 1441]   130  1441   115081    12067      68       3        0             0 gnome-settings-
Feb 12 08:21:55 asrocke350m1 kernel: [ 1480]     0  1480     7986     2033      14       3        0             0 gdm-session-wor
Feb 12 08:21:55 asrocke350m1 kernel: [ 1486]  1000  1486     2369     1371       8       3        0             0 systemd
Feb 12 08:21:55 asrocke350m1 kernel: [ 1487]  1000  1487     7355      402      12       3        0             0 (sd-pam)
Feb 12 08:21:55 asrocke350m1 kernel: [ 1491]  1000  1491    10023     1922      13       3        0             0 gnome-keyring-d
Feb 12 08:21:55 asrocke350m1 kernel: [ 1494]  1000  1494     7424     1403      11       3        0             0 gdm-x-session
Feb 12 08:21:55 asrocke350m1 kernel: [ 1496]  1000  1496    33920    14489      59       3        0             0 Xorg
Feb 12 08:21:55 asrocke350m1 kernel: [ 1500]  1000  1500     1589      997       7       3        0             0 dbus-daemon
Feb 12 08:21:55 asrocke350m1 kernel: [ 1505]  1000  1505    14890     7121      24       3        0             0 awesome
Feb 12 08:21:55 asrocke350m1 kernel: [ 1534]  1000  1534     4310     1673      10       3        0             0 arbtt-capture
Feb 12 08:21:55 asrocke350m1 kernel: [ 1628]  1000  1628     1180       54       6       3        0             0 ssh-agent
Feb 12 08:21:55 asrocke350m1 kernel: [ 1635]  1000  1635    11361     1459      12       3        0             0 at-spi-bus-laun
Feb 12 08:21:55 asrocke350m1 kernel: [ 1640]  1000  1640     1564      873       6       3        0             0 dbus-daemon
Feb 12 08:21:55 asrocke350m1 kernel: [ 1644]  1000  1644     7594     1294      13       3        0             0 at-spi2-registr
Feb 12 08:21:55 asrocke350m1 kernel: [ 1646]  1000  1646    21040     8672      36       3        0             0 gnome-terminal-
Feb 12 08:21:55 asrocke350m1 kernel: [ 1653]  1000  1653    10027     1331      15       3        0             0 gvfsd
Feb 12 08:21:55 asrocke350m1 kernel: [ 1658]  1000  1658    13148     1359      15       3        0             0 gvfsd-fuse
Feb 12 08:21:55 asrocke350m1 kernel: [ 1672]  1000  1672     1773     1222       7       3        0             0 bash
Feb 12 08:21:55 asrocke350m1 kernel: [ 1681]  1000  1681    15803    13913      36       3        0             0 gdb
Feb 12 08:21:55 asrocke350m1 kernel: [ 1689]  1000  1689     1759     1205       6       3        0             0 bash
Feb 12 08:21:55 asrocke350m1 kernel: [ 1700]  1000  1700     1759     1211       7       3        0             0 bash
Feb 12 08:21:55 asrocke350m1 kernel: [ 1713]  1000  1713    19912     7108      29       3        0             0 nm-applet
Feb 12 08:21:55 asrocke350m1 kernel: [ 1724]  1000  1724    10075     1350      12       3        0             0 gnome-keyring-d
Feb 12 08:21:55 asrocke350m1 kernel: [ 1730]     0  1730     2026      927       7       3        0             0 dhclient
Feb 12 08:21:55 asrocke350m1 kernel: [ 1925]     0  1925      555      321       5       3        0             0 run-parts
Feb 12 08:21:56 asrocke350m1 kernel: [ 2208]    33  2208     2123     1124       8       3        0             0 apache2
Feb 12 08:21:56 asrocke350m1 kernel: [ 2209]    33  2209     2123      795       7       3        0             0 apache2
Feb 12 08:21:56 asrocke350m1 kernel: [ 2210]    33  2210     2123      795       7       3        0             0 apache2
Feb 12 08:21:57 asrocke350m1 kernel: [ 2211]    33  2211     2123      795       7       3        0             0 apache2
Feb 12 08:21:58 asrocke350m1 kernel: [ 2212]    33  2212     2123      795       7       3        0             0 apache2
Feb 12 08:21:59 asrocke350m1 kernel: [ 2213]    33  2213     2123      795       7       3        0             0 apache2
Feb 12 08:21:59 asrocke350m1 kernel: [ 2227]     0  2227     3831     1969      12       3        0             0 cupsd
Feb 12 08:21:59 asrocke350m1 kernel: [ 2308]     0  2308     1307      712       6       3        0             0 mlocate
Feb 12 08:22:00 asrocke350m1 kernel: [ 2313]     0  2313     1008      160       6       3        0             0 flock
Feb 12 08:22:01 asrocke350m1 kernel: [ 2314]     0  2314     1183      726       7       3        0             0 updatedb.mlocat
Feb 12 08:22:01 asrocke350m1 kernel: [ 2780]  1000  2780     2559     1409       8       3        0             0 ssh
Feb 12 08:22:01 asrocke350m1 kernel: [ 2789]  1000  2789     1773     1264       7       3        0             0 bash
Feb 12 08:22:01 asrocke350m1 kernel: [ 2798]  1000  2798   208137    83634     291       3        0             0 firefox-esr
Feb 12 08:22:01 asrocke350m1 kernel: [ 2807]  1000  2807     1760     1253       7       3        0             0 bash
Feb 12 08:22:01 asrocke350m1 kernel: [ 2833]  1000  2833     3332     1638      11       3        0             0 gconfd-2
Feb 12 08:22:01 asrocke350m1 kernel: [ 2883]  1000  2883     2944     2168       8       3        0             0 vim
Feb 12 08:22:02 asrocke350m1 kernel: [ 2895]  1000  2895     1784     1327       7       3        0             0 bash
Feb 12 08:22:03 asrocke350m1 kernel: [ 4298]  1000  4298     3444     2800      10       3        0             0 debcheckout
Feb 12 08:22:03 asrocke350m1 kernel: [ 4302]  1000  4302     1898      998       7       3        0             0 git
Feb 12 08:22:03 asrocke350m1 kernel: [ 4303]  1000  4303    11054     6508      22       3        0             0 git-remote-http
Feb 12 08:22:04 asrocke350m1 kernel: [ 4306]  1000  4306     4232      983       8       3        0             0 git
Feb 12 08:22:04 asrocke350m1 kernel: [ 4309]  1000  4309     2621     1522       8       3        0             0 git
Feb 12 08:22:04 asrocke350m1 kernel: [ 4322]     0  4322     1408      829       6       3        0             0 inetd
Feb 12 08:22:05 asrocke350m1 kernel: Out of memory: Kill process 2798 (firefox-esr) score 27 or sacrifice child
Feb 12 08:22:05 asrocke350m1 kernel: Killed process 2798 (firefox-esr) total-vm:832548kB, anon-rss:248168kB, file-rss:86300kB, shmem-rss:68kB
Feb 12 08:22:06 asrocke350m1 kernel: perf: interrupt took too long (2505 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
```

The sum of the RSS values is 312,260. According to the article [1], one
page is 4 KB in size. That make it less then 1.3 GB.

```
$ more /proc/meminfo # after OOM run
MemTotal:        7886032 kB
MemFree:         1613032 kB
MemAvailable:    4510132 kB
Buffers:          160536 kB
Cached:          3103908 kB
SwapCached:            0 kB
Active:          4004748 kB
Inactive:        1629624 kB
Active(anon):    2255340 kB
Inactive(anon):   158560 kB
Active(file):    1749408 kB
Inactive(file):  1471064 kB
Unevictable:         232 kB
Mlocked:             232 kB
HighTotal:       7065828 kB
HighFree:        1555764 kB
LowTotal:         820204 kB
LowFree:           57268 kB
SwapTotal:       4194300 kB
SwapFree:        4194300 kB
Dirty:                12 kB
Writeback:             0 kB
AnonPages:       2370220 kB
Mapped:           419968 kB
Shmem:             43972 kB
Slab:             460304 kB
SReclaimable:     423320 kB
SUnreclaim:        36984 kB
KernelStack:        4224 kB
PageTables:        14580 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8137316 kB
Committed_AS:    5379296 kB
VmallocTotal:     122880 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       10232 kB
DirectMap2M:      901120 kB
```

So I wonder, why the OOM killer kicked in at all.

Hints and insight is appreciated.


Thanks,

Paul


[1] http://careers.directi.com/display/tu/Understanding+and+optimizing+Memory+utilization
Download attachment "signature.asc" of type "application/pgp-signature" (196 bytes)

Powered by blists - more mailing lists