Message-Id: <33B96043-86E4-4A8E-8282-F117524F1CA3@kernel.crashing.org>
Date:	Mon, 15 Jan 2007 11:13:54 -0600
From:	Kumar Gala <galak@...nel.crashing.org>
To:	Nick Piggin <nickpiggin@...oo.com.au>
Cc:	Linux Kernel list <linux-kernel@...r.kernel.org>
Subject: Re: tuning/tweaking VM settings for low memory (preventing OOM)


On Jan 12, 2007, at 10:54 PM, Nick Piggin wrote:

> Kumar Gala wrote:
>> I'm working on an embedded PPC setup with 64M of memory and no  
>> swap.   I'm trying to figure out how best to tune the VM for an  
>> OOM situation  I'm running into.
>> I'm running a 2.6.16.35 kernel and have a bittorrent app that  
>> appears  to be initializing a large file for it to download into.   
>> What I see  before running the app:
>> /bigfoot/usb_disk # cat /proc/meminfo
>> MemTotal:        62520 kB
>> MemFree:         49192 kB
>> Buffers:          8240 kB
>> Cached:            740 kB
>> SwapCached:          0 kB
>> Active:           8196 kB
>> Inactive:         1236 kB
>> HighTotal:           0 kB
>> HighFree:            0 kB
>> LowTotal:        62520 kB
>> LowFree:         49192 kB
>> SwapTotal:           0 kB
>> SwapFree:            0 kB
>> Dirty:               0 kB
>> Writeback:           0 kB
>> Mapped:            916 kB
>> Slab:             2224 kB
>> CommitLimit:     31260 kB
>> Committed_AS:     1704 kB
>> PageTables:         88 kB
>> VmallocTotal:   933872 kB
>> VmallocUsed:      9416 kB
>> VmallocChunk:   923628 kB
>> after the OOM:
>> /bigfoot/usb_disk # cat /proc/meminfo
>> MemTotal:        62520 kB
>> MemFree:          1608 kB
>> Buffers:          8212 kB
>> Cached:          42780 kB
>> SwapCached:          0 kB
>> Active:           6228 kB
>> Inactive:        45176 kB
>> HighTotal:           0 kB
>> HighFree:            0 kB
>> LowTotal:        62520 kB
>> LowFree:          1608 kB
>> SwapTotal:           0 kB
>> SwapFree:            0 kB
>> Dirty:           35208 kB
>> Writeback:        5616 kB
>> Mapped:            892 kB
>> Slab:             7788 kB
>> CommitLimit:     31260 kB
>> Committed_AS:     1704 kB
>> PageTables:         88 kB
>> VmallocTotal:   933872 kB
>> VmallocUsed:      9416 kB
>> VmallocChunk:   923628 kB
>> Which makes me think that we aren't writing back fast enough.  If  
>> I  mount the drive "sync" the issue clearly goes away.
>> It appears from an strace we are doing ftruncate64(5, 178257920)  
>> when  we OOM.
>> Any ideas on VM parameters to tweak so we throttle this from  
>> occurring?
>
> You don't give us the actual OOM message. In newer kernels, there  
> has been
> quite a bit of work done to improve the OOM situation -- search  
> changelogs
> in mm/oom_kill.c mm/vmscan.c mm/page_alloc.c.

Here's the OOM message:

[ 3639.341858] oom-killer: gfp_mask=0xd0, order=0
[ 3639.341879] Call Trace:
[ 3639.341891] [C3D65CC0] [C0007644] show_stack+0x48/0x194 (unreliable)
[ 3639.341931] [C3D65CF0] [C00418D4] out_of_memory+0x210/0x244
[ 3639.341962] [C3D65D30] [C0042A54] __alloc_pages+0x254/0x2b0
[ 3639.341988] [C3D65D70] [C0042B34] __get_free_pages+0x84/0xa0
[ 3639.342012] [C3D65D80] [C007368C] __pollwait+0x48/0xe4
[ 3639.342042] [C3D65DA0] [C015D758] datagram_poll+0x128/0x154
[ 3639.342075] [C3D65DB0] [C01A33C4] udp_poll+0x24/0x178
[ 3639.342111] [C3D65DD0] [C015526C] sock_poll+0x28/0x38
[ 3639.342134] [C3D65DE0] [C0074850] do_sys_poll+0x328/0x460
[ 3639.342159] [C3D65E50] [C00749CC] sys_poll+0x44/0x7c
[ 3639.342181] [C3D65E60] [C000D698] ret_from_syscall+0x0/0x38
[ 3639.342206] --- Exception: c01 at netb_zero_receive_thread_proc 
+0x5a0/0xa78 [netb_driver]
[ 3639.342331]     LR = netb_zero_receive_thread_proc+0x928/0xa78  
[netb_driver]
[ 3639.342350] [C3D65FC0] [C003193C] kthread+0xf4/0x130
[ 3639.342374] [C3D65FF0] [C000E778] kernel_thread+0x44/0x60
[ 3639.342397] Mem-info:
[ 3639.342406] DMA per-cpu:
[ 3639.342419] cpu 0 hot: high 18, batch 3 used:2
[ 3639.342434] cpu 0 cold: high 6, batch 1 used:5
[ 3639.342445] DMA32 per-cpu: empty
[ 3639.342456] Normal per-cpu: empty
[ 3639.342467] HighMem per-cpu: empty
[ 3639.342485] Free pages:        1420kB (0kB HighMem)
[ 3639.342507] Active:1268 inactive:11572 dirty:8693 writeback:1888  
unstable:0 free:355 slab:1967 mapped:184 pagetables:26
[ 3639.342540] DMA free:1420kB min:1024kB low:1280kB high:1536kB  
active:5072kB inactive:46288kB present:65536kB pages_scanned:26684  
all_unreclaimable? no
[ 3639.342564] lowmem_reserve[]: 0 0 0 0
[ 3639.342588] DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB  
inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
[ 3639.342609] lowmem_reserve[]: 0 0 0 0
[ 3639.342633] Normal free:0kB min:0kB low:0kB high:0kB active:0kB  
inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
[ 3639.342655] lowmem_reserve[]: 0 0 0 0
[ 3639.342680] HighMem free:0kB min:128kB low:128kB high:128kB active: 
0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
[ 3639.342702] lowmem_reserve[]: 0 0 0 0
[ 3639.342719] DMA: 73*4kB 7*8kB 3*16kB 8*32kB 0*64kB 2*128kB 0*256kB  
1*512kB 0*1024kB 0*2048kB 0*4096kB = 1420kB
[ 3639.342768] DMA32: empty
[ 3639.342778] Normal: empty
[ 3639.342788] HighMem: empty
[ 3639.342799] Free swap:            0kB
[ 3639.345716] 16384 pages of RAM
[ 3639.345728] 752 reserved pages
[ 3639.345738] 10652 pages shared
[ 3639.345748] 0 pages swap cached
[ 3639.345811] Out of Memory: Kill process 742 (sh) score 45 and  
children.
[ 3639.352458] Out of memory: Killed process 863 (killerbt-standa).
[ 3639.372795] oom-killer: gfp_mask=0x200d2, order=0
[ 3639.373137] Call Trace:
[ 3639.373426] [C3777D10] [C0007644] show_stack+0x48/0x194 (unreliable)
[ 3639.373894] [C3777D40] [C00418D4] out_of_memory+0x210/0x244
[ 3639.374298] [C3777D80] [C0042A54] __alloc_pages+0x254/0x2b0
[ 3639.374686] [C3777DC0] [C003D834] find_or_create_page+0x98/0xf0
[ 3639.375114] [C3777DE0] [C0061D40] cont_prepare_write+0xa8/0x2cc
[ 3639.375511] [C3777E20] [C00CACE0] fat_prepare_write+0x30/0x40
[ 3639.375929] [C3777E30] [C005F69C] __generic_cont_expand+0xa4/0x158
[ 3639.376330] [C3777E50] [C00CA4F8] fat_notify_change+0xf4/0x208
[ 3639.376718] [C3777E80] [C007D528] notify_change+0x220/0x23c
[ 3639.377142] [C3777EB0] [C005A318] do_truncate+0x58/0x88
[ 3639.377536] [C3777F10] [C005A6C4] do_sys_ftruncate+0x180/0x1a8
[ 3639.377972] [C3777F40] [C000D698] ret_from_syscall+0x0/0x38
[ 3639.378373] --- Exception: c01 at 0x3013d7ec
[ 3639.378709]     LR = 0x1001c304
[ 3639.379186] Mem-info:
[ 3639.379482] DMA per-cpu:
[ 3639.379800] cpu 0 hot: high 18, batch 3 used:1
[ 3639.380613] cpu 0 cold: high 6, batch 1 used:0
[ 3639.380949] DMA32 per-cpu: empty
[ 3639.381280] Normal per-cpu: empty
[ 3639.381602] HighMem per-cpu: empty
[ 3639.382041] Free pages:        1300kB (0kB HighMem)
[ 3639.382356] Active:1303 inactive:11592 dirty:8662 writeback:1889  
unstable:0 free:325 slab:1973 mapped:229 pagetables:26
[ 3639.382692] DMA free:1300kB min:1024kB low:1280kB high:1536kB  
active:5212kB inactive:46368kB present:65536kB pages_scanned:33  
all_unreclaimable? no
[ 3639.383054] lowmem_reserve[]: 0 0 0 0
[ 3639.383570] DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB  
inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
[ 3639.383915] lowmem_reserve[]: 0 0 0 0
[ 3639.384494] Normal free:0kB min:0kB low:0kB high:0kB active:0kB  
inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
[ 3639.384810] lowmem_reserve[]: 0 0 0 0
[ 3639.385691] HighMem free:0kB min:128kB low:128kB high:128kB active: 
0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
[ 3639.386053] lowmem_reserve[]: 0 0 0 0
[ 3639.386558] DMA: 35*4kB 11*8kB 3*16kB 8*32kB 0*64kB 2*128kB  
0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1300kB
[ 3639.387016] DMA32: empty
[ 3639.387341] Normal: empty
[ 3639.387659] HighMem: empty
[ 3639.388029] Free swap:            0kB
[ 3639.391280] 16384 pages of RAM
[ 3639.391627] 752 reserved pages
[ 3639.391961] 10670 pages shared
[ 3639.392268] 0 pages swap cached
[ 3639.931258] killerbt-standa: page allocation failure. order:0,  
mode:0x200d2
[ 3639.931765] Call Trace:
[ 3639.932005] [C3777D50] [C0007644] show_stack+0x48/0x194 (unreliable)
[ 3639.932334] [C3777D80] [C0042A88] __alloc_pages+0x288/0x2b0
[ 3639.932604] [C3777DC0] [C003D834] find_or_create_page+0x98/0xf0
[ 3639.932894] [C3777DE0] [C0061D40] cont_prepare_write+0xa8/0x2cc
[ 3639.933181] [C3777E20] [C00CACE0] fat_prepare_write+0x30/0x40
[ 3639.933455] [C3777E30] [C005F69C] __generic_cont_expand+0xa4/0x158
[ 3639.933721] [C3777E50] [C00CA4F8] fat_notify_change+0xf4/0x208
[ 3639.934017] [C3777E80] [C007D528] notify_change+0x220/0x23c
[ 3639.934283] [C3777EB0] [C005A318] do_truncate+0x58/0x88
[ 3639.934554] [C3777F10] [C005A6C4] do_sys_ftruncate+0x180/0x1a8
[ 3639.934821] [C3777F40] [C000D698] ret_from_syscall+0x0/0x38
[ 3639.935076] --- Exception: c01 at 0x3013d7ec
[ 3639.935099]     LR = 0x1001c304
[ 3639.935109] Mem-info:
[ 3639.935118] DMA per-cpu:
[ 3639.935132] cpu 0 hot: high 18, batch 3 used:1
[ 3639.935147] cpu 0 cold: high 6, batch 1 used:5
[ 3639.935158] DMA32 per-cpu: empty
[ 3639.935169] Normal per-cpu: empty
[ 3639.935180] HighMem per-cpu: empty
[ 3639.935197] Free pages:           0kB (0kB HighMem)
[ 3639.935219] Active:1532 inactive:11664 dirty:8919 writeback:1839  
unstable:0 free:0 slab:1999 mapped:230 pagetables:26
[ 3639.935251] DMA free:0kB min:1024kB low:1280kB high:1536kB active: 
6128kB inactive:46656kB present:65536kB pages_scanned:168  
all_unreclaimable? no
[ 3639.935274] lowmem_reserve[]: 0 0 0 0
[ 3639.935298] DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB  
inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
[ 3639.935319] lowmem_reserve[]: 0 0 0 0
[ 3639.935344] Normal free:0kB min:0kB low:0kB high:0kB active:0kB  
inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
[ 3639.935365] lowmem_reserve[]: 0 0 0 0
[ 3639.935390] HighMem free:0kB min:128kB low:128kB high:128kB active: 
0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
[ 3639.935412] lowmem_reserve[]: 0 0 0 0
[ 3639.935429] DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB  
0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[ 3639.935477] DMA32: empty
[ 3639.935486] Normal: empty
[ 3639.935496] HighMem: empty
[ 3639.935507] Free swap:            0kB
[ 3639.938491] 16384 pages of RAM
[ 3639.938502] 752 reserved pages
[ 3639.938513] 10960 pages shared
[ 3639.938522] 0 pages swap cached

Any ideas?  The amount in dirty & writeback doesn't seem as bad  
compared to /proc/meminfo.

- k




-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists