linux-kernel - Re: [question] how to figure out OOM reason? should dump slab/vmalloc info when OOM?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <52DE6AA0.1000801@huawei.com>
Date:	Tue, 21 Jan 2014 20:40:00 +0800
From:	Jianguo Wu <wujianguo@...wei.com>
To:	David Rientjes <rientjes@...gle.com>
CC:	Andrew Morton <akpm@...ux-foundation.org>,
	Johannes Weiner <hannes@...xchg.org>,
	Rik van Riel <riel@...hat.com>,
	"linux-mm@...ck.org" <linux-mm@...ck.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [question] how to figure out OOM reason? should dump slab/vmalloc
 info when OOM?

On 2014/1/21 13:34, David Rientjes wrote:

> On Mon, 20 Jan 2014, Jianguo Wu wrote:
> 
>> When OOM happen, will dump buddy free areas info, hugetlb pages info,
>> memory state of all eligible tasks, per-cpu memory info.
>> But do not dump slab/vmalloc info, sometime, it's not enough to figure out the
>> reason OOM happened.
>>
>> So, my questions are:
>> 1. Should dump slab/vmalloc info when OOM happen? Though we can get these from proc file,
>> but usually we do not monitor the logs and check proc file immediately when OOM happened.
>>
> 

Hi David,
Thank you for your patience to answer!

> The problem is that slabinfo becomes excessively verbose and dumping it 
> all to the kernel log often times causes important messages to be lost.  
> This is why we control things like the tasklist dump with a VM sysctl.  It 
> would be possible to dump, say, the top ten slab caches with the highest 
> memory usage, but it will only be helpful for slab leaks.  Typically there 
> are better debugging tools available than analyzing the kernel log; if you 
> see unusually high slab memory in the meminfo dump, you can enable it.
> 

But, when OOM has happened, we can only use kernel log, slab/vmalloc info from proc
is stale. Maybe we can dump slab/vmalloc with a VM sysctl, and only top 10/20 entrys?

Thanks.

>> 2. /proc/$pid/smaps and pagecache info also helpful when OOM, should also be dumped?

>>
> 
> Also very verbose and would cause important messages to be lost, we try to 
> avoid spamming the kernel log with all of this information as much as 
> possible.
> 
>> 3. Without these info, usually how to figure out OOM reason?
>>
> 
> Analyze the memory usage in the meminfo and determine what is unusually 
> high; if it's mostly anonymous memory, you can usually correlate it back 
> to a high rss for a process in the tasklist that you didn't suspect to be 
> using that much memory, for example.
> 
> 



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/