linux-kernel - Re: OOM-Killer kills too much with 2.6.32.2

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Tue, 26 Jan 2010 12:33:29 +0000
From:	Chris Wilson <chris@...is-wilson.co.uk>
To:	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>,
	Roman Jarosz <kedgedev@...il.com>
Cc:	kosaki.motohiro@...fujitsu.com,
	lkml <linux-kernel@...r.kernel.org>,
	A Rojas <nqn1976list@...il.com>,
	Hugh Dickins <hugh.dickins@...cali.co.uk>,
	"A. Boulan" <arnaud.boulan@...ertysurf.fr>, michael@...nelt.co.at,
	jcnengel@...glemail.com, rientjes@...gle.com, earny@...4u.de,
	Jesse Barnes <jbarnes@...tuousgeek.org>,
	Eric Anholt <eric@...olt.net>
Subject: Re: OOM-Killer kills too much with 2.6.32.2

On Tue, 26 Jan 2010 20:07:43 +0900 (JST), KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com> wrote:
>         obj->filp = shmem_file_setup("drm mm object", size, VM_NORESERVE);
> (snip)
>         /* Basically we want to disable the OOM killer and handle ENOMEM
>          * ourselves by sacrificing pages from cached buffers.
>          * XXX shmem_file_[gs]et_gfp_mask()
>          */
>         mapping_set_gfp_mask(obj->filp->f_path.dentry->d_inode->i_mapping,
>                              GFP_HIGHUSER |
>                              __GFP_COLD |
>                              __GFP_FS |
>                              __GFP_RECLAIMABLE |
>                              __GFP_NORETRY |
>                              __GFP_NOWARN |
>                              __GFP_NOMEMALLOC);
> 
> 
> This comment is lie. __GFP_NORETY cause ENOMEM to shmem, not GEM itself.
> GEM can't handle nor recover it. I suspect following commit is wrong. 

Indeed, the NORETRY flag is required for the inode mapping routines to
return ENOMEM instead of triggering the OOM killer themselves. GEM has
code to handle the ENOMEM as returned from shmem (please at least read the
code before commenting, and comments are appreciated), by attempting to
free up some of its own inactive buffers before retrying the allocation
(with NORETRY removed, so the OOM killer will be invoked on the second
instance). The reason for this convoluted approach is that GEM's inactive
list shrinker requires the struct mutex and so cannot be run when GEM
itself is attempting and failing to allocate memory. We could recover from
more situations if we made some more invasive changes to our locking.

This is without a doubt an area that needs improvement.
-- 
Chris Wilson, Intel Open Source Technology Centre
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/