linux-kernel - Re: [PATCH] remove throttle_vm

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Fri, 05 Oct 2007 09:32:57 +0200
From:	Peter Zijlstra <a.p.zijlstra@...llo.nl>
To:	Andrew Morton <akpm@...ux-foundation.org>
Cc:	Miklos Szeredi <miklos@...redi.hu>, wfg@...l.ustc.edu.cn,
	linux-mm@...ck.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH] remove throttle_vm_writeout()

On Thu, 2007-10-04 at 16:09 -0700, Andrew Morton wrote:
> On Fri, 05 Oct 2007 00:39:16 +0200
> Miklos Szeredi <miklos@...redi.hu> wrote:
> 
> > > throttle_vm_writeout() should be a per-zone thing, I guess.  Perhaps fixing
> > > that would fix your deadlock.  That's doubtful, but I don't know anything
> > > about your deadlock so I cannot say.
> > 
> > No, doing the throttling per-zone won't in itself fix the deadlock.
> > 
> > Here's a deadlock example:
> > 
> > Total memory = 32M
> > /proc/sys/vm/dirty_ratio = 10
> > dirty_threshold = 3M
> > ratelimit_pages = 1M
> > 
> > Some program dirties 4M (dirty_threshold + ratelimit_pages) of mmap on
> > a fuse fs.  Page balancing is called which turns all these into
> > writeback pages.
> > 
> > Then userspace filesystem gets a write request, and tries to allocate
> > memory needed to complete the writeout.
> > 
> > That will possibly trigger direct reclaim, and throttle_vm_writeout()
> > will be called.  That will block until nr_writeback goes below 3.3M
> > (dirty_threshold + 10%).  But since all 4M of writeback is from the
> > fuse fs, that will never happen.
> > 
> > Does that explain it better?
> > 
> 
> yup, thanks.
> 
> This is a somewhat general problem: a userspace process is in the IO path. 
> Userspace block drivers, for example - pretty much anything which involves
> kernel->userspace upcalls for storage applications.
> 
> I solved it once in the past by marking the userspace process as
> PF_MEMALLOC and I beleive that others have implemented the same hack.
> 
> I suspect that what we need is a general solution, and that the solution
> will involve explicitly telling the kernel that this process is one which
> actually cleans memory and needs special treatment.
> 
> Because I bet there will be other corner-cases where such a process needs
> kernel help, and there might be optimisation opportunities as well.
> 
> Problem is, any such mark-me-as-special syscall would need to be
> privileged, and FUSE servers presently don't require special perms (do
> they?)

I think just adding nr_cpus * ratelimit_pages to the dirth_thresh in
throttle_vm_writeout() will also solve the problem

Download attachment "signature.asc" of type "application/pgp-signature" (190 bytes)