linux-kernel - Re: [ck] Re: -mm merge plans for 2.6.23

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <46947784.6050100@yahoo.com.au>
Date:	Wed, 11 Jul 2007 16:24:04 +1000
From:	Nick Piggin <nickpiggin@...oo.com.au>
To:	Ray Lee <ray-lk@...rabbit.org>
CC:	Matthew Hawkins <darthmdh@...il.com>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Con Kolivas <kernel@...ivas.org>, ck list <ck@....kolivas.org>,
	Ingo Molnar <mingo@...e.hu>, Paul Jackson <pj@....com>,
	linux-mm@...ck.org, linux-kernel@...r.kernel.org
Subject: Re: [ck] Re: -mm merge plans for 2.6.23

Ray Lee wrote:
> On 7/10/07, Nick Piggin <nickpiggin@...oo.com.au> wrote:
> 
>> >> OK that's a good data point. It would be really good to be able to
>> >> do an analysis on your overnight IO patterns and the corresponding
>> >> memory reclaim behaviour and see why things are getting evicted.
>> >
>> > Eviction can happen for multiple reasons, as I'm sure you're painfully
>> > aware. It can happen because of poor balancing choices, or it can
>>
>> s/balancing/reclaim, yes. And for the nightly cron job case, this is
>> could quite possibly be the cause. At least updatedb should be fairly
>> easy to apply use-once heuristics for, so if they're not working then
>> we should hopefully be able to improve it.
> 
> 
> <nod> Sorry, I'm not so clear on the terminology, am I.
> 
> So, that's one part of it: one could argue that for that bit swap
> prefetch is a bit of a band-aid over the issue. A useful band-aid,
> that works today, isn't invasive, and can be ripped out at some future
> time if the underlying issue is eventually solved by a proper use-once
> aging mechanism, but nevertheless a band-aid.

I think for some workloads it is probably a bandaid, and for others
the concept of prefetching likely to be used again data back in is
undeniably going to be a win for others.

A lot of postitive reports I have seen about this say that desktop
the next morning is more responsive. So I kind of want to know what's
happening here -- as far as I can tell, swap prefetching shouldn't
help a huge amount to recover from a simple updatedb alone --
although if other cron stuff happened that used a bit more memory
afterwards and pushing out some of updatedb's cache, perhaps that's
when swap prefetching finds its niche. I don't know.

However, I don't like the fact that there is _any_ swap happening
on 1GB desktops after a single updatedb run. Is something else
running that hogs a huge amount of memory? Maybe that explains it,
but I don't know. I do know that we probably don't do very good
use-once algorithms on the dentry and inode caches, so updatedb
might cause them to push swap out. We could test that by winding
the vfs reclaim right up.

> The other part is when I've got evolution and a few other things open,
> then I run gimp on a raw photo and do some work on it, quit out of
> gimp, do a couple of things in a shell to upload the photo to my
> server, then switch back to evolution. Hang, waiting on swap in. Well,
> the kernel had some free time there to repopulate evolution's working
> set, and swap prefetch would help that, while better (or perfect!)
> heuristics in the reclaim *won't*.
> 
> That's the real issue here.

Yeah that's an issue, and swap prefetching has the potential to
help there no doubt at all. How much is the saving? I don't think
it will be like an order of magnitude because unfortunately we
also get mapped pagecache being thrown out as well as swap, so
for example all your evolution mailbox, libraries, executable,
etc. is still going to have to be paged back in.

Regarding swap prefetching. I'm not going to argue for or against
it anymore because I have really stopped following where it is
up to, for now. If the code and the results meet the standard that
Andrew wants then I don't particularly mind if he merges it.

It would be nice if some of you guys would still report and test
problems with reclaim when prefetching is turned off -- I have
never encountered the morning after sluggishness (although I don't
doubt for a minute that it is a problem for some).

-- 
SUSE Labs, Novell Inc.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/