linux-kernel - Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20070728000823.GA14933@atjola.homenet>
Date:	Sat, 28 Jul 2007 02:08:23 +0200
From:	Björn Steinbrink <B.Steinbrink@....de>
To:	Andi Kleen <andi@...stfloor.org>
Cc:	Rene Herman <rene.herman@...il.com>,
	Daniel Hazelton <dhazelton@...er.net>,
	Mike Galbraith <efault@....de>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Ingo Molnar <mingo@...e.hu>,
	Frank Kingswood <frank@...gswood-consulting.co.uk>,
	Nick Piggin <nickpiggin@...oo.com.au>,
	Ray Lee <ray-lk@...rabbit.org>,
	Jesper Juhl <jesper.juhl@...il.com>,
	ck list <ck@....kolivas.org>, Paul Jackson <pj@....com>,
	linux-mm@...ck.org, linux-kernel@...r.kernel.org
Subject: Re: RFT: updatedb "morning after" problem [was: Re: -mm merge
	plans for 2.6.23]

On 2007.07.28 01:29:19 +0200, Andi Kleen wrote:
> > Any faults in that reasoning?
> 
> GNU sort uses a merge sort with temporary files on disk. Not sure
> how much it keeps in memory during that, but it's probably less
> than 150MB. At some point the dirty limit should kick in and write back the 
> data of the temporary files; so it's not quite the same as anonymous memory. 
> But it's not that different given.

Hm, does that change anything? The files need to be read at the end (so
they go into the cache) and are delete afterwards (cache gets freed I
guess?).

> It would be better to measure than to guess. At least Andrew's measurements
> on 128MB actually didn't show updatedb being really that big a problem.

Here's a before/after memory usage for an updatedb run:
root@...ola:~# free -m
             total       used       free     shared    buffers     cached
Mem:          2011       1995         15          0        269        779
-/+ buffers/cache:        946       1064
Swap:         1945          0       1945
root@...ola:~# updatedb
root@...ola:~# free -m
             total       used       free     shared    buffers     cached
Mem:          2011       1914         96          0        209        746
-/+ buffers/cache:        958       1052
Swap:         1945          0       1944

81MB more unused RAM afterwards.

If anyone can make use of that, here's a snippet from /proc/$PID/smaps
of updatedb's sort process, when it was at about its peak memory usage
(according to the RSS column in top), which was about 50MB.

2b90ab3c1000-2b90ae4c3000 rw-p 2b90ab3c1000 00:00 0 
Size:              50184 kB
Rss:               50184 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:     50184 kB
Referenced:        50184 kB

> Perhaps some people have much more files or simply a less efficient
> updatedb implementation?

sort (GNU coreutils) 5.97

GNU updatedb version 4.2.31

> I guess the people who complain here that loudly really need to supply
> some real numbers. 

Just to clarify: I'm not complaining either way, neither about not
merging swap prefetch, nor about someone wanting that to be merge. It
was rather the "discussion" that caught my attention... Just in case ;-)

Björn

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/