linux-kernel - Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20070729055352.21797a85.pj@sgi.com>
Date:	Sun, 29 Jul 2007 05:53:52 -0700
From:	Paul Jackson <pj@....com>
To:	Andi Kleen <andi@...stfloor.org>
Cc:	B.Steinbrink@....de, rene.herman@...il.com, dhazelton@...er.net,
	efault@....de, akpm@...ux-foundation.org, mingo@...e.hu,
	frank@...gswood-consulting.co.uk, andi@...stfloor.org,
	nickpiggin@...oo.com.au, ray-lk@...rabbit.org,
	jesper.juhl@...il.com, ck@....kolivas.org, linux-mm@...ck.org,
	linux-kernel@...r.kernel.org
Subject: Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans
 for 2.6.23]

Andi wrote:
> GNU sort uses a merge sort with temporary files on disk. Not sure
> how much it keeps in memory during that, but it's probably less
> than 150MB. 

If I'm reading the source code for GNU sort correctly, then the
following snippet of shell code displays how much memory it uses
for its primary buffer on typical GNU/Linux systems:

    head -2 /proc/meminfo | awk '
	NR == 1 { memtotal = $2 }
	NR == 2 { memfree = $2 }
	END     {
		   if (memfree > memtotal/8)
		       m = memfree
		   else
		       m = memtotal/8
		   print "sort size:", m/2, "kB"
	}
    '

That is, over simplifying, GNU sort looks at the first two entries
in /proc/meminfo, which for example on a machine near me happen to be:

  MemTotal:      2336472 kB
  MemFree:        110600 kB

and then uses one-half of whichever is -greater- of MemTotal/8 or
MemFree.

... However ... for the typical GNU locate updatedb run, it is sorting
the list of pathnames for almost all files on the system, which is
usually larger than fits in one of these sized buffers.   So it ends up
using quite a few of the temporary files you mention, which tends to
chew up easily freed memory.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@....com> 1.925.600.0401
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/