linux-kernel - Re: userspace pagecache management tool

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Sat, 3 Mar 2007 15:34:54 -0800
From:	Andrew Morton <akpm@...ux-foundation.org>
To:	ray-gmail@...rabbit.org
Cc:	"Ray Lee" <madrabbit@...il.com>, linux-kernel@...r.kernel.org
Subject: Re: userspace pagecache management tool

On Sat, 3 Mar 2007 14:58:48 -0800 "Ray Lee" <madrabbit@...il.com> wrote:

> On 3/3/07, Andrew Morton <akpm@...ux-foundation.org> wrote:
> > It is to address the "waah, backups fill my memory with pagecache" and the
> > "waah, updatedb swapped everything out" and the "waah, copying a DVD
> > gobbled all my memory" problems.
> 
> Is the updatedb problem really due to pagecache?

It's a combination of pagecache, slab cache and of course contention for
the disk.  In my experience the latter preponderates: the disk is sekeing
like mad and I can't get its attention.  Others report lots of swapout,
which will be a combination of slab and pagecache, varying degrees of each.

> > When running
> >
> >         pagecache-management.sh dd if=100-mb-file of=foo
> >         or
> >         pagecache-management.sh cp -a /usr/src/linux-2.6.20 /usr/src/foo
> >
> > the amount of pagecache in the machine is pretty much unaltered.  Maybe a
> > megabyte of additional cache in the second case, because of ext3 indirect
> > blocks.
> 
> ray@...enix:~/work/home/pagecache-management$ grep ext3_i
> /proc/slabinfo; ./pagecache-management.sh sudo updatedb; grep ext3_i
> /proc/slabinfo
> ext3_inode_cache   21024  23722   1584    2    1 : tunables   24   12
>   0 : slabdata  11861  11861      0
> ext3_inode_cache   41332  41332   1584    2    1 : tunables   24   12
>   0 : slabdata  20666  20666      0
> ray@...enix:~/work/home/pagecache-management$ echo $(( 1584 * (41332-21024) ))
> 32167872

If 32 MB is the whole lot then by eliminating pagecache, we just solved the
problem.  But perhaps you instantiated a lot more VFS cache and all you're
seeing there is the leftovers.

> Or is there a /proc/sys/vm/* knob that can be tweaked for this
> before/after the updatedb?

/proc/sys/vm/vfs_cache_pressure should help.  I don't recall anyone
reporting its effects with updatedb.

> But yeah, I for one would happily submit patches to upstream authors
> to address this there. There's no reason code should be making the
> kernel guess its intention on these things.

I think so.  We're dealing with super-special cases here and often trying
to fix those in-kernel will degrade other, often more common cases.

<wonders about sys_reclaim_dentry(const char *pathname)>
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/