| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <6d4bc9fc0608040053x4d7a9e14xe9de793cd0787736@mail.gmail.com> Date: Fri, 4 Aug 2006 09:53:14 +0200 From: "Maarten Maathuis" <madman2003@...il.com> To: "Dave Kleikamp" <shaggy@...tin.ibm.com>, linux-kernel@...r.kernel.org Subject: Re: heavy file i/o on ext3 filesystem leads to huge ext3_inode_cache and dentry_cache that doesn't return to normal for hours On 8/4/06, Dave Kleikamp <shaggy@...tin.ibm.com> wrote: > On Thu, 2006-08-03 at 18:27 +0200, Maarten Maathuis wrote: > > I have a kernel specific problem and this seemed like a suitable place to ask. > > > > I would like responces to be CC'ed to me if possible. > > > > I use a 2.6.17-ck1 kernel on an amd64 system. I have observed this > > problem on other/older kernels. > > > > Whenever there is serious hard drive activity (such as doing "slocate > > -u") ext3_inode_cache and dentry_cache grow to a combined 400-500 MiB. > > The behavior of slocate (updatedb) is pretty well-known, but nobody has > come up with a real solution. > > > The amount of objects is more than half a million. > > > > This will slowly decrease to normal, but will take many hours. It does > > not result in any OOM, because i have 1 GiB of memory. > > > > As far as i understand hard drive cache should not be in the slab. Are > > these just the inode's, because the amount of memory consumption seems > > large for that? > > inodes and directory cache entries (dentries). In general, it's a good > idea to cache inodes and dentries that have recently been read. slocate > is a special case since it will traverse all of the directories and > never look at them again (until the next time it runs). The kernel > doesn't have any idea that it may be a good idea to free those objects. > Wasn't there something like a possible read only flag to prevent that? > > > > I have found a way to clear the memory (and unfortunately most of the cache): > > > > echo 100 > /proc/sys/vm/nr_hugepages > > echo 0 > /proc/sys/vm/nr_hugepages > > A better way to clear just the inodes and dentries (that aren't in use) > is: > > echo 2 > /proc/sys/vm/drop_caches > > This feature is relatively new. > Thank you, i tried echo'ing a 1 into that and it had no effect iirc. Documentation on /proc/sys/vm seems pretty scarce. > > This suggest the kernel can free this memory. It's not the caching > > that bothers me, what bothers me is that it seems to reside in the > > slab. > > I'm not sure why that bothers you. A more common complaint is that all > the inodes and dentries being cached push out other pages to swap. > The reason i asked was because i expected cache related things to be seperated from "normal" memory usage. Since free already seperates normal and buffers/cache, it was very strange seeing 400 or 500 MiB memory usage. > Completely free memory doesn't do the system any good. The kernel > attempts to keep as much as possible in cache in case something is > needed again. These objects are easily reclaimable, so when more memory > is needed, they can be freed with very little overhead. > > > I am not a developer, so please keep that in mind when replying. > > > > I hope someone can be of help. > > > > Sincerely, > > > > Maarten Maathuis. > > Shaggy > -- > David Kleikamp > IBM Linux Technology Center > > - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists