linux-kernel - Re: [PATCH] fs: add fincore(2) (mincore(2) for file descriptors)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20100127181424.GA21585@shareable.org>
Date:	Wed, 27 Jan 2010 18:14:24 +0000
From:	Jamie Lokier <jamie@...reable.org>
To:	Chris Frost <frost@...ucla.edu>
Cc:	Heiko Carstens <heiko.carstens@...ibm.com>,
	Alexander Viro <viro@...iv.linux.org.uk>,
	Benny Halevy <bhalevy@...asas.com>,
	Andrew Morton <akpm@...ux-foundation.org>,
	linux-fsdevel@...r.kernel.org, linux-mm@...ck.org,
	linux-kernel@...r.kernel.org,
	Steve VanDeBogart <vandebo-lkml@...dbox.net>
Subject: Re: [PATCH] fs: add fincore(2) (mincore(2) for file descriptors)

Chris Frost wrote:
> We introduced this system call while modifying SQLite and the GIMP to
> request large prefetches for what would otherwise be non-sequential reads.
> As a macrobenchmark, we see a 125s SQLite query (72s system time) reduced
> to 75s (18s system time) by using fincore() instead of mincore(). This
> speedup of course varies by benchmark and benchmarks size; we've seen
> both minimal speedups and 1000x speedups. More on these benchmarks in the
> publication _Reducing Seek Overhead with Application-Directed Prefetching_
> in USENIX ATC 2009 and at http://libprefetch.cs.ucla.edu/.

My first thought was:

Why is calling fincore() and then issuing reads better than simply
calling readahead() on the same range?  I.e. why is readahead() (or
POSIX_FADV_WILLNEED) unsuitable to give the same result?  Or even
issuing lots of AIO requests.

I knew that I was missing something, so I read the paper ;-) I don't
fully understand it, but *think* that it says fincore() is used to
detect when the kernel is evicting pages faster than libprefetch had
planned for, implying memory pressure, so it adjusts its planning in
response.

Interesting idea, though I wonder if it wouldn't be even better to
have a direct way to ask the kernel "tell me when there is memory
pressure causing my file to be evicted".

-- Jamie
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/