linux-kernel - Re: Unexpected splice "always copy" behavior observed

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 20 May 2010 02:27:29 +1000
From:	Nick Piggin <npiggin@...e.de>
To:	Mathieu Desnoyers <mathieu.desnoyers@...icios.com>
Cc:	Steven Rostedt <rostedt@...dmis.org>,
	Miklos Szeredi <miklos@...redi.hu>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	peterz@...radead.org, fweisbec@...il.com, tardyp@...il.com,
	mingo@...e.hu, acme@...hat.com, tzanussi@...il.com,
	paulus@...ba.org, linux-kernel@...r.kernel.org,
	arjan@...radead.org, ziga.mahkovec@...il.com, davem@...emloft.net,
	linux-mm@...ck.org, akpm@...ux-foundation.org,
	kosaki.motohiro@...fujitsu.com, cl@...ux-foundation.org,
	tj@...nel.org, jens.axboe@...cle.com
Subject: Re: Unexpected splice "always copy" behavior observed

On Wed, May 19, 2010 at 11:57:32AM -0400, Mathieu Desnoyers wrote:
> * Steven Rostedt (rostedt@...dmis.org) wrote:
> > On Wed, 2010-05-19 at 17:33 +0200, Miklos Szeredi wrote:
> > > On Wed, 19 May 2010, Linus Torvalds wrote:
> > > > Btw, since you apparently have a real case - is the "splice to file" 
> > > > always just an append? IOW, if I'm not right in assuming that the only 
> > > > sane thing people would reasonable care about is "append to a file", then 
> > > > holler now.
> > > 
> > > Virtual machines might reasonably need this for splicing to a disk
> > > image.
> > 
> > This comes down to balancing speed and complexity. Perhaps a copy is
> > fine in this case.
> > 
> > I'm concerned about high speed tracing, where we are always just taking
> > pages from the trace ring buffer and appending them to a file or sending
> > them off to the network. The slower this is, the more likely you will
> > lose events.
> > 
> > If the "move only on append to file" is easy to implement, I would
> > really like to see that happen. The speed of splicing a disk image for a
> > virtual machine only impacts the patience of the user. The speed of
> > splicing tracing output, impacts how much you can trace without losing
> > events.
> 
> I'm with Steven here. I only care about appending full pages at the end of a
> file. If possible, I'd also like to steal back the pages after waiting for the
> writeback I/O to complete so we can put them back in the ring buffer without
> stressing the page cache and the page allocator needlessly.

Got to think about complexity and how much is really worth trying to
speed up strange cases. The page allocator is the generic "pipe" in
the kernel to move pages between subsystems when they become unused :)

The page cache can be directed to be written out and discarded with
fadvise and such.

You might also consider using direct IO.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/