linux-kernel - Re: [RFC] extending splice for copy offloading

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <5249C7C7.7020207@itwm.fraunhofer.de>
Date:	Mon, 30 Sep 2013 20:49:43 +0200
From:	Bernd Schubert <bernd.schubert@...m.fraunhofer.de>
To:	"Myklebust, Trond" <Trond.Myklebust@...app.com>
CC:	Miklos Szeredi <miklos@...redi.hu>,
	Ric Wheeler <rwheeler@...hat.com>,
	"J. Bruce Fields" <bfields@...ldses.org>,
	Zach Brown <zab@...hat.com>,
	Anna Schumaker <schumaker.anna@...il.com>,
	Kernel Mailing List <linux-kernel@...r.kernel.org>,
	Linux-Fsdevel <linux-fsdevel@...r.kernel.org>,
	"linux-nfs@...r.kernel.org" <linux-nfs@...r.kernel.org>,
	"Schumaker, Bryan" <Bryan.Schumaker@...app.com>,
	"Martin K. Petersen" <mkp@....net>, Jens Axboe <axboe@...nel.dk>,
	Mark Fasheh <mfasheh@...e.com>,
	Joel Becker <jlbec@...lplan.org>,
	Eric Wong <normalperson@...t.net>
Subject: Re: [RFC] extending splice for copy offloading

On 09/30/2013 08:02 PM, Myklebust, Trond wrote:
> On Mon, 2013-09-30 at 19:48 +0200, Bernd Schubert wrote:
>> On 09/30/2013 07:44 PM, Myklebust, Trond wrote:
>>> On Mon, 2013-09-30 at 19:17 +0200, Bernd Schubert wrote:
>>>> It would be nice if there would be way if the file system would get a
>>>> hint that the target file is supposed to be copy of another file. That
>>>> way distributed file systems could also create the target-file with the
>>>> correct meta-information (same storage targets as in-file has).
>>>> Well, if we cannot agree on that, file system with a custom protocol at
>>>> least can detect from 0 to SSIZE_MAX and then reset metadata. I'm not
>>>> sure if this would work for pNFS, though.
>>>
>>> splice() does not create new files. What you appear to be asking for
>>> lies way outside the scope of that system call interface.
>>>
>>
>> Sorry I know, definitely outside the scope of splice, but in the context
>> of offloaded file copies. So the question is, what is the best way to
>> address/discuss that?
>
> Why does it need to be addressed in the first place?

An offloaded copy is still not efficient if different storage 
servers/targets used by from-file and to-file.

>
> What is preventing an application from retrieving and setting this
> information using standard libc functions such as fstat()+open(), and
> supplemented with libattr attr_setf/getf(), and libacl acl_get_fd/set_fd
> where appropriate?
>

At a minimum this requires network and metadata overhead. And while I'm 
working on FhGFS now, I still wonder what other file system need to do - 
for example Lustre pre-allocates storage-target files on creating a 
file, so file layout changes mean even more overhead there.
Anyway, if we could agree on to use libattr or libacl to teach the file 
system about the upcoming splice call I would be fine. Metadata overhead 
is probably negligible for large files.




Thanks,
Bernd

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/