linux-ext4 - Re: [PATCH 0/3] ext4: introduce two new ioctls

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20130624024459.GJ29376@dastard>
Date:	Mon, 24 Jun 2013 12:44:59 +1000
From:	Dave Chinner <david@...morbit.com>
To:	Eric Sandeen <esandeen@...hat.com>
Cc:	Andreas Dilger <adilger@...ger.ca>,
	Namjae Jeon <linkinjeon@...il.com>,
	"tytso@....edu" <tytso@....edu>,
	"adilger.kernel@...ger.ca" <adilger.kernel@...ger.ca>,
	"linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>,
	"a.sangwan@...sung.com" <a.sangwan@...sung.com>,
	Namjae Jeon <namjae.jeon@...sung.com>
Subject: Re: [PATCH 0/3] ext4: introduce two new ioctls

On Sun, Jun 23, 2013 at 08:32:32PM -0400, Eric Sandeen wrote:
> On Jun 23, 2013, at 12:01 PM, Andreas Dilger <adilger@...ger.ca> wrote:
> 
> > On 2013-06-23, at 0:07, Namjae Jeon <linkinjeon@...il.com> wrote:
> > 
> >> From: Namjae Jeon <namjae.jeon@...sung.com>
> >> 
> >> This patch series introduces 2 new ioctls for ext4.
> >> 
> >> Truncate_block_range ioctl truncates blocks from source file.
> > 
> > How is this different from fallocate(FALLOC_FL_PUNCH_HOLE)?  That is already in existing kernels, and portable across multiple filesystems. 
> > 
> Same question.  Punch hole should do this already...

Hole punch doesn't change the offsets of subsequent extents in the
file - it leaves a hole. This doesn't leave a hole at all - all the 
extents above the range are shifted down to offset where the extents
being punched out started.

> >> Transfer_block_range ioctl transfers data blocks from source file
> >> and append them at the end of destination file.
> > 
> > There is already a similar ioctl for defragmenting files. Is it possible to use that, or does it have different semantics?
> > 
> >> Ioctl1:        EXT4_IOC_TRUNCATE_BLOCK_RANGE:
> >> This ioctl truncates a range of data blocks from file.
> >> It is useful to remove easily and quickly the garbage data
> >> at the middle of file.
> >> 
> >> e.g. we have a movie file and there is long advertisement in movie file.
> >> user want to remove only advertisement range.
> > 
> > While this works in theory, there is very little chance that the movie data will align exactly to filesystem block boundaries. 
> > 
> Or more importantly on compression codec boundaries.   Wouldn't this look like corruption at playback time?

Not necessarily. Video codecs are encapsulated in a container that
can be used to link key frames together so you can do this sort of
manipulation of the file contents and just change an
offset-to-next-keyframe value in the container and it all just
works. Non linear editting (NLE) software has been doing this
manually for 15 years by copying data around - this just optimises
the operation by manipulating the extent mapping rather than needing
to physically copy the data.

FWIW, I've heard persistent rumors going back several years of
various DVR companies shipping equivalent ioctl-based functionality
for XFS filesystems to both insert and remove chunks in video
streams, but I've never been able to find the code for it anywhere.

Hence, at minimum, this should be a fallocate() operation, not a ext4
specific ioctl as it is relatively trivial to implement on most
extent based filesystems.

However, My conditions for merging such functionality into fallocate
are:
	1. it needs xfs_io support [to provide]
	2. comprehensive xfstests coverage, similar to the current
	   hole punch coverage we have.

> >> #define EXT4_IOC_TRUNCATE_BLOCK_RANGE  _IOW('f', 18, struct truncate_range)
> >> struct truncate_range {
> >>      __u32 start_block;
> >>      __u32 length;
> >> };

And have 64 bit file size support, please!

Also FALLOC_FL_COLLAPSE_RANGE is probably a better name for the
operation being done ;)

Cheers,

Dave.
-- 
Dave Chinner
david@...morbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html