| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Sun, 23 Jun 2013 20:32:32 -0400 (EDT)
From: Eric Sandeen <esandeen@...hat.com>
To: Andreas Dilger <adilger@...ger.ca>
Cc: Namjae Jeon <linkinjeon@...il.com>,
"tytso@....edu" <tytso@....edu>,
"adilger.kernel@...ger.ca" <adilger.kernel@...ger.ca>,
"linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>,
"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
"linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>,
"a.sangwan@...sung.com" <a.sangwan@...sung.com>,
Namjae Jeon <namjae.jeon@...sung.com>
Subject: Re: [PATCH 0/3] ext4: introduce two new ioctls
On Jun 23, 2013, at 12:01 PM, Andreas Dilger <adilger@...ger.ca> wrote:
> On 2013-06-23, at 0:07, Namjae Jeon <linkinjeon@...il.com> wrote:
>
>> From: Namjae Jeon <namjae.jeon@...sung.com>
>>
>> This patch series introduces 2 new ioctls for ext4.
>>
>> Truncate_block_range ioctl truncates blocks from source file.
>
> How is this different from fallocate(FALLOC_FL_PUNCH_HOLE)? That is already in existing kernels, and portable across multiple filesystems.
>
Same question. Punch hole should do this already...
>> Transfer_block_range ioctl transfers data blocks from source file
>> and append them at the end of destination file.
>
> There is already a similar ioctl for defragmenting files. Is it possible to use that, or does it have different semantics?
>
>> Ioctl1: EXT4_IOC_TRUNCATE_BLOCK_RANGE:
>> This ioctl truncates a range of data blocks from file.
>> It is useful to remove easily and quickly the garbage data
>> at the middle of file.
>>
>> e.g. we have a movie file and there is long advertisement in movie file.
>> user want to remove only advertisement range.
>
> While this works in theory, there is very little chance that the movie data will align exactly to filesystem block boundaries.
>
Or more importantly on compression codec boundaries. Wouldn't this look like corruption at playback time?
Eric
> Cheers, Andreas
>
>> 1) Movie file (8GB), There is the adverisement of 500MB size.
>> ____________________________________________________________________
>> | | | |
>> | a) range | b) Advertisement | c) range |
>> | | (unneeded data) | |
>> |_____________________|___________________|_________________________|
>>
>> 2) Currently if user want to remove portion b), the conventional way
>> would be to copy a) and c) (7.5GB) to new file by reading data from
>> original file and writing to new file, followed up by delete original
>> file and rename new file. It will take long time.
>> When we measure time, it takes around 3 minutes.
>>
>> 3) If we use EXT4_IOC_TRUNCATE_BLOCK_RANGE, we can have garbage data removed
>> in less than a second. Also, no need to perform deletion and rename.
>> _______________________________________________
>> | | |
>> | a) range | c) range |
>> | | |
>> |_____________________|________________________|
>>
>>
>> #define EXT4_IOC_TRUNCATE_BLOCK_RANGE _IOW('f', 18, struct truncate_range)
>> struct truncate_range {
>> __u32 start_block;
>> __u32 length;
>> };
>>
>> example =>
>> Originally the file "abc" has the below extent tree:
>> debugfs: ex abc
>> Level Entries Logical Physical Length Flags
>> 0/ 0 1/ 3 0 - 4 33615 - 33619 5
>> 0/ 0 2/ 3 5 - 9 33855 - 33859 5
>> 0/ 0 3/ 3 10 - 14 69657 - 69661 5
>>
>> ls -lh abc
>> -rw-r--r-- 1 root 0 60.0K Jan 1 00:01 abc
>>
>> du -h abc
>> 60.0K abc
>>
>> e4_truncate_block_range abc 2 10
>> Return:
>> : Success
>>
>> After executing truncate_block_range ioctl, the extent tree:
>> ex abc
>> Level Entries Logical Physical Length Flags
>> 0/ 0 1/ 2 0 - 1 33615 - 33616 2
>> 0/ 0 2/ 2 2 - 4 69659 - 69661 3
>>
>> ls -lh abc
>> -rw-r--r-- 1 root 0 20.0K Jan 1 00:08 abc
>>
>> du -h abc
>> 20.0K abc
>>
>> This ioctl works in 2 parts:
>> 1) remove _only_ data blocks that resides within specified range.
>> If the entire range is a hole than nothing is removed.
>>
>> 2) update file's logical block offsets ranging from block number
>> "start_block + length" to last logical block of file such that
>> lblk_number = lblk_number - length;
>> This is done by updating starting block of all the extents that
>> resides within the range.
>>
>> If "start_block + length" is already equal to the last block of file
>> than no block is updated. This case is similar to convential truncate.
>>
>> In the above example:
>> The data blocks ranging from [2 - 11] have been removed
>> and the logical offsets of the file beyond block number 12 till last block
>> of file are updated by subtracting length from each of logical numbers.
>> This gives a contiguous logical space to the file.
>> Also, the logical size and disksize of the file are updated accordingly.
>>
>> Ioctl2: EXT4_IOC_TRANSFER_BLOCK_RANGE:
>> This ioctl transfers a range of data blocks from source file and append
>> them at the end of the destination file.
>> This is not actual data transfer but only metadata is moved.
>>
>> ____________________________________________________________________
>> | | | |
>> | a) range | b) range | c) range |
>> | | | |
>> |_____________________|___________________|_________________________|
>>
>> If user does not want b) in the orig file but wants to make a new file
>> comprising only b) OR wants b) at the end of an already existing file,
>> the conventional way of doing it would be to:
>> 1) Copy b) to new file
>> 2) Copy c) to temp file
>> 3) Truncate orig file to a)
>> 4) Copy c) from temp file to the end of orig file.
>> 5) Delete temp file.
>>
>> After this operations =>
>> orig_file:
>> __________________________________________
>> | | |
>> | a) range | c) range |
>> | | |
>> |_____________________|___________________|
>>
>> new_file:
>> _______________________
>> | |
>> | b) range |
>> | |
>> |_____________________|
>>
>> Again, this operation would take a long time (depending on the sizes of range)
>> if done using conventional way while using transfer_block_range ioctl reduces
>> the time within a second.
>>
>> #define EXT4_IOC_TRANSFER_BLOCK_RANGE _IOW('f', 19, struct transfer_range)
>> struct transfer_range {
>> __u32 dest_fd;
>> __u32 start_block;
>> __u32 length;
>> };
>>
>> example=>
>> debugfs: ex source
>> Level Entries Logical Physical Length Flags
>> 0/ 1 1/ 1 0 - 24 32809 25
>> 1/ 1 1/ 5 0 - 4 4071 - 4075 5
>> 1/ 1 2/ 5 5 - 9 4081 - 4085 5
>> 1/ 1 3/ 5 10 - 14 4091 - 4095 5
>> 1/ 1 4/ 5 15 - 19 4101 - 4105 5
>> 1/ 1 5/ 5 20 - 24 4151 - 4155 5
>>
>> debugfs: ex dest
>> Level Entries Logical Physical Length Flags
>> 0/ 0 1/ 3 0 - 4 32825 - 32829 5
>> 0/ 0 2/ 3 5 - 9 33545 - 33549 5
>> 0/ 0 3/ 3 10 - 14 33615 - 33619 5
>>
>> ls -lh source
>> -rw-r--r-- 1 root 0 100.0K Jan 1 00:01 source
>> ls -lh dest
>> -rw-r--r-- 1 root 0 60.0K Jan 1 00:01 dest
>>
>> du -h source
>> 104.0K source
>> du -h dest
>> 60.0K dest
>>
>> e4_transfer_block_range source dest 2 10
>> Return:
>> : Success
>>
>> debugfs: ex source
>> Level Entries Logical Physical Length Flags
>> 0/ 1 1/ 1 0 - 24 32809 25
>> 1/ 1 1/ 4 0 - 1 4071 - 4072 2
>> 1/ 1 2/ 4 12 - 14 4093 - 4095 3
>> 1/ 1 3/ 4 15 - 19 4101 - 4105 5
>> 1/ 1 4/ 4 20 - 24 4151 - 4155 5
>> debugfs: ex dest
>> Level Entries Logical Physical Length Flags
>> 0/ 1 1/ 1 0 - 24 32835 25
>> 1/ 1 1/ 6 0 - 4 32825 - 32829 5
>> 1/ 1 2/ 6 5 - 9 33545 - 33549 5
>> 1/ 1 3/ 6 10 - 14 33615 - 33619 5
>> 1/ 1 4/ 6 15 - 17 4073 - 4075 3
>> 1/ 1 5/ 6 18 - 22 4081 - 4085 5
>> 1/ 1 6/ 6 23 - 24 4091 - 4092 2
>>
>> ls -lh source
>> -rw-r--r-- 1 root 0 100.0K Jan 1 00:04 source
>> ls -lh dest
>> -rw-r--r-- 1 root 0 100.0K Jan 1 00:04 dest
>>
>> du -h source
>> 64.0K source
>> du -h dest
>> 104.0K dest
>>
>> The data blocks lying between [start_block to start_block + length) are appended
>> contiguously at the end of destination file.
>> The block transfer leaves a hole in the source file.
>> If any hole is encountered in the range, it is ommited.
>>
>> This ioctl does not change the logical size of the source file hence
>> leaves a hole in place of transfered range.
>> If user want contiguous logical space for source file,
>> it can truncate the hole by calling truncate_range_ioctl for source file.
>>
>> Example for above "source" file:
>> e4_truncate_block_range source 2 10
>> Return:
>> : Success
>> debugfs: ex source
>> Level Entries Logical Physical Length Flags
>> 0/ 1 1/ 1 0 - 14 32809 15
>> 1/ 1 1/ 4 0 - 1 4071 - 4072 2
>> 1/ 1 2/ 4 2 - 4 4093 - 4095 3
>> 1/ 1 3/ 4 5 - 9 4101 - 4105 5
>> 1/ 1 4/ 4 10 - 14 4151 - 4155 5
>>
>> Namjae Jeon (3):
>> ext4: Add EXT4_IOC_TRUNCATE_BLOCK_RANGE ioctl
>> ext4: make mext_next_extent non static and move get_ext_path
>> ext4: Add EXT4_IOC_TRANSFER_BLOCK_RANGE ioctl
>>
>> --
>> 1.7.9.5
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists