linux-ext4 - [Patch 0/4] RFC : Support for data gradation of a single file.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAExFE6n2npo8Uomk=bNYnC4CVL9aWQDMRdOeav0vAdce1prNMQ@mail.gmail.com>
Date:   Fri, 6 Apr 2018 17:11:27 +0530
From:   Sayan Ghosh <sgdgp.2014@...il.com>
To:     linux-ext4@...r.kernel.org
Cc:     linux-fsdevel@...r.kernel.org,
        "Bhattacharya, Suparna" <suparna.bhattacharya@....com>,
        niloy ganguly <ganguly.niloy@...il.com>,
        Madhumita Mallick <madhu.cse.ju@...il.com>,
        "Bharde, Madhumita" <madhumita.bharde@....com>
Subject: [Patch 0/4] RFC : Support for data gradation of a single file.

Hi all,

The following series of patches aim to store a file with a graded
information. Consider a scenario of video indexing for learning
programme where some of the portions of the video is annotated and
important than other portions, hence to be accessed more often. We
consider the similar scenario where we have a file along with a grade
information that mentions which blocks are important and which are
not. The grades we consider are binary with 1 denoting high grade.
Now the file is stored in a LVM which comprises of different set of
storage devices belong to different tiers (as ext4 doesn’t support
spanning over multiple block driver), - one combination could be
persistent memory and hard-disk. The target is to store the higher
graded blocks in the higher performance tier and the lower graded
blocks in the lower performance tier.
Consider a C code where the grade of the file blocks are being set in
the user space through extended attribute. The grade structure stores
the span of different high graded segments in the file with starting
high grade block numbers and the span length of the segments. We
assume grade of rest of the blocks as 0 (low).

---
typedef struct _grade{
   unsigned long long block_num;
   unsigned long long length;
} grade_extents;

int fd = open(filename, O_CREAT|O_RDWR, (mode_t)00777);
int xattr_value = 1;
int status1 = fsetxattr(fd, "user.is_graded", (const void *)&xattr_value,
                    sizeof(int), 0 );

grade_extents grade_array[] = {{1,2},{50,10}};
int status2 = fsetxattr(fd, "user.grade_array", (const void *)grade_array,
                        count*sizeof(grade_struct), 0 );

/* creating a 1 MB file */
int status3 = fallocate(fd, 0, 0, (1024 * 1024));
----

The first 2 patches of the series aim to read the grades and
pre-allocate space through fallocate in the respective tiers.
The next task is to write and read data to and from these files
(respectively). The 3rd patch aims at solving this issue.
The final patch in this patch series helps to get a reduced view of
the file, ie. just shows the high graded blocks of the file - the
motivation being  an application may need to access only the important
portions of the file such as accessing only the annotated parts of a
learning video.
We made the patches on top of Linux Kernel 4.7.2.

---
 fs/dax.c          | 139 +++++++++++++++++++++++++++++++++
 fs/ext4/ext4.h    |  17 +++++
 fs/ext4/extents.c | 151 +++++++++++++++++++++++++++++++++++-
 fs/ext4/file.c    | 225 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 4 files changed, 525 insertions(+), 7 deletions(-)

Regards,
Sayan Ghosh
IIT Kharagpur
‌