linux-ext4 - Re: [Patch 0/4] RFC : Support for data gradation of a single file.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <63F7D7EC-EDF0-49FC-BAFA-F49695F1263F@dilger.ca>
Date:   Fri, 6 Apr 2018 15:31:39 -0600
From:   Andreas Dilger <adilger@...ger.ca>
To:     Sayan Ghosh <sgdgp.2014@...il.com>
Cc:     Ext4 Developers List <linux-ext4@...r.kernel.org>,
        Linux FS Devel <linux-fsdevel@...r.kernel.org>,
        "Bhattacharya, Suparna" <suparna.bhattacharya@....com>,
        niloy ganguly <ganguly.niloy@...il.com>,
        Madhumita Mallick <madhu.cse.ju@...il.com>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        "Bharde, Madhumita" <madhumita.bharde@....com>,
        Jens Axboe <axboe@...nel.dk>
Subject: Re: [Patch 0/4] RFC : Support for data gradation of a single file.

On Apr 6, 2018, at 5:41 AM, Sayan Ghosh <sgdgp.2014@...il.com> wrote:
> 
> Hi all,
> 
> The following series of patches aim to store a file with a graded
> information. Consider a scenario of video indexing for learning
> programme where some of the portions of the video is annotated and
> important than other portions, hence to be accessed more often. We
> consider the similar scenario where we have a file along with a grade
> information that mentions which blocks are important and which are
> not. The grades we consider are binary with 1 denoting high grade.
> Now the file is stored in a LVM which comprises of different set of
> storage devices belong to different tiers (as ext4 doesn’t support
> spanning over multiple block driver), - one combination could be
> persistent memory and hard-disk. The target is to store the higher
> graded blocks in the higher performance tier and the lower graded
> blocks in the lower performance tier.
> Consider a C code where the grade of the file blocks are being set in
> the user space through extended attribute. The grade structure stores
> the span of different high graded segments in the file with starting
> high grade block numbers and the span length of the segments. We
> assume grade of rest of the blocks as 0 (low).

There was a considerable amount of work and discussion on implementing
Stream IDs for the block layer.  This would annotate writes from userspace
and allow the underlying storage (filesystem and block layer) to use the
stream ID for block allocation.  See the following for more details:

    https://lwn.net/Articles/717755/
    https://lwn.net/Articles/726477/
    http://lists.infradead.org/pipermail/linux-nvme/2017-June/011322.html

In the absence of other information, the Stream ID would just mean "group
allocations with the same ID together". After some discussion, it looks
like the latest patch has generic "lifetime" hints rather than "stream IDs",
but the end result is largely the same.

It would make sense for you to spend time testing and fixing that patch
series instead of trying to introduce a new interface.  IMHO, there is
no need to make these hints persistent on disk, since their state could
be inferred by the allocation placement directly.

Cheers, Andreas






Download attachment "signature.asc" of type "application/pgp-signature" (874 bytes)