linux-ext4 - Re: ext4-lazy (SMR-optimizations) landing to kernel?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <A10FB389-CE93-493F-93D1-D61DD1A9466D@dilger.ca>
Date:   Mon, 17 Apr 2017 13:59:14 -0600
From:   Andreas Dilger <adilger@...ger.ca>
To:     Eric Sandeen <sandeen@...hat.com>
Cc:     Ts'o Theodore <tytso@....edu>,
        linux-ext4 <linux-ext4@...r.kernel.org>
Subject: Re: ext4-lazy (SMR-optimizations) landing to kernel?

On Apr 17, 2017, at 8:18 AM, Eric Sandeen <esandeen@...hat.com> wrote:
> 
> On 4/10/17 10:06 PM, Andreas Dilger wrote:
>> Hi Ted,
>> now that FAST'17 is behind us, is there any plan to land the ext4-lazy code
>> (SMR optimizations) to the upstream kernel?  This looks like it improves
>> some workloads even without SMR disks, and doesn't have any noticeable
>> overhead for other workloads.
>> 
>> I'd guess the one thing that we might want to do is still allow the journal
>> to optionally checkpoint the metadata to the filesystem in the background,
>> when the filesystem is otherwise idle, so that in case of journal loss for
>> some reason the whole filesystem is not lost?
> 
> IIRC even the new larger default journal size was a big win by itself, yes?

For many-thread modification that is definitely a win.  We've used
journal sizes up to 1GB for Lustre object targets and up to 4GB for
metadata targets, just because worst-case journal credit reservation
causes transaction stalls even if the transaction doesn't grow large.

That is especially true for fast devices like SSD metadata targets
that do tens of thousands of ops/sec with quotas, ACLs, xattrs, etc.
This is somewhat worse on Lustre because we also store additional
xattrs and also update Lustre-specific transaction log files in the
same transaction as each filesystem modifying operation.

IMHO, the ext4-lazy feature would also potentially be useful for non-SMR
devices, where we could do full data journaling (optimistically, small
files?) to a large flash journal device, and only write to the disk device
periodically (once the journal gets near full, or when the HDD is spun up
from sleep).

Cheers, Andreas

Download attachment "signature.asc" of type "application/pgp-signature" (196 bytes)