linux-ext4 - Re: Query about ext4 commit interval vs dirty_expire

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAMoswegmo08i-7TMpbM7x=RHiRsu-g40Vq2wmPzYsx7=gCi5MA@mail.gmail.com>
Date:   Tue, 17 Dec 2019 14:42:48 +0000
From:   Paul Richards <paul.richards@...il.com>
To:     Jan Kara <jack@...e.cz>
Cc:     linux-ext4@...r.kernel.org
Subject: Re: Query about ext4 commit interval vs dirty_expire_centisecs

On Fri, 13 Dec 2019 at 15:59, Jan Kara <jack@...e.cz> wrote:
>
> Hello!
>
> On Tue 19-11-19 08:47:31, Paul Richards wrote:
> > I'm trying to understand the interaction between the ext4 `commit`
> > interval option, and the `vm.dirty_expire_centisecs` tuneable.
> >
> > The ext4 `commit` documentation says:
> >
> > > Ext4 can be told to sync all its data and metadata every 'nrsec' seconds. The default value is 5 seconds. This means that if you lose your power, you will lose as much as the latest 5 seconds of work (your filesystem will not be damaged though, thanks to the journaling).
> >
> > The `dirty_expire_centisecs` documentation says:
> >
> > > This tunable is used to define when dirty data is old enough to be eligible for writeout by the kernel flusher threads. It is expressed in 100'ths of a second. Data which has been dirty in-memory for longer than this interval will be written out next time a flusher thread wakes up.
> >
> >
> > Superficially these sound like they have a very similar effect.  They
> > periodically flush out data that hasn't been explicitly fsync'd by the
> > application.  I'd like to understand a bit more the interaction
> > between these.
>
> Yes, the effect is rather similar but not quite the same. The first thing
> to observe is kind of obvious fact that ext4 commit interval influences
> just the particular filesystem while dirty_expire_centisecs influences
> behavior of global writeback over all filesystems.
>
> Secondly, commit interval is really the maximum age of ext4 transation.  So
> if there is metadata change pending in the journal, it will become
> persistent at latest after this time. So for say 'mkdir' that will be
> persistent at latest after this time. For data operations things are more
> complex. E.g. when delayed allocation is used (which is the default), the
> change gets logged in the journal only during writeback. So it can take up
> to dirty_expire_centisecs for data to be written back from page cache, that
> results in filesystem journalling block allocations etc. and then it can
> take upto commit interval for these changes to become persistent. So in
> this case the intervals add up. There are also other special cases
> somewhere in between but generally it is reasonable to assume that data gets
> automatically persistent in dirty_expire_centisecs + commit_interval time.
> Note both these times are actually times when writeback is triggered so
> if the disk gets too busy, the actual time when data is completely on disk
> may be much higher.
>

Thanks for taking the time to reply!

Since automatic persisting of data occurs only after
dirty_expire_centisecs + commit_interval,
should the ext4 docs be corrected?  They currently state (for the
commit interval option):

"The default value is 5 seconds. This means that if you lose
your power, you will lose as much as the latest 5 seconds of work"