linux-ext4 - Re: Query about ext4 commit interval vs dirty_expire

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Fri, 13 Dec 2019 16:59:12 +0100
From:   Jan Kara <jack@...e.cz>
To:     Paul Richards <paul.richards@...il.com>
Cc:     linux-ext4@...r.kernel.org
Subject: Re: Query about ext4 commit interval vs dirty_expire_centisecs

Hello!

On Tue 19-11-19 08:47:31, Paul Richards wrote:
> I'm trying to understand the interaction between the ext4 `commit`
> interval option, and the `vm.dirty_expire_centisecs` tuneable.
> 
> The ext4 `commit` documentation says:
> 
> > Ext4 can be told to sync all its data and metadata every 'nrsec' seconds. The default value is 5 seconds. This means that if you lose your power, you will lose as much as the latest 5 seconds of work (your filesystem will not be damaged though, thanks to the journaling).
> 
> The `dirty_expire_centisecs` documentation says:
> 
> > This tunable is used to define when dirty data is old enough to be eligible for writeout by the kernel flusher threads. It is expressed in 100'ths of a second. Data which has been dirty in-memory for longer than this interval will be written out next time a flusher thread wakes up.
> 
> 
> Superficially these sound like they have a very similar effect.  They
> periodically flush out data that hasn't been explicitly fsync'd by the
> application.  I'd like to understand a bit more the interaction
> between these.

Yes, the effect is rather similar but not quite the same. The first thing
to observe is kind of obvious fact that ext4 commit interval influences
just the particular filesystem while dirty_expire_centisecs influences
behavior of global writeback over all filesystems.

Secondly, commit interval is really the maximum age of ext4 transation.  So
if there is metadata change pending in the journal, it will become
persistent at latest after this time. So for say 'mkdir' that will be
persistent at latest after this time. For data operations things are more
complex. E.g. when delayed allocation is used (which is the default), the
change gets logged in the journal only during writeback. So it can take up
to dirty_expire_centisecs for data to be written back from page cache, that
results in filesystem journalling block allocations etc. and then it can
take upto commit interval for these changes to become persistent. So in
this case the intervals add up. There are also other special cases
somewhere in between but generally it is reasonable to assume that data gets
automatically persistent in dirty_expire_centisecs + commit_interval time.
Note both these times are actually times when writeback is triggered so
if the disk gets too busy, the actual time when data is completely on disk
may be much higher.

								Honza

-- 
Jan Kara <jack@...e.com>
SUSE Labs, CR