linux-ext4 - Re: [ext3][kernels >= 2.6.20.7 at least] KDE going comatose when FS is under heavy write load (massive starvation)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Fri, 17 Aug 2007 02:02:53 -0700
From:	Andrew Morton <akpm@...ux-foundation.org>
To:	Alex Tomas <alex@...sterfs.com>
Cc:	"linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>
Subject: Re: [ext3][kernels >= 2.6.20.7 at least] KDE going comatose when FS
 is under heavy write load (massive starvation)

On Fri, 17 Aug 2007 12:36:32 +0400 Alex Tomas <alex@...sterfs.com> wrote:

> Andrew Morton wrote:
> > Sort-of.  But the per-superpblock, per-inode writeback code is pretty
> > careful to avoid livelocks.  The per-inode writeback is a strict single
> > linear sweep across the file.  It'll basically write out anything which was
> > dirty when it was called.  The per-superblock inode walk isn't as accurate
> > as that, becuase of the difficulties of juggling list_heads.  But we're
> > slowly working on that, and I suspect it'll be ggod enough for ext3
> > purposes already.
> 
> I'd say that these are two different mechanism solving different problems:
> 1) VFS/MM does periodic updates and uses regular writeback
> 2) data=ordered is to avoid metadata pointing to not-written-yet data

VFS/MM can do _much_ more than that!  Look at struct writeback_control.

That code path has many different modes of operation: it is used for
regular pdflush writeback, sync, fsync, throttling, etc.  Probably one of
its modes will be sufficient.  If we want to change ext3's existing
semantics and add an "only writeback uninitialised blocks" mode then
that'll be pretty straightforward: add more control information to
writeback_control and go for it.

> we can't use regular writeback in commit thread as long as it can fall into
> allocation. so, we'd have to add one more WB mode (btw, i have a patch which
> skips non-allocated blocks in writeback if special WB mode is requested).

yup

> OTOH, the faster we go through data sync part of commit, the better. given
> that lots of inodes can be dirty with no data to sync, it's going to take
> long in some cases. it's especially bad because commit doesn't scale to many
> CPUs.

eh?

> also, why would we need to flush *everything* every 5s? just because ext3 does
> this? sounds strange. if somebody really need this we could add this possibility
> to regular writeback path (making it tunable). but I'd rather prefer to have
> a separate (fast, lightweight, scalable) mechanism to support data=ordered.
> 

Yeah, that would make sense, perhaps.

Or just speed the existing stuff up.  iirc the main problem in there is unrelated
to data writeback.  There are situations where the running transaction has to block
behind metadata writeout which the committing transaction is performing.  I
reluctantly put that in years ago to get us out of a tight spot and it
never got optimised.


-
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html