linux-kernel - Re: I/O blocked while dirty pages are being flushed

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20130324065605.GA19984@dcvr.yhbt.net>
Date:	Sun, 24 Mar 2013 06:56:06 +0000
From:	Eric Wong <normalperson@...t.net>
To:	Fredrik Tolf <fredrik@...da2000.com>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: I/O blocked while dirty pages are being flushed

Fredrik Tolf <fredrik@...da2000.com> wrote:
> It is worth noting, also, that this seems to be a situation
> introduced somewhere between 2.6.26 and 2.6.32, because I started
> noticing it when I upgraded from Debian 5.0 to 6.0. I've since tried
> it on 3.2.0, 3.5.4 and 3.7.1, and it appears in every version.
> However, I can't easily go back and bisect, because the new init
> scripts don't support kernels older than 2.6.32, unfortunately.

I'm not sure about Debian-specific changes to the kernel, but
in the stock kernel, the dirty*ratios changes could affect you:

before 2.6.22  dirty_ratio=40 dirty_background_ratio=10
2.6.22-2.6.29  dirty_ratio=10 dirty_background_ratio=5
2.6.30-...     dirty_ratio=20 dirty_background_ratio=10

So try lowering these sysctls to 2.6.26 levels (or lower) and see if
that helps.

Fwiw, I usually use dirty_ratio=2 dirty_background_ratio=1 on servers
with a few gigs of RAM (or appropriately low dirty*bytes values).

Lowering dirty*ratio helps servers get more consistent performance under
constant I/O pressure and aggressively throttles processes before a
large amount of dirty pages becomes a problem (as you've noticed).

High dirty*ratio is good for some bursty desktop workloads and some
benchmarks, though...

ref: commit 07db59bd6b0f279c31044cba6787344f63be87ea
ref: commit 1b5e62b42b55c509eea04c3c0f25e42c8b35b564

Heck, on a particularly bad server (2.6.18, pre-dirty_*bytes sysctl)
with lots of RAM and horrible disk throughput (~10M/s), I set both
dirty_writeback_centisecs and dirty_expire_centisecs to 100 to get
acceptable performance for writing HTTP access logs.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/