linux-kernel - Slow long-term increase in dirty pages

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20090318145347.3d709de8@nacelle.mxtelecom.com>
Date:	Wed, 18 Mar 2009 14:53:47 +0000
From:	Paul Evans <paul@...elecom.com>
To:	linux-kernel@...r.kernel.org
Subject: Slow long-term increase in dirty pages

We have a server whose dirty page count keeps increasing all the time,
to the point where 'sync' takes ages to flush the pages:

  root@...ehand:~# time sync

  real    1m15.570s
  user    0m0.000s
  sys     0m0.052s

We have some graphs of the dirty page count, as captured
from /proc/vmstat's "nr_dirty" entry:

  http://opensource.mxtelecom.com/tmp/freehand-dirty-day.png
  http://opensource.mxtelecom.com/tmp/freehand-dirty-week.png

I have tuned the dirty page flushing sysctls to the following:

  root@...ehand:~# for F in /proc/sys/vm/dirty_*; do echo -n "$F: "; cat $F; done
  /proc/sys/vm/dirty_background_ratio: 1
  /proc/sys/vm/dirty_expire_centisecs: 3000
  /proc/sys/vm/dirty_ratio: 3
  /proc/sys/vm/dirty_writeback_centisecs: 500

The role of the machine itself is that it performings large amount of
kernel iptables routing/firewalling traffic, and runs a set of apache
servers as HTTP<->Tomcat gateways.

  root@...ehand:~# uname -r
  2.6.27-fes

(this is a build of stock 2.6.27 source, with some extra iptables
patches. There shouldn't be anything mm-related here)

By my understanding of the dirty page flush algorithm, we shouldn't be
accumulating these pages all the time; any page older than 30 seconds
ought to be written out, yes?

If we manually 'sync', as above, then the count drops to zero, but then
slowly starts ramping up again as observed.

As a temporary workaround I've put 'sync' in cron every 10 minutes, but
is there some more tuning I can do; or at least probing to see where
these pages are being accumulated from?

-- 
Paul Evans <paul@...elecom.com>
Tel: +44 (0) 845 666 7778
Fax: +44 (0) 870 163 4694
http://www.mxtelecom.com

Download attachment "signature.asc" of type "application/pgp-signature" (198 bytes)