linux-kernel - Re: [PATCH] fs: limit maximum concurrent coredumps

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Tue, 22 Jun 2010 11:18:15 +0900 (JST)
From:	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>
To:	Edward Allcutt <edward@...cutt.me.uk>
Cc:	kosaki.motohiro@...fujitsu.com,
	Alexander Viro <viro@...iv.linux.org.uk>,
	Randy Dunlap <rdunlap@...otime.net>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Jiri Kosina <jkosina@...e.cz>,
	Dave Young <hidave.darkstar@...il.com>,
	Martin Schwidefsky <schwidefsky@...ibm.com>,
	"H. Peter Anvin" <hpa@...or.com>, Oleg Nesterov <oleg@...hat.com>,
	Neil Horman <nhorman@...driver.com>,
	Roland McGrath <roland@...hat.com>,
	Ingo Molnar <mingo@...e.hu>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	"Eric W. Biederman" <ebiederm@...ssion.com>,
	linux-kernel@...r.kernel.org, linux-doc@...r.kernel.org,
	linux-fsdevel@...r.kernel.org
Subject: Re: [PATCH] fs: limit maximum concurrent coredumps

> A moderately large number of large core files being dumped simultaneously
> can impose severe latency penalties on other processes due to IO load.
> 
> For example, a common configuration for PHP web-servers includes apache's
> prefork MPM, mod_php and a PHP opcode cache utilizing shared memory. In
> certain failure modes, all requests serviced by PHP result in a segfault.
> Enabling coredumps might lead to 10-20 coredumps per second, all attempting
> to write a 150-200MB core file. This leads to the whole system becoming
> entirely unresponsive for many minutes.
> 
> The ability to limit concurrent coredumps allows dumping core to be safely
> enabled in these situations without affecting responsiveness of the system
> as a whole.
> 
> I have several servers running with this patch applied (actually backported
> to v2.6.26) and it has allowed me to deal successfully with the situation
> described above.
> 
> In do_coredump I have pulled dump_count back out to the top-level scope
> and core_dump_count is now incremented in the normal path.
> 
> Added sysctl parameter for tuning core_max_concurrency.
> 
> checkpatch complains about "extern ... core_max_concurrency" but this seems
> to be an acceptable exception based on the preponderance of other extern
> variables in kernel/sysctl.c

To be honest, My customers have very similar problem for long time. If 
HPC MPI job or apache prefork MPM processces die, The system load to 
make large concurrent core dump I/O and consume all I/O band width. 
Eventually, all other service stall long time and HA cluster software
shutdown such node forcely.

My case used two technique, 1) limit concurrent coredump (but not skip
as sugessted Roland) 2) reduce I/O priority automatically in do_coredump().
I don't know my way was correct way. but I believe we need something case.



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/