linux-kernel - Re: Debugging system freezes on filesystem writes

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20121113135159.GA18651@quack.suse.cz>
Date:	Tue, 13 Nov 2012 14:51:59 +0100
From:	Jan Kara <jack@...e.cz>
To:	Marcus Sundman <marcus@...ox.fi>
Cc:	Jan Kara <jack@...e.cz>, linux-kernel@...r.kernel.org
Subject: Re: Debugging system freezes on filesystem writes

On Fri 09-11-12 15:12:43, Marcus Sundman wrote:
> On 09.11.2012 01:41, Marcus Sundman wrote:
> >On 07.11.2012 18:17, Jan Kara wrote:
> >>On Fri 02-11-12 04:19:24, Marcus Sundman wrote:
> >>>Also, and this might be important, according to iotop there is
> >>>almost no disk writing going on during the freeze. (Occasionally
> >>>there are a few MB/s, but mostly it's 0-200 kB/s.) Well, at least
> >>>when an iotop running on nice -20 hasn't frozen completely, which it
> >>>does during the more severe freezes.
> >>   OK, it seems as if your machine has some problems with memory
> >>allocations. Can you capture /proc/vmstat before the freeze and
> >>after the
> >>freeze and send them for comparison. Maybe it will show us what is the
> >>system doing.
> >
> >t=01:06 http://sundman.iki.fi/vmstat.pre-freeze.txt
> >t=01:08 http://sundman.iki.fi/vmstat.during-freeze.txt
> >t=01:12 http://sundman.iki.fi/vmstat.post-freeze.txt
> 
> Here are some more vmstats:
> http://sundman.iki.fi/vmstats.tar.gz
> 
> They are from running this:
> while true; do cat /proc/vmstat > "vmstat.$(date +%FT%X).txt"; sleep
> 10; done
> 
> There were lots and lots of freezes for almost 20 mins from 14:37:45
> onwards, pretty much constantly, but at 14:56:50 the freezes
> suddenly stopped and everything went back to how it should be.
  I was looking into the data but they didn't show anything problematic.
The machine seems to be writing a lot but there's always some free memory,
even direct reclaim isn't ever entered. Hum, actually you wrote iotop isn't
showing much IO going on but vmstats show there is about 1 GB written
during the freeze. It is not a huge amount given the time span but it
certainly gives a few MB/s of write load.

There's surprisingly high number of allocations going on but that may be
due to the IO activity. So let's try something else: Can you switch to
console and when the hang happens press Alt-Sysrq-w (or you can just do
"echo w >/proc/sysrq-trigger" if the machine is live enough to do that).
Then send me the output from dmesg.  Thanks!

								Honza
-- 
Jan Kara <jack@...e.cz>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/