| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Tue, 27 Nov 2012 18:14:42 +0200 From: Marcus Sundman <marcus@...ox.fi> To: Jan Kara <jack@...e.cz> CC: linux-kernel@...r.kernel.org Subject: Re: Debugging system freezes on filesystem writes On 22.11.2012 01:30, Jan Kara wrote: > On Fri 16-11-12 03:11:22, Marcus Sundman wrote: >> On 13.11.2012 15:51, Jan Kara wrote: >>> On Fri 09-11-12 15:12:43, Marcus Sundman wrote: >>>> On 09.11.2012 01:41, Marcus Sundman wrote: >>>>> On 07.11.2012 18:17, Jan Kara wrote: >>>>>> On Fri 02-11-12 04:19:24, Marcus Sundman wrote: >>>>>>> Also, and this might be important, according to iotop there is >>>>>>> almost no disk writing going on during the freeze. (Occasionally >>>>>>> there are a few MB/s, but mostly it's 0-200 kB/s.) Well, at least >>>>>>> when an iotop running on nice -20 hasn't frozen completely, which it >>>>>>> does during the more severe freezes. >>>>>> OK, it seems as if your machine has some problems with memory >>>>>> allocations. Can you capture /proc/vmstat before the freeze and >>>>>> after the >>>>>> freeze and send them for comparison. Maybe it will show us what is the >>>>>> system doing. >>>>> t=01:06 http://sundman.iki.fi/vmstat.pre-freeze.txt >>>>> t=01:08 http://sundman.iki.fi/vmstat.during-freeze.txt >>>>> t=01:12 http://sundman.iki.fi/vmstat.post-freeze.txt >>>> Here are some more vmstats: >>>> http://sundman.iki.fi/vmstats.tar.gz >>>> >>>> They are from running this: >>>> while true; do cat /proc/vmstat > "vmstat.$(date +%FT%X).txt"; sleep >>>> 10; done >>>> >>>> There were lots and lots of freezes for almost 20 mins from 14:37:45 >>>> onwards, pretty much constantly, but at 14:56:50 the freezes >>>> suddenly stopped and everything went back to how it should be. >>> I was looking into the data but they didn't show anything problematic. >>> The machine seems to be writing a lot but there's always some free memory, >>> even direct reclaim isn't ever entered. Hum, actually you wrote iotop isn't >>> showing much IO going on but vmstats show there is about 1 GB written >>> during the freeze. It is not a huge amount given the time span but it >>> certainly gives a few MB/s of write load. >> I didn't watch iotop during this particular freeze. I'll try to keep >> an eye on iotop in the future. Is there some particular options I >> should run iotop with, or is a "nice -n -20 iotop -od3" fine? > I'm not really familiar with iotop :). Usually I use iostat... OK, which options for iostat should I use then? :) >>> There's surprisingly high number of allocations going on but that may be >>> due to the IO activity. So let's try something else: Can you switch to >>> console and when the hang happens press Alt-Sysrq-w (or you can just do >>> "echo w >/proc/sysrq-trigger" if the machine is live enough to do that). >>> Then send me the output from dmesg. Thanks! >> Sure! Here are two: >> http://sundman.iki.fi/dmesg-1.txt >> http://sundman.iki.fi/dmesg-2.txt > Thanks for those and sorry for the delay (I was busy with other stuff). > I had a look into those traces and I have to say I'm not much wiser. In the > first dump there is just kswapd waiting for IO. In the second dump there > are more processes waiting for IO (mostly for reads - nautilus, > thunderbird, opera, ...) but nothing really surprising. So I'm lost what > could cause the hangs you observe. Yes, mostly it's difficult to trigger the sysrq thingy, because by the time I manage to switch to the console or running that echo to proc in a terminal the worst is already over. > Recalling you wrote even simple programs > like top hang, maybe it is some CPU scheduling issue? Can you boot with > noautogroup kernel option? Sure. I've been running with noautogroup for almost a week now, but no big change one way or the other. (E.g., it's still impossible to listen to music, because the songs will start skipping/looping several times during each song even if there isn't any big "hang" happening. And uncompressing a 100 MB archive (with nice '19' and ionice 'idle') is still, after a while, followed by a couple of minutes of superhigh I/O wait causing everything to become really slow.) - Marcus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists