linux-kernel - Re: Debugging system freezes on filesystem writes

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20130912131051.GA14664@quack.suse.cz>
Date:	Thu, 12 Sep 2013 15:10:51 +0200
From:	Jan Kara <jack@...e.cz>
To:	Marcus Sundman <marcus@...ox.fi>
Cc:	Jan Kara <jack@...e.cz>, Theodore Ts'o <tytso@....edu>,
	Dave Chinner <david@...morbit.com>,
	linux-kernel@...r.kernel.org
Subject: Re: Debugging system freezes on filesystem writes

On Thu 12-09-13 15:57:32, Marcus Sundman wrote:
> On 27.02.2013 01:17, Jan Kara wrote:
> >On Tue 26-02-13 20:41:36, Marcus Sundman wrote:
> >>On 24.02.2013 03:20, Theodore Ts'o wrote:
> >>>On Sun, Feb 24, 2013 at 11:12:22AM +1100, Dave Chinner wrote:
> >>>>>>/dev/sda6 /home ext4 rw,noatime,discard 0 0
> >>>>                                    ^^^^^^^
> >>>>I'd say that's your problem....
> >>>Looks like the Sandisk U100 is a good SSD for me to put on my personal
> >>>"avoid" list:
> >>>
> >>>http://thessdreview.com/our-reviews/asus-zenbook-ssd-review-not-necessarily-sandforce-driven-shows-significant-speed-bump/
> >>>
> >>>There are a number of SSD's which do not implement "trim" efficiently,
> >>>so these days, the recommended way to use trim is to run the "fstrim"
> >>>command out of crontab.
> >>OK. Removing 'discard' made it much better (the 60-600 second
> >>freezes are now 1-50 second freezes), but it's still at least an
> >>order of magnitude worse than a normal HD. When writing, that is --
> >>reading is very fast (when there's no writing going on).
> >>
> >>So, after reading up a bit on this trimming I'm thinking maybe my
> >>filesystem's block sizes don't match up with my SSD's blocks (or
> >>whatever its write unit is called). Then writing a FS block would
> >>always write to multiple SSD blocks, causing multiple
> >>read-erase-write sequences, right? So how can I check this, and how
> >>can I make the FS blocks match the SSD blocks?
> >   As Ted wrote, alignment isn't usually a problem with SSDs. And even if it
> >was, it would be at most a factor 2 slow down and we don't seem to be at
> >that fine grained level :)
> >
> >At this point you might try mounting the fs with nobarrier mount option (I
> >know you tried that before but without discard the difference could be more
> >visible), switching IO scheduler to CFQ (for crappy SSDs it actually isn't
> >a bad choice), and we'll see how much we can squeeze out of your drive...
> 
> I repartitioned the drive and reinstalled ubuntu and after that it
> gladly wrote over 100 MB/s to the SSD without any hangs. However,
> after a couple of months I noticed it had degraded considerably, and
> it keeps degrading. Now it's slowly becoming completely unusable
> again, with write speeds of the magnitude 1 MB/s and dropping.
> 
> As far as I can tell I have not made any relevant changes. Also, the
> amount of free space hasn't changed considerably, but it seems that
> the longer it's been since I reformatted the drive the more free
> space is required for it to perform well.
> 
> So, maybe the cause is fragmentation? I tried running e4defrag and
> then fstrim, but it didn't really help (well, maybe a little bit,
> but after a couple of days it was back in unusable-land). Also,
> "e4defrag -c" gives a fragmenation score of less than 5, so...
> 
> Any ideas?
  So now you run without 'discard' mount option, right? My guess then would
be that the FTL layer on your SSD is just crappy and as the erase blocks
get more fragmented as the filesystem is used it cannot keep up. But it's
easy to put blame on someone else :)

You can check whether this is a problem of Linux or your SSD by writing a
large file (few GB or more) like 'dd if=/dev/zero of=testfile bs=1M
count=4096 oflag=direct'. What is the throughput? If it is bad, check output
of 'filefrag -v testfile'. If the extents are reasonably large (1 MB and
more), then the problem is in your SSD firmware. Not much we can do about
it in that case...

If it really is SSD's firmware, maybe you could try f2fs or similar flash
oriented filesystem which should put lower load on the disk's FTL.

								Honza
-- 
Jan Kara <jack@...e.cz>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/