linux-kernel - Re: Linux 2.6.29

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Sat, 04 Apr 2009 08:40:03 -0400
From:	Mark Lord <lkml@....ca>
To:	Andrew Morton <akpm@...ux-foundation.org>
Cc:	Jeff Garzik <jeff@...zik.org>,
	Lennart Sorensen <lsorense@...lub.uwaterloo.ca>,
	torvalds@...ux-foundation.org, tytso@....edu, drees76@...il.com,
	jesper@...gh.cc, linux-kernel@...r.kernel.org
Subject: Re: Linux 2.6.29

Andrew Morton wrote:
> On Fri, 03 Apr 2009 14:59:12 -0400 Jeff Garzik <jeff@...zik.org> wrote:
> 
>> Lennart Sorensen wrote:
>>> On Fri, Apr 03, 2009 at 10:46:34AM -0400, Mark Lord wrote:
>>>> My Myth box here was running 2.6.18 when originally set up,
>>>> and even back then it still took *minutes* to delete large files.
>>>> So that part hasn't really changed much in the interim.
>>>>
>>>> Because of the multi-minute deletes, the distro shutdown scripts
>>>> would fails, and power off the box while it was still writing
>>>> to the drives.  Ouch.
>>>>
>>>> That system has had XFS on it for the past year and a half now,
>>>> and for Myth, there's no reason not to use XFS.  It's great!
>>> Mythtv has a 'slow delete' option that I believe works by slowly
>>> truncating the file.  Seems they believe that ext3 is bad at handling
>>> large file deletes, so they try to spread out the pain.  I don't remember
>>> if that option is on by default or not.  I turned it off.
>> It's pretty painful for super-large files with lots of metadata.
>>
> 
> yeah.
> 
> There's a dirty hack you can do where you append one byte to the file
> every 4MB, across 1GB (say).  That will then lay the file out on-disk as
> 
> one bitmap block
> one data block
> one bitmap block
> one data block
> one bitmap block
> one data block
> one bitmap block
> one data block
> <etc>
> lots-of-data-blocks
> 
> So when the time comes to delete that gigabyte, the bitmaps blocks are
> only one block apart, and reading them is much faster.
> 
> That was one of the gruesome hacks I did way back when I was in the
> streaming video recording game.
> 
> Another was the slow-delete thing.
> 
> - open the file
> 
> - unlink the file
> 
> - now sit in a loop, slowly nibbling away at the tail with
>   ftruncate() until the file is gone.
> 
> The open/unlink was there so that if the system were to crash midway,
> ext3 orphan recovery at reboot time would fully delete the remainder of
> the file.
..

That's similar to what Mythtv currently does.
Except it nibbles away in painfully tiny chunks,
so deleting takes hours that way.

Which means it's still in progress when the system
auto-shutdowns between uses.  So the delete process
gets killed, and the subsequent remount,ro and umount
calls simply fail (fs is still busy), and it then
powers off while the drive light is still solidly busy.

That's where I modified the shutdown script to check
the result code, sleep, and loop again, for up to five
minutes before pulling the plug.

But switching to xfs cured all of that.  :)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/