linux-kernel - Re: 2.6.28.9: EXT3/NFS inodes corruption

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <6278d2220908210405n35778e04mc1728f2c37ef0c0f@mail.gmail.com>
Date:	Fri, 21 Aug 2009 12:05:10 +0100
From:	Daniel J Blueman <daniel.blueman@...il.com>
To:	Sylvain Rochet <gradator@...dator.net>
Cc:	Linux Kernel <linux-kernel@...r.kernel.org>
Subject: Re: 2.6.28.9: EXT3/NFS inodes corruption

Hi Sylvain,

On Tue, Jul 28, 2009 at 5:45 PM, Sylvain Rochet<gradator@...dator.net> wrote:
> On Tue, Jul 28, 2009 at 09:40:42AM -0700, Daniel J Blueman wrote:
>> On Apr 20, 5:30 pm, Sylvain Rochet <grada...@...dator.net> wrote:
>> > Hi,
>> >
>> > We(TuxFamily) are having some inodes corruptions on a NFS server.
>> >
>> > So, let's start with the facts.
>> >
>> > ==== NFS Server
>> >
>> > Linux bazooka 2.6.28.9 #1 SMP Mon Mar 30 12:58:22 CEST 2009 x86_64 GNU/Linux
>> [snip]
>>
>> Can you do a 'lspci -v' on the server please?
>
> Of course yes.
>
> I attached the 'lspci -v' of the previous and the current storage
> server.

The reason I ask, I was chasing data corruption across the PCIe bus
with some high-performance Quadrics interconnect adapters a while ago.
The reproducer involved multiple outstanding main memory read requests
to related addresses and a small block of data would be returned from
the wrong offset.

In the end, I found the nVidia CK804 (also MCP55) HT->PCIe bridge was
at fault and later found disk corruption when doing heavy rsyncs to
network. This was never publicly acknowledged, but I guess it
illustrates the need for some micro-tests to verify data-soundness
under duress; it took a day (and petabytes of data) of the production
I/O workload to get this data corruption, and 3 seconds with the right
reproducer, (still non-trivial to catch on a PCIe protocol analyser).

Sometime I'll develop a stress-test driver for a common SATA or
network controller to drive it's DMA engine with I/O patterns to and
from main memory, checking the data integrity every few seconds; this
could be generalised with OpenGL nicely for graphics cards on
workstations I imagine.

Daniel
-- 
Daniel J Blueman
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/