linux-kernel - Re: amd64 sata_nv (massive) memory corruption

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <489675DC.2080906@shaw.ca>
Date:	Sun, 03 Aug 2008 21:22:04 -0600
From:	Robert Hancock <hancockr@...w.ca>
To:	linasvepstas@...il.com
CC:	John Stoffel <john@...ffel.org>,
	Alistair John Strachan <alistair@...zero.co.uk>,
	linux-kernel@...r.kernel.org
Subject: Re: amd64 sata_nv (massive) memory corruption

Linas Vepstas wrote:
> What I don't like is that the corruption was utterly silent -- and disastereous:
> Originally, I had the sata disk paired to a pata disk in a RAID array, and the
> raid array was getting corrupted -- corrupted system files would get worse,
> as I tried reinstalling them. It took a while to realize that it was the sata
> disk, and it took a bit longer to realize it wasn't the disk itself, but the
> bad-ram-on-sata-channel.
> 
> So I'm wondering: can we devise a test to validate system-bus interactions
> like this? Clearly, the memtest86 test validates the RAM and the northbridge
> bus between CPU and system RAM, so that seems OK.

I wouldn't be sure about that..

> 
> I assume the sata controller is attached via pci or pci-e -- although the pci
> controller and the sata controller are on the same chip, (nVidia nForce 570
> chipset) so it may be an 'emulated' pci bus of some sort. The problem would
> seem to be some sort of bus timing issue between this particular RAM,
> and the pci bus in the chipset -- bad "eyes" on some signal line, or ground
> bounce or whatever, or maybe a rare chipset bug.

The SATA controller is part of the chipset, and I think it talks 
HyperTransport directly, it only looks like PCI or PCI Express. These 
systems have an on-die memory controller in the CPU, so the SATA 
controller has to talk HyperTransport to the CPU which then is what 
physically accesses the DIMMs.

In theory the DIMMs have no idea whether the accesses are from the CPU 
itself or from the chipset. However, it's possible that the particular 
timing or burst sizes of the transfers done by the SATA controller 
triggered a problem with marginal timing on the DIMMs and caused the 
data corruption.

> 
> So the question is: is there some sort of sata (or pci) "loopback mode",
> where we could pump data through all of the busses and controllers, up
> near to the point where it would normally go out to the serdes to the disk,
> but instead have it loop back, so that we could test the buses between
> endpoints?  I've never heard of a pci/pci-e loopback, but that doesn't mean it
> doesn't exist. I have no clue about SATA.  Is there possibly some ide or
> scsi command that can be used to loop-back? Some sort of "send bytes
> to disk, but don't actually write them to platter" command? Maybe just
> a write to some scratch ram on the disk drive itself? Even just a few bytes
> would be enough to implement a loopback test. Maybe some sort of
> "queue this block, but don't write it yet", followed by a "give me dump of
> the command queue" -- such a loopback test would have found my problem
> pretty quickly I suspect.
> 
> Ideas solicited.

I don't imagine that would be very useful in this case. The SATA link, 
PCI Express bus, HyperTransport bus all have parity or CRC error 
checking, so presumably they couldn't be likely to cause undetected 
errors. The transitions between them could cause problems, and most 
desktop machines don't have ECC memory which could catch memory timing 
problems or bad RAM (which is rather unfortunate), so those are the most 
likely places for a problem to show up.

> 
> --linas
> 
> p.s. the corruption appears to be single bits -- the rest of the word, and
> surrounding words, seem fine.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/