linux-ext4 - Re: DMAR regression in 2.6.31 leads to ext4 corruption?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Fri, 9 Oct 2009 17:09:26 -0700
From:	Chris Wright <chrisw@...s-sol.org>
To:	Andy Isaacson <adi@...apodia.org>
Cc:	linux-kernel@...r.kernel.org, linux-ext4@...r.kernel.org,
	iommu@...ts.linux-foundation.org
Subject: Re: DMAR regression in 2.6.31 leads to ext4 corruption?

* Andy Isaacson (adi@...apodia.org) wrote:
> Today while running 817b33d38 I got the following (on a Thinkpad X200
> I'd replaced the Dell with, just in case it was previously-good hardware
> going bad).
> 
> [   29.450550] EXT4-fs error (device sda1): ext4_lookup: deleted inode referenced: 79
> [   30.022328] DRHD: handling fault status reg 3
> [   30.022328] DMAR:[DMA Write] Request device [00:02.0] fault addr ddae28000 
> [   30.022328] DMAR:[fault reason 05] PTE Write access is not set
> [   30.146136] DRHD: handling fault status reg 3
> [   30.248938] DMAR:[DMA Write] Request device [00:02.0] fault addr ddae28000 
> [   30.248939] DMAR:[fault reason 05] PTE Write access is not set

There's some timing coincidence there, but it's a full 1/2 second between
the ext4 error and the DMAR fault (and there's various DMAR faults
along the way for the same buffer before and after the ext4 error).
That fault is quite typical of a driver bug, and it's the VGA device
(rather its driver) that is culpable.  The IOMMU caught the VGA device
trying to do a DMA write to a buffer mapped r/o.

> The full output of fsck and full dmesg are at the URL below.
> 
> I don't know that DMAR is resulting in my repeated filesystem
> corruption, but it does seem like a potential cause (and would explain
> why I'm seeing this whereas most people aren't, since few people are
> using VT-d *and* i915).

I do use it every day on my primary workstation (x200), and haven't
had any issue (I'm using ext3).

> I see that the BROKEN_GFX_WA code has been removed; do we actually
> believe that the relevant code is working?  Could it be corrupting my
> AHCI DMAs if not?

It should be for your adapter (after 66a4fe0c merged in agp fixes).
While it could still be broken (aside of the initial faults before the
device is even initialized in Linux -- I'm not seeing any faults, btw),
iommu=pt will put all devices in a 1:1 mapped domain and would suppress
the DMAR faults you see (similar to intel_iommu=off, but allowing the
iommu to still be used for pci device assignment).  However, doing that
or enabling the gfx workaround would allow the device to generate invalid
DMA requests since if effectively disables the IOMMU for the gfx device,
which would leave a better opportunity for DMA related corruption.
The earlier fs issues we saw w/ the IOMMU were when it was actively
blocking disk DMA requests, but that's not happening here.

> At the end of the last thread Ted thought that we'd
> lost a write of an inode block; this time the symptoms look different,
> in that I don't see one inode block representing a significant data
> loss (though I'm by no means an expert).
> 
> Complete dmesg etc are at
> http://web.hexapodia.org/~adi/bugs/20091008-ext4-dmar/
> 
> I'll try running with BROKEN_GFX_WA turned back on and see if that
> improves things at all.

thanks,
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html