linux-kernel - RE: [BUG] Possible silent data corruption in filesystems/page cache

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 6 Jun 2016 07:29:42 +0000
From:	"Barczak, Mariusz" <mariusz.barczak@...el.com>
To:	Andreas Dilger <adilger@...ger.ca>
CC:	Andrew Morton <akpm@...ux-foundation.org>,
	Jens Axboe <axboe@...nel.dk>,
	Alexander Viro <viro@...iv.linux.org.uk>,
	"linux-mm@...ck.org" <linux-mm@...ck.org>,
	"linux-block@...r.kernel.org" <linux-block@...r.kernel.org>,
	"linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"Wysoczanski, Michal" <michal.wysoczanski@...el.com>,
	"Baldyga, Robert" <robert.baldyga@...el.com>,
	"Roman, Agnieszka" <agnieszka.roman@...el.com>
Subject: RE: [BUG] Possible silent data corruption in filesystems/page cache

Hi, Let me elaborate problem in detail. 

For buffered IO data are copied into memory pages. For this case,
the write IO is not submitted (generally). In the background opportunistic
cleaning of dirty pages takes place and IO is generated to the
device. An IO error is observed on this path and application
is not informed about this. Summarizing flushing of dirty page fails.
And probably, this page is dropped but in fact it should not be.
So if above situation happens between application write and sync
then no error is reported. In addition after some time, when the
application reads the same LBA on which IO error occurred, old data
content is fetched.

We did own fault injector in order to do error in specific condition
described above.

Regards,
Mariusz.

-----Original Message-----
From: Andreas Dilger [mailto:adilger@...ger.ca] 
Sent: Thursday, June 2, 2016 21:32
To: Barczak, Mariusz <mariusz.barczak@...el.com>
Cc: Andrew Morton <akpm@...ux-foundation.org>; Jens Axboe <axboe@...nel.dk>; Alexander Viro <viro@...iv.linux.org.uk>; linux-mm@...ck.org; linux-block@...r.kernel.org; linux-fsdevel@...r.kernel.org; linux-kernel@...r.kernel.org; Wysoczanski, Michal <michal.wysoczanski@...el.com>; Baldyga, Robert <robert.baldyga@...el.com>; Roman, Agnieszka <agnieszka.roman@...el.com>
Subject: Re: [BUG] Possible silent data corruption in filesystems/page cache

On Jun 1, 2016, at 3:51 AM, Barczak, Mariusz <mariusz.barczak@...el.com> wrote:
> 
> We run data validation test for buffered workload on filesystems:
> ext3, ext4, and XFS.
> In context of flushing page cache block device driver returned IO error.
> After dropping page cache our validation tool reported data corruption.

Hi Mariusz,
it isn't clear what you expect to happen here?  If there is an IO error then the data is not written to disk and cannot be correct when read.

The expected behaviour is the IO error will either be returned immediately at write() time (this used to be more common with older filesystems), or it will be returned when calling sync() on the file to flush cached data to disk.

> We provided a simple patch in order to inject IO error in device mapper.
> We run test to verify md5sum of file during IO error.
> Test shows checksum mismatch.
> 
> Attachments:
> 0001-drivers-md-dm-add-error-injection.patch - device mapper patch

There is already the dm-flakey module that allows injecting errors into the IO path.

Cheers, Andreas

--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.