linux-ext4 - Heads up: data corruption with AIO/DIO with unwritten blocks when blocksize

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Date:   Wed, 11 Nov 2020 14:13:51 -0500
From:   "Theodore Y. Ts'o" <tytso@....edu>
To:     Ritesh Harjani <riteshh@...ux.ibm.com>
Cc:     linux-ext4@...r.kernel.org
Subject: Heads up: data corruption with AIO/DIO with unwritten blocks when
 blocksize < page size

I noticed that we are occasionally seeing data corruption when block
size is less than page size and we are doing AIO/DIO with unwritten
regions.  Fortunately, in practice most applications which use AIO/DIO
make sure all of the data files are pre-zeroed and initialized --- we
only noticed this because of another bug where the pre-zeroing of the
blocks after fallocate(2) got accidentally skipped.

This can be seen using the following xfstests which I've attached to
this message.  It's not a reliable failure; the test is failing 70% of
the time, and in fact most individual AIO operations don't actually
corrupt data.  When it does fail, the file system is fine, but there
are checksum failures in the data blocks.

verify: bad magic header b587, wanted acca at file /xt-vdc/test-file offset 539779072, length 4198400 (requested block: offset=539779072, length=4198400)
       hdr_fail data dumped as test-file.539779072.hdr_fail
fio: pid=4891, err=84/file:verify.c:1444, func=async_verify, error=Invalid or incomplete multibyte or wide character
crc32c: verify failed at file /xt-vdc/test-file offset 2728620032, length 4198400 (requested block: offset=2728620032, length=4198400, flags=84)
       Expected CRC: e61a5d6a
       Received CRC: e390f0ed
       received data dumped as test-file.2728620032.received
       expected data dumped as test-file.2728620032.expected
crc32c: verify failed at file /xt-vdc/test-file offset 3029954560, length 4198400 (requested block: offset=3029954560, length=4198400, flags=84)
       Expected CRC: b56632a2
       Received CRC: 427982ce
       received data dumped as test-file.3029954560.received
       expected data dumped as test-file.3029954560.expected

Sample run:

TESTRUNID: tytso-20201110144948
KERNEL:    kernel 5.10.0-rc3-xfstests-00032-g91808cd6c243 #2012 SMP Mon Nov 9 21:37:55 EST 2020
x86_64
CMDLINE:   -C 20 generic/999

ext4/4k: 20 tests, 756 seconds
ext4/1k: 20 tests, 14 failures, 808 seconds
  Failures: generic/999 generic/999 generic/999 generic/999
    generic/999 generic/999 generic/999 generic/999 generic/999
    generic/999 generic/999 generic/999 generic/999 generic/999
ext4/ext3: 20 tests, 20 skipped, 83 seconds
ext4/encrypt: 20 tests, 20 skipped, 12 seconds
ext4/nojournal: 20 tests, 756 seconds
ext4/ext3conv: 20 tests, 753 seconds
ext4/adv: 20 tests, 751 seconds
ext4/dioread_nolock: 20 tests, 755 seconds
ext4/data_journal: 20 tests, 20 skipped, 17 seconds
ext4/bigalloc: 20 tests, 1049 seconds
ext4/bigalloc_1k: 20 tests, 762 seconds
Totals: 220 tests, 60 skipped, 14 failures, 0 errors, 6502s

I haven't had a chance to look at this, but I wanted to report this to
the ext4 list.  This is actually much better than before we moved to
using IOMAP for direct I/O, when we were failing this for all
conigurations(!) which support Direct I/O.

TESTRUNID: tytso-20201111100315
KERNEL:    kernel 4.19.84-xfstests #2 SMP Fri Nov 15 13:38:45 EST 2019 x86_64
CMDLINE:   -C 5 generic/999 --kernel gs://gce-xfstests/bzImage-4.19

ext4/4k: 5 tests, 5 failures, 184 seconds
  Failures: generic/999 generic/999 generic/999 generic/999 
    generic/999 
ext4/1k: 5 tests, 5 failures, 192 seconds
  Failures: generic/999 generic/999 generic/999 generic/999 
    generic/999 
ext4/ext3: 5 tests, 5 skipped, 13 seconds
ext4/encrypt: 5 tests, 5 skipped, 4 seconds
ext4/nojournal: 5 tests, 5 failures, 184 seconds
  Failures: generic/999 generic/999 generic/999 generic/999 
    generic/999 
ext4/ext3conv: 5 tests, 5 failures, 185 seconds
  Failures: generic/999 generic/999 generic/999 generic/999 
    generic/999 
ext4/adv: 5 tests, 5 failures, 182 seconds
  Failures: generic/999 generic/999 generic/999 generic/999 
    generic/999 
ext4/dioread_nolock: 5 tests, 5 failures, 184 seconds
  Failures: generic/999 generic/999 generic/999 generic/999 
    generic/999 
ext4/data_journal: 5 tests, 5 skipped, 5 seconds
ext4/bigalloc: 5 tests, 5 failures, 254 seconds
  Failures: generic/999 generic/999 generic/999 generic/999 
    generic/999 
ext4/bigalloc_1k: 5 tests, 5 failures, 186 seconds
  Failures: generic/999 generic/999 generic/999 generic/999 
    generic/999 
Totals: 55 tests, 15 skipped, 40 failures, 0 errors, 1573s

	   	     	      	 	   - Ted

View attachment "999" of type "text/plain" (1870 bytes)

View attachment "999.out" of type "text/plain" (64 bytes)