linux-kernel - Re: Reproducible data corruption with sendfile+vsftp

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <474FC4D9.3020506@cosmosbay.com>
Date:	Fri, 30 Nov 2007 09:07:53 +0100
From:	Eric Dumazet <dada1@...mosbay.com>
To:	Holger Hoffstaette <holger@...ards.de>
CC:	linux-kernel@...r.kernel.org,
	Linux Netdev List <netdev@...r.kernel.org>
Subject: Re: Reproducible data corruption with sendfile+vsftp - splice regression?

Holger Hoffstaette a écrit :
> Hi -
> 
> This regular Linux user and lkml lurker just noticed data corruption in
> ftp'ed files and narrowed it down to vsftpd using sendfile(). So far this
> has never caused problems in the past; I have not noticed this with
> 2.6.22.x but may have missed it. I do remember reading about some changes
> to the underlying splice stuff since .23 so that may have something to do
> with it.
> 
> The scenario:
> 
> - created a file with known bit pattern on Linux server
> - ftp-got this file to Windows client: file has bad crc (yes, binary)
> - verified with another client: same result
> 
> I have thus far eliminated (to the best of my knowledge) NICs, switches,
> cables, the Windows FTP clients, the hard disk in the server (SATA, ext3):
> nothing suspicious in any logs. Box is an AMD Sempron 2600+ with 1.5 GB
> RAM, added rt8169 card, Gentoo, vsftpd stable 2.0.5 - nothing fancy.
> Transferring the file with samba (interestingly with sendfile enabled) and
> via ftp but from /dev/shm repeatably works fine; pulling from disk creates
> bad crc, every time. The file is readable and can be copied, verified etc.
> over and over so I'm sure that I'm not falling prey to a false positive.
> ifconfig indicates no dropped or otherwise corrupted packets.
> I noticed this first with 2.6.4-rc3, but also just tried the latest stable
> 2.6.23.9 with the same config, with no change in behaviour. After setting
> vsftpd to use_sendfile=NO, gigs can be transferred without corruption.
> 
> The data corruption is sporadic, but absolutely repeatable. The file with
> the known good pattern just contains multiple lines of:
> 
> 012345678901234567890123456789012345678901234567890
> 012345678901234567890123456789012345678901234567890
> 012345678901234567890123456789012345678901234567890
> ..etc..
> 
> A corrupted file is missing random characters, so that the corrupted lines
> looks like this (line numbers added by me):
> 
> 19785: 012345678901234567890123456789012345678901234567890
> 19786: 01234567890123456789012345678901234567890123678901234567890
> 19787: 012345678901234567890123456789012345678901234567890
> 
> or:
> 
> 20074: 012345678901234567890123456789012345678901234567890
> 20075: 01234567890123456789012345678901234567890123012345678901234567890123456789012345678901234567890
> 20076: 012345678901234567890123456789012345678901234567890
> 
> Again, other network or hd traffic shows no signs of gremlins; the box is
> perfectly stable, and turning sendfile on or off triggers/untriggers the
> corruption reliably.  I will try 2.6.22.x over the weekend, and before I
> bother lkml with dmesg/.config etc. I wanted to fish for initial thoughts.
> 

CC to netdev, it might concern network guys

Could you try with a test file containing unique patterns ?

like a 80 MB file :

#include <stdio.h>
main()
{
unsigned long ul;
for (ul = 0 ; ul < 10000000 ; ul++)
    printf("%8lu", ul);
return 0;
}


Thank you

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/