Message-ID: <473EEEC7.1070300@abinetworks.biz>
Date:	Sat, 17 Nov 2007 14:38:15 +0100
From:	Gianluca Alberici <gianluca@...networks.biz>
To:	Trond Myklebust <trond.myklebust@....uio.no>
CC:	markus reichelt <ml@...eichelt.de>, linux-kernel@...r.kernel.org
Subject: Re: NFS Bug in 2.6.23 ?

Hello,

I have made couple of tests:

1) (The fast one) Checked out, even if .config was the same onto both 
kernels, DIRECTIO: it was not set on both kernel configs.
2.6.20 works fine, .23 not.
This somehow seems to agree with Trond's P.O.V.
Finally i am wondering why should we set DIRECTIO to fix things on 
2.6.23 (supposed that it worked, i will try) when this could lead to 
inefficiencies, especialli where (CFSD) we need performance.
So far my opinion is that we should investigate deeper and find whats 
the problem.

2) (The verbose one) tcpdumped loopback iface onto my server (cfs is a 
loopback extended nfs server). For the test i set up an encrypted dir in 
which i put some webfiles and tested both listing the directory and 
accessing files in it by a remote mozilla setting httpd's DocumentRoot 
on /opt/clear/www. Got some output, here i go:

tcpdump -vv -i lo -s 9000
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 9000 
bytes
[...]

Now i ls /opt/clear/www

17:12:07.150036 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.675 > debian.3049: [bad udp cksum f303!] UDP, 
length: 116
17:12:07.150301 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.675: [bad udp cksum 47c5!] UDP, length: 96
17:12:13.405741 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.675 > debian.3049: [bad udp cksum 7b69!] UDP, 
length: 116
17:12:13.405790 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.675: [bad udp cksum f6de!] UDP, length: 96
17:12:13.405814 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.675 > debian.3049: [bad udp cksum 7b68!] UDP, 
length: 116
17:12:13.405827 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.675: [bad udp cksum f6dd!] UDP, length: 96
17:12:20.787680 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.675 > debian.3049: [bad udp cksum 7467!] UDP, 
length: 116
17:12:20.787727 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.675: [bad udp cksum f6dc!] UDP, length: 96
17:12:27.076069 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.675 > debian.3049: [bad udp cksum 6e66!] UDP, 
length: 116
17:12:27.076114 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.675: [bad udp cksum f6db!] UDP, length: 96
17:12:38.936258 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.675 > debian.3049: [bad udp cksum 6265!] UDP, 
length: 116
17:12:38.936415 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.675: [bad udp cksum f6da!] UDP, length: 96

it worked, now i fetch a webpage from mozilla on the crypto FS....

17:13:20.226643 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.675 > debian.3049: [bad udp cksum a8fd!] UDP, 
length: 116
17:13:20.226719 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.675: [bad udp cksum 47bf!] UDP, length: 96
17:13:20.226745 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.675 > debian.3049: [bad udp cksum 3863!] UDP, 
length: 116
17:13:20.226767 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.675: [bad udp cksum f6d8!] UDP, length: 96
17:13:20.226990 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.675 > debian.3049: [bad udp cksum f5f1!] UDP, 
length: 116
17:13:20.227009 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.675: [bad udp cksum 4664!] UDP, length: 96
17:13:20.227352 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.675 > debian.3049: [bad udp cksum f5f0!] UDP, 
length: 116
17:13:20.227388 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.675: [bad udp cksum 4663!] UDP, length: 96

...it worked....now ls again (port 917 instead of 675 ?)

18:15:38.567669 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum a7ee!] UDP, 
length: 116
18:15:38.567907 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.917: [bad udp cksum a9cd!] UDP, length: 96

this last worked...and finally it breaks (after an hour), couple of ls 
/opt/clear/www and Mozilla Reloads:

10:45:43.231051 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 152) debian.917 > debian.3049: [bad udp cksum 3fc4!] UDP, 
length: 124
10:45:43.231094 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 156) debian.3049 > debian.917: [bad udp cksum c483!] UDP, 
length: 128
10:45:43.231132 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 8d4c!] UDP, 
length: 116
10:45:43.231141 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 76) debian.3049 > debian.917: [bad udp cksum cdf1!] UDP, length: 48
10:45:43.231163 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 9d4b!] UDP, 
length: 116
10:45:43.231175 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.917: [bad udp cksum a912!] UDP, length: 96
10:45:43.231185 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 8d4a!] UDP, 
length: 116
10:45:43.231193 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 76) debian.3049 > debian.917: [bad udp cksum cdef!] UDP, length: 48
10:45:43.231258 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 152) debian.917 > debian.3049: [bad udp cksum 3fc0!] UDP, 
length: 124
10:45:43.231273 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 156) debian.3049 > debian.917: [bad udp cksum c47f!] UDP, 
length: 128
10:45:43.231289 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 8d48!] UDP, 
length: 116
10:45:43.231297 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 76) debian.3049 > debian.917: [bad udp cksum cded!] UDP, length: 48
10:45:43.231310 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 9d47!] UDP, 
length: 116
10:45:43.231323 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.917: [bad udp cksum a90e!] UDP, length: 96
10:45:43.231333 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 8d46!] UDP, 
length: 116
10:45:43.231341 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 76) debian.3049 > debian.917: [bad udp cksum cdeb!] UDP, length: 48
10:45:43.231355 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 152) debian.917 > debian.3049: [bad udp cksum 3fbc!] UDP, 
length: 124
10:45:43.231367 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 156) debian.3049 > debian.917: [bad udp cksum c47b!] UDP, 
length: 128
10:45:43.231382 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 8d44!] UDP, 
length: 116
10:45:43.231390 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 76) debian.3049 > debian.917: [bad udp cksum cde9!] UDP, length: 48
10:45:43.231414 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 9d43!] UDP, 
length: 116
10:45:43.231425 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.917: [bad udp cksum a90a!] UDP, length: 96
10:45:43.231435 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 8d42!] UDP, 
length: 116
10:45:43.231443 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 76) debian.3049 > debian.917: [bad udp cksum cde7!] UDP, length: 48
10:45:43.231490 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 152) debian.917 > debian.3049: [bad udp cksum 3fb8!] UDP, 
length: 124
10:45:43.231503 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 156) debian.3049 > debian.917: [bad udp cksum c477!] UDP, 
length: 128
10:45:43.231519 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 8d40!] UDP, 
length: 116
10:45:43.231527 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 76) debian.3049 > debian.917: [bad udp cksum cde5!] UDP, length: 48
10:45:43.231541 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 9d3f!] UDP, 
length: 116
10:45:43.231552 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.917: [bad udp cksum a906!] UDP, length: 96
10:45:43.231562 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 8d3e!] UDP, 
length: 116
10:45:43.231570 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 76) debian.3049 > debian.917: [bad udp cksum cde3!] UDP, length: 48
10:45:43.231584 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 152) debian.917 > debian.3049: [bad udp cksum 3fb4!] UDP, 
length: 124
10:45:43.231596 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 156) debian.3049 > debian.917: [bad udp cksum c473!] UDP, 
length: 128
10:45:43.231611 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 8d3c!] UDP, 
length: 116
10:45:43.231618 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 76) debian.3049 > debian.917: [bad udp cksum cde1!] UDP, length: 48
10:45:43.231632 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 9d3b!] UDP, 
length: 116
10:45:43.231643 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.917: [bad udp cksum a902!] UDP, length: 96
10:45:43.231654 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 8d3a!] UDP, 
length: 116
10:45:43.231661 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 76) debian.3049 > debian.917: [bad udp cksum cddf!] UDP, length: 48
10:46:29.073400 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum a79e!] UDP, 
length: 116
10:46:29.073433 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.917: [bad udp cksum 31d4!] UDP, length: 96

Trying to ls /opt/cle[TAB] (command completion) lists /opt/clear in 
which cattached dirs live, this refreshed and everything goes up again:

ls /opt/clear/www

10:54:57.359421 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum ab95!] UDP, 
length: 116
10:54:57.359458 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.917: [bad udp cksum 31cd!] UDP, length: 96
10:54:57.359616 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 732f!] UDP, 
length: 116
10:54:57.359637 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.917: [bad udp cksum 9a0f!] UDP, length: 96
10:54:57.359721 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 732e!] UDP, 
length: 116
10:54:57.359735 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.917: [bad udp cksum 9a0e!] UDP, length: 96
10:54:57.360367 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 144) debian.917 > debian.3049: [bad udp cksum 732d!] UDP, 
length: 116
10:54:57.360396 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], 
length: 124) debian.3049 > debian.917: [bad udp cksum 9a0d!] UDP, length: 96

**************************************************************

Thats all, i noticed:
- bad checksums (always, working or not)
- Port changes (portmapper ?)
- Packet length variations: sometimes an ls  takes 2 packs, sometimes 
10. Anyway it seems that pakets with length 48 are only present when 
system doesnt work.

What do you think ?

Regards,

Gianluca




Trond Myklebust wrote:

>On Fri, 2007-11-16 at 13:11 +0100, markus reichelt wrote:
>  
>
>>* Gianluca Alberici <gianluca@...networks.biz> wrote:
>>
>>    
>>
>>>If i cd into it and ls it is like doing a refresh: everything works
>>>again for a certain amount of time, then, again. It reminds me an
>>>old version of CFS which used to claim: 'stale NFS file handle'.
>>>      
>>>
>>"there are no fish in my pond"
>>
>>
>># CONFIG_NFS_DIRECTIO is not set
>>
>>Ive had probs with that option set, disabling did the trick for me.
>>    
>>
>
>Huh? Why would directio have anything at all to do with readdir()?
>
>Trond
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@...r.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>  
>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists