netdev - Re: [Intel-wired-lan] NFS over NAT causes e1000e transmit hangs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <1492968274.6453.36.camel@edumazet-glaptop3.roam.corp.google.com>
Date:   Sun, 23 Apr 2017 10:24:34 -0700
From:   Eric Dumazet <eric.dumazet@...il.com>
To:     Florian Fainelli <f.fainelli@...il.com>
Cc:     "Neftin, Sasha" <sasha.neftin@...el.com>, netdev@...r.kernel.org,
        intel-wired-lan@...ts.osuosl.org
Subject: Re: [Intel-wired-lan] NFS over NAT causes e1000e transmit hangs

On Sun, 2017-04-23 at 10:08 -0700, Florian Fainelli wrote:
> 
> On 04/22/2017 11:46 PM, Neftin, Sasha wrote:
> > On 4/20/2017 00:15, Florian Fainelli wrote:
> >> On 04/19/2017 01:52 AM, Neftin, Sasha wrote:
> >>> On 4/18/2017 22:05, Florian Fainelli wrote:
> >>>> On 04/18/2017 12:03 PM, Eric Dumazet wrote:
> >>>>> On Tue, 2017-04-18 at 11:18 -0700, Florian Fainelli wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> I am using NFS over a NAT with two e1000e adapters and with eth1
> >>>>>> being
> >>>>>> the LAN interface and eth0 the WAN interface. The kernel is Ubuntu's
> >>>>>> 16.10 kernel: 4.8.0-46-generic. The device doing NAT over NFS is just
> >>>>>> mounting a remote folder and doing normal execution/file accesses.
> >>>>>> It's
> >>>>>> enough to untar a file from this device onto a NFS share to expose
> >>>>>> the
> >>>>>> problem.
> >>>>>>
> >>>>>> The transmit hangs look like the ones below, doing a rmmod/insmod
> >>>>>> does
> >>>>>> not help eliminated the problem, nor does a power cycle. Stopping the
> >>>>>> NFS over NAT definitively does let the adapter recover.
> >>>>> Is this NFS over TCP or UDP ?
> >>>> This is NFS over TCP mounted with the following:
> >>>>
> >>>> type nfs
> >>>> (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=2049,timeo=70,retrans=3,sec=sys,local_lock=none,addr=X.X.X.X)
> >>>>
> >>>>
> >>>>
> >>>> Thanks Eric!
> >>> Please, try disable TCP segmentation offload: ethtool -K <adapter>
> >>> tso off.
> >> I am not able to reproduce the hangs with TSO turned off. Is there a
> >> specific patch you would want me to try?
> > 
> > Please, work with TSO turned off so. There is no patch for this specific
> > problem.
> 
> OK, are not we interested in somehow being able to identify such
> problematic packets coming from the networking stack and force not using
> TSO for those? Would an acceptable solution be to force the disabling of
> TSO for this specific NIC model (provided it is some kind of HW bug)?
> 
> NB: I understand this is very old hardware for you at Intel, but
> conversely, it is very widespread, and chances of people running into
> similar issues are pretty high, so fixing it once would de-facto lower
> the amount of support you'd have to provide in the future.

Indeed it is very odd to disable TSO, especially if the problem only
shows up with NAT.

We probably have a nasty bug somewhere, or we might be able to have a
work around some hardware 'feature'.