netdev - Re: Fw: [Bug 201423] New: eth0: hw csum failure

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <c81fc59a-5fdb-18e5-3c16-68299458cd44@tomt.net>
Date:   Sun, 4 Nov 2018 06:43:03 +0100
From:   Andre Tomt <andre@...t.net>
To:     Eric Dumazet <eric.dumazet@...il.com>,
        Eric Dumazet <edumazet@...gle.com>
Cc:     Stephen Hemminger <stephen@...workplumber.org>,
        netdev <netdev@...r.kernel.org>, rossi.f@...ind.it,
        Dimitris Michailidis <dmichail@...gle.com>
Subject: Re: Fw: [Bug 201423] New: eth0: hw csum failure

On 31.10.2018 05:08, Andre Tomt wrote:
> On 30.10.2018 12:04, Andre Tomt wrote:
>> On 30.10.2018 11:58, Andre Tomt wrote:
>>> On 27.10.2018 23:41, Andre Tomt wrote:
>>>> On 26.10.2018 13:45, Andre Tomt wrote:
>>>>> On 25.10.2018 19:38, Eric Dumazet wrote:
>>>>>>
>>>>>>
>>>>>> On 10/24/2018 12:41 PM, Andre Tomt wrote:
>>>>>>>
>>>>>>> It eventually showed up again with mlx4, on 4.18.16 + fix and 
>>>>>>> also on 4.19. I still do not have a useful packet capture.
>>>>>>>
>>>>>>> It is running a torrent client serving up various linux 
>>>>>>> distributions.
>>>>>>>
>>>>>>
>>>>>> Have you also applied this fix ?
>>>>>>
>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=db4f1be3ca9b0ef7330763d07bf4ace83ad6f913 
>>>>>>
>>>>>>
>>>>>
>>>>> No. I've applied it now to 4.19 and will report back if anything 
>>>>> shows up.
>>>>
>>>> Just hit it on the simpler server; no VRF, no tunnels, no 
>>>> nat/conntrack. Only a basic stateless nftables ruleset and a vlan 
>>>> netdev (unlikely to be the one triggering this I guess; it has only 
>>>> v4 traffic).
>>>
>>> I'm currently testing 4.19 with the recomended commit added, plus 
>>> these to sort out some GRO issues (on a hunch, unsure if related):
>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=a8305bff685252e80b7c60f4f5e7dd2e63e38218 
>>>
>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=992cba7e276d438ac8b0a8c17b147b37c8c286f7 
>>>
>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=ece23711dd956cd5053c9cb03e9fe0668f9c8894 
>>>
>>>
>>> and I *think* it is behaving better now? it's not conclusive as it 
>>> could take a while to trip in this environment but some of the test 
>>> servers have not shown anything bad in almost 24h.
>>
>> Sorry, s/some of the/none of the
> 
> I think it is fairly safe to say 4.19 + mlx4 + these 4 commits is OK. At 
> least for my workload. Servers are now 51-61 hours in, no splats. I also 
> added ntp pool traffic to one of them to make things a little more 
> exciting.
> 
> Not sure what is needed for 4.18, I dont have the mental bandwidth to 
> test that right now. Also no idea about the similar looking mlx5 splats 
> reported elsewhere.

As expected conntrack/nat + vlan + forwarding still splats.
sch_cake, IFB and VRF was removed from this setup.

Here is a conntrack splat without IFB/VRF/Cake inteference:
> [34458.506346] wanib: hw csum failure
> [34458.506371] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.19.0-1 #1
> [34458.506374] Hardware name: Supermicro Super Server/X10SDV-4C-TLN2F, BIOS 2.0 06/13/2018
> [34458.506377] Call Trace:
> [34458.506381]  <IRQ>
> [34458.506388]  dump_stack+0x5c/0x80
> [34458.506392]  __skb_checksum_complete+0xac/0xc0
> [34458.506402]  icmp_error+0x1c8/0x1f0 [nf_conntrack]
> [34458.506406]  ? skb_copy_bits+0x13d/0x220
> [34458.506411]  nf_conntrack_in+0xd8/0x390 [nf_conntrack]
> [34458.506416]  ? ___pskb_trim+0x192/0x330
> [34458.506421]  nf_hook_slow+0x43/0xc0
> [34458.506426]  ip_rcv+0x90/0xb0
> [34458.506430]  ? ip_rcv_finish_core.isra.0+0x310/0x310
> [34458.506435]  __netif_receive_skb_one_core+0x42/0x50
> [34458.506438]  netif_receive_skb_internal+0x24/0xb0
> [34458.506441]  napi_gro_frags+0x177/0x210
> [34458.506446]  mlx4_en_process_rx_cq+0x8df/0xb50 [mlx4_en]
> [34458.506459]  ? mlx4_eq_int+0x38f/0xcb0 [mlx4_core]
> [34458.506463]  mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
> [34458.506466]  net_rx_action+0xe1/0x2c0
> [34458.506469]  __do_softirq+0xe7/0x2d3
> [34458.506475]  irq_exit+0x96/0xd0
> [34458.506478]  do_IRQ+0x85/0xd0
> [34458.506483]  common_interrupt+0xf/0xf
> [34458.506486]  </IRQ>
> [34458.506491] RIP: 0010:cpuidle_enter_state+0xb9/0x320
> [34458.506495] Code: e8 3c 16 bc ff 80 7c 24 0b 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 3b 02 00 00 31 ff e8 5e fb c0 ff fb 66 0f 1f 44 00 00 <48> b8 ff ff ff ff f3 01 00 00 48 2b 1c 24 ba ff ff ff 7f 48 39 c3
> [34458.506497] RSP: 0018:ffff978d41943ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdb
> [34458.506500] RAX: ffff8d8f6fa60fc0 RBX: 00001f56ff07af28 RCX: 000000000000001f
> [34458.506501] RDX: 00001f56ff07af28 RSI: 000000003a2e90d6 RDI: 0000000000000000
> [34458.506503] RBP: ffff8d8f6fa698c0 R08: 0000000000000002 R09: 0000000000020840
> [34458.506504] R10: 0004ea58f2899595 R11: ffff8d8f6fa601e8 R12: 0000000000000001
> [34458.506505] R13: ffffffff8a0ac638 R14: 0000000000000001 R15: 0000000000000000
> [34458.506509]  ? cpuidle_enter_state+0x94/0x320
> [34458.506512]  do_idle+0x1e4/0x220
> [34458.506515]  cpu_startup_entry+0x5f/0x70
> [34458.506519]  start_secondary+0x185/0x1a0
> [34458.506521]  secondary_startup_64+0xa4/0xb0

Stateless filtered non-forwarding host still looks like it has been 
fixed (the udp6_gro_* splats are still all gone). Also seems fine when 
moving the traffic over a vlan device. These fixes went into 4.19.1-rc1 
(checksum_complete + unlink gro packets on overflow fixes)