Message-ID: <36fb2ca8-493d-fabc-5220-abe7f5d5389f@akamai.com>
Date:   Mon, 2 Nov 2020 13:23:55 -0500
From:   Vishwanath Pai <vpai@...mai.com>
To:     Yunsheng Lin <linyunsheng@...wei.com>,
        Cong Wang <xiyou.wangcong@...il.com>
Cc:     "Hunt, Joshua" <johunt@...mai.com>,
        Jamal Hadi Salim <jhs@...atatu.com>,
        Jiri Pirko <jiri@...nulli.us>,
        David Miller <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>,
        Linux Kernel Network Developers <netdev@...r.kernel.org>,
        LKML <linux-kernel@...r.kernel.org>,
        "linuxarm@...wei.com" <linuxarm@...wei.com>,
        John Fastabend <john.fastabend@...il.com>,
        Eric Dumazet <eric.dumazet@...il.com>,
        "dsahern@...il.com" <dsahern@...il.com>,
        Joakim Tjernlund <Joakim.Tjernlund@...inera.com>
Subject: Re: [PATCH v2 net] net: sch_generic: aviod concurrent reset and
 enqueue op for lockless qdisc

On 11/2/20 4:08 AM, Yunsheng Lin wrote:
 > On 2020/10/30 1:20, Vishwanath Pai wrote:
 >> On 10/29/20 6:24 AM, Yunsheng Lin wrote:
 >>> On 2020/10/29 12:50, Vishwanath Pai wrote:
 >>>> On 10/28/20 10:37 PM, Yunsheng Lin wrote:
 >>>>> On 2020/10/29 4:04, Vishwanath Pai wrote:
 >>>>>> On 10/28/20 1:47 PM, Cong Wang wrote:
 >>>>>>> On Wed, Oct 28, 2020 at 8:37 AM Pai, Vishwanath 
<vpai@...mai.com> wrote:
 >>>>>>>> Hi,
 >>>>>>>>
 >>>>>>>> We noticed some problems when testing the latest 5.4 LTS 
kernel and traced it
 >>>>>>>> back to this commit using git bisect. When running our tests 
the machine stops
 >>>>>>>> responding to all traffic and the only way to recover is a 
reboot. I do not see
 >>>>>>>> a stack trace on the console.
 >>>>>>>
 >>>>>>> Do you mean the machine is still running fine just the network 
is down?
 >>>>>>>
 >>>>>>> If so, can you dump your tc config with stats when the problem 
is happening?
 >>>>>>> (You can use `tc -s -d qd show ...`.)
 >>>>>>>
 >>>>>>>>
 >>>>>>>> This can be reproduced using the packetdrill test below, it 
should be run a
 >>>>>>>> few times or in a loop. You should hit this issue within a few 
tries but
 >>>>>>>> sometimes might take up to 15-20 tries.
 >>>>>>> ...
 >>>>>>>> I can reproduce the issue easily on v5.4.68, and after 
reverting this commit it
 >>>>>>>> does not happen anymore.
 >>>>>>>
 >>>>>>> This is odd. The patch in this thread touches netdev reset 
path, if packetdrill
 >>>>>>> is the only thing you use to trigger the bug (that is netdev is 
always active),
 >>>>>>> I can not connect them.
 >>>>>>>
 >>>>>>> Thanks.
 >>>>>>
 >>>>>> Hi Cong,
 >>>>>>
 >>>>>>> Do you mean the machine is still running fine just the network 
is down?
 >>>>>>
 >>>>>> I was able to access the machine via serial console, it looks 
like it is
 >>>>>> up and running, just that networking is down.
 >>>>>>
 >>>>>>> If so, can you dump your tc config with stats when the problem 
is happening?
 >>>>>>> (You can use `tc -s -d qd show ...`.)
 >>>>>>
 >>>>>> If I try running tc when the machine is in this state the 
command never
 >>>>>> returns. It doesn't print anything but doesn't exit either.
 >>>>>>
 >>>>>>> This is odd. The patch in this thread touches netdev reset 
path, if packetdrill
 >>>>>>> is the only thing you use to trigger the bug (that is netdev is 
always active),
 >>>>>>> I can not connect them.
 >>>>>>
 >>>>>> I think packetdrill creates a tun0 interface when it starts the
 >>>>>> test and tears it down at the end, so it might be hitting this 
code path
 >>>>>> during teardown.
 >>>>>
 >>>>> Hi, Is there any preparation setup before running the above 
packetdrill test
 >>>>> case, I run the above test case in 5.9-rc4 with this patch 
applied without any
 >>>>> preparation setup, did not reproduce it.
 >>>>>
 >>>>> By the way, I am newbie to packetdrill:), it would be good to 
provide the
 >>>>> detail setup to reproduce it,thanks.
 >>>>>
 >>>>>>
 >>>>>> P.S: My mail server is having connectivity issues with 
vger.kernel.org
 >>>>>> so messages aren't getting delivered to netdev. It'll hopefully get
 >>>>>> resolved soon.
 >>>>>>
 >>>>>> Thanks,
 >>>>>> Vishwanath
 >>>>>>
 >>>>>>
 >>>>>> .
 >>>>>>
 >>>>
 >>>> I can't reproduce it on v5.9-rc4 either, it is probably an issue 
only on
 >>>> 5.4 then (and maybe older LTS versions). Can you give it a try on
 >>>> 5.4.68?
 >>>>
 >>>> For running packetdrill, download the latest version from their github
 >>>> repo, then run it in a loop without any special arguments. This is 
what
 >>>> I do to reproduce it:
 >>>>
 >>>> while true; do ./packetdrill <test-file>; done
 >>>>
 >>>> I don't think any other setup is necessary.
 >>>
 >>> Hi, run the above test for above an hour using 5.4.68, still did not
 >>> reproduce it, as below:
 >>>
 >>>
 >>> root@(none)$ cd /home/root/
 >>> root@(none)$ ls
 >>> creat_vlan.sh  packetdrill    test.pd
 >>> root@(none)$ cat test.pd
 >>> 0 `echo 4 > /proc/sys/net/ipv4/tcp_min_tso_segs`
 >>>
 >>> 0.400 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
 >>> 0.400 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
 >>>
 >>> // set maxseg to 1000 to work with both ipv4 and ipv6
 >>> 0.500 setsockopt(3, SOL_TCP, TCP_MAXSEG, [1000], 4) = 0
 >>> 0.500 bind(3, ..., ...) = 0
 >>> 0.500 listen(3, 1) = 0
 >>>
 >>> // Establish connection
 >>> 0.600 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 5>
 >>> 0.600 > S. 0:0(0) ack 1 <...>
 >>>
 >>> 0.800 < . 1:1(0) ack 1 win 320
 >>> 0.800 accept(3, ..., ...) = 4
 >>>
 >>> // Send 4 data segments.
 >>> +0 write(4, ..., 4000) = 4000
 >>> +0 > P. 1:4001(4000) ack 1
 >>>
 >>> // Receive a SACK
 >>> +.1 < . 1:1(0) ack 1 win 320 <sack 1001:2001,nop,nop>
 >>>
 >>> +.3 %{ print "TCP CA state: ",tcpi_ca_state  }%
 >>> root@(none)$ cat creat_vlan.sh
 >>> #!/bin/sh
 >>>
 >>> for((i=0; i<10000; i++))
 >>> do
 >>>     ./packetdrill test.pd
 >>> done
 >>> root@(none)$ ./creat_vlan.sh
 >>> TCP CA state:  3
 >>> ^C
 >>> root@(none)$ ifconfig
 >>> eth0      Link encap:Ethernet  HWaddr 5c:e8:83:0d:f7:ed
 >>>           inet addr:192.168.1.93  Bcast:192.168.1.255 
Mask:255.255.255.0
 >>>           UP BROADCAST RUNNING MULTICAST  MTU:1500 Metric:1
 >>>           RX packets:3570 errors:0 dropped:0 overruns:0 frame:0
 >>>           TX packets:3190 errors:0 dropped:0 overruns:0 carrier:0
 >>>           collisions:0 txqueuelen:1000
 >>>           RX bytes:1076349 (1.0 MiB)  TX bytes:414874 (405.1 KiB)
 >>>
 >>> eth2      Link encap:Ethernet  HWaddr 5c:e8:83:0d:f7:ec
 >>>           inet addr:192.168.100.1 Bcast:192.168.100.255 
Mask:255.255.255.0
 >>>           UP BROADCAST RUNNING MULTICAST  MTU:1500 Metric:1
 >>>           RX packets:81848576 errors:0 dropped:0 overruns:0 frame:78
 >>>           TX packets:72497816 errors:0 dropped:0 overruns:0 carrier:0
 >>>           collisions:0 txqueuelen:1000
 >>>           RX bytes:2044282289568 (1.8 TiB)  TX bytes:2457441698852 
(2.2 TiB)
 >>>
 >>> lo        Link encap:Local Loopback
 >>>           inet addr:127.0.0.1  Mask:255.0.0.0
 >>>           UP LOOPBACK RUNNING  MTU:65536  Metric:1
 >>>           RX packets:1 errors:0 dropped:0 overruns:0 frame:0
 >>>           TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
 >>>           collisions:0 txqueuelen:1000
 >>>           RX bytes:68 (68.0 B)  TX bytes:68 (68.0 B)
 >>>
 >>> root@(none)$ ./creat_vlan.sh
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> TCP CA state:  3
 >>> ^C
 >>> root@(none)$ cat /proc/cmdline
 >>> BOOT_IMAGE=/linyunsheng/Image.5.0 rdinit=/init 
console=ttyAMA0,115200 earlycon=pl011,mmio32,0x94080000 iommu.strict=1
 >>> root@(none)$ cat /proc/version
 >>> Linux version 5.4.68 (linyunsheng@...ntu) (gcc version 5.4.0 
20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.12)) #1 SMP PREEMPT Thu Oct 
29 16:59:37 CST 2020
 >>> root@(none)$
 >>>
 >>>
 >>>
 >>>>
 >>>> -Vishwanath
 >>>>
 >>>> .
 >>>>
 >> I couldn't get it to reproduce on a ubuntu VM, maybe something is
 >> different with the way we setup our machines. We do have some scripts in
 >> /etc/network/{if-up.d,if-post-down.d} etc, or probably something else.
 >> I'll let you know when I can reliably reproduce it on the VM.
 >
 > Hi, Vishwanath
 >     Please see if the patch in the below link fix your problem, thanks.
 > 
https://urldefense.com/v3/__https://www.spinics.net/lists/netdev/msg695908.html__;!!GjvTz_vk!Gc1MbfkLP7lWuWFxK2Bauofrqq53Tt79VJaozx08T8cmH0dRxAXuilS7xFt0$ 

 >
 >>
 >> .
 >>

I tried out the patch on top of 5.4.71 and it fixes the issue for me
too. Thanks for the quick fix.

- Vishwanath

Powered by blists - more mailing lists