netdev - Re: net: hang in unregister_netdevice: waiting for lo to become free

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CACT4Y+beZ2+O=XdjKCpAkXOTBDgHYfn1JW8kn-TGRrph_RmV7Q@mail.gmail.com>
Date:   Tue, 20 Feb 2018 09:14:41 +0100
From:   Dmitry Vyukov <dvyukov@...gle.com>
To:     Tommi Rantala <tommi.t.rantala@...ia.com>
Cc:     Xin Long <lucien.xin@...il.com>, David Ahern <dsahern@...il.com>,
        Daniel Borkmann <daniel@...earbox.net>,
        Cong Wang <xiyou.wangcong@...il.com>,
        David Miller <davem@...emloft.net>,
        Eric Dumazet <edumazet@...gle.com>,
        Willem de Bruijn <willemb@...gle.com>,
        Jakub Kicinski <jakub.kicinski@...ronome.com>,
        Rasmus Villemoes <linux@...musvillemoes.dk>,
        netdev <netdev@...r.kernel.org>,
        LKML <linux-kernel@...r.kernel.org>,
        Alexey Kuznetsov <kuznet@....inr.ac.ru>,
        Hideaki YOSHIFUJI <yoshfuji@...ux-ipv6.org>,
        syzkaller <syzkaller@...glegroups.com>,
        Dan Streetman <ddstreet@...onical.com>,
        "Eric W. Biederman" <ebiederm@...ssion.com>,
        Alexey Kodanev <alexey.kodanev@...cle.com>,
        Neil Horman <nhorman@...driver.com>,
        Marcelo Ricardo Leitner <marcelo.leitner@...il.com>,
        linux-sctp@...r.kernel.org
Subject: Re: net: hang in unregister_netdevice: waiting for lo to become free

On Tue, Feb 20, 2018 at 8:56 AM, Tommi Rantala
<tommi.t.rantala@...ia.com> wrote:
> On 19.02.2018 20:59, Dmitry Vyukov wrote:
>>
>> On Sat, Feb 3, 2018 at 1:15 PM, Xin Long <lucien.xin@...il.com> wrote:
>>>>>
>>>>> On 1/30/18 1:57 PM, David Ahern wrote:
>>>>>>
>>>>>> On 1/30/18 1:08 PM, Daniel Borkmann wrote:
>>>>>>>
>>>>>>> On 01/30/2018 07:32 PM, Cong Wang wrote:
>>>>>>>>
>>>>>>>> On Tue, Jan 30, 2018 at 4:09 AM, Dmitry Vyukov <dvyukov@...gle.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> The following program creates a hang in unregister_netdevice.
>>>>>>>>> cleanup_net work hangs there forever periodically printing
>>>>>>>>> "unregister_netdevice: waiting for lo to become free. Usage count =
>>>>>>>>> 3"
>>>>>>>>> and creation of any new network namespaces hangs forever.
>>>>>>>>
>>>>>>>>
>>>>>>>> Interestingly, this is not reproducible on net-next.
>>>>>>>
>>>>>>>
>>>>>>> The most recent change on netns refcnt was 4ee806d51176 ("net: tcp:
>>>>>>> close
>>>>>>> sock if net namespace is exiting") in net/net-next from 5 days ago,
>>>>>>> maybe
>>>>>>> fixed due to that?
>>>>>>>
>>>>>>
>>>>>> This appears to be the commit introducing the refcnt leak:
>>>>>>
>>>>>> $ git bisect bad
>>>>>> dbc2b5e9a09e9a6664679a667ff81cff6e5f2641 is the first bad commit
>>>>>> commit dbc2b5e9a09e9a6664679a667ff81cff6e5f2641
>>>>>> Author: Xin Long <lucien.xin@...il.com>
>>>>>> Date:   Fri May 12 14:39:52 2017 +0800
>>>>>>
>>>>>>      sctp: fix src address selection if using secondary addresses for
>>>>>> ipv6
>>>>>>
>>>>>>
>>>>>> v4.14 is bad. Running bisect in the background while doing other
>>>>>> things....
>>>>>>
>>>>>
>>>>> Interesting. The commit that avoids the refcnt leak is
>>>>>
>>>>> commit 955ec4cb3b54c7c389a9f830be7d3ae2056b9212
>>>>> Author: David Ahern <dsahern@...il.com>
>>>>> Date:   Wed Jan 24 19:45:29 2018 -0800
>>>>>
>>>>>      net/ipv6: Do not allow route add with a device that is down
>>>>>
>>>>> That commit does not intentionally address the problem so it is just
>>>>> masking the problematic code introduced by the commit above.
>>>>
>>>> Thanks, David A.
>>>>
>>>> I'm still on a trip. will look into this asap.
>>>
>>>
>>> Alexey and Tommi already had the patches for this issue on
>>> both SCTP v4 and v6 dst_get, Thanks.
>>
>>
>>
>>
>> Is this meant to be fixed already? I am still seeing this on the
>> latest upstream tree.
>>
>
> These two commits are in v4.16-rc1:
>
> commit 4a31a6b19f9ddf498c81f5c9b089742b7472a6f8
> Author: Tommi Rantala <tommi.t.rantala@...ia.com>
> Date:   Mon Feb 5 21:48:14 2018 +0200
>
>     sctp: fix dst refcnt leak in sctp_v4_get_dst
> ...
>     Fixes: 410f03831 ("sctp: add routing output fallback")
>     Fixes: 0ca50d12f ("sctp: fix src address selection if using secondary
> addresses")
>
>
> commit 957d761cf91cdbb175ad7d8f5472336a4d54dbf2
> Author: Alexey Kodanev <alexey.kodanev@...cle.com>
> Date:   Mon Feb 5 15:10:35 2018 +0300
>
>     sctp: fix dst refcnt leak in sctp_v6_get_dst()
> ...
>     Fixes: dbc2b5e9a09e ("sctp: fix src address selection if using secondary
> addresses for ipv6")
>
>
> I guess we missed something if it's still reproducible.
>
> I can check it later this week, unless someone else beat me to it.

Hi Tommi,

Hmmm, I can't claim that it's exactly the same bug. Perhaps it's
another one then. But I am still seeing these:

[   58.799130] unregister_netdevice: waiting for lo to become free.
Usage count = 4
[   60.847138] unregister_netdevice: waiting for lo to become free.
Usage count = 4
[   62.895093] unregister_netdevice: waiting for lo to become free.
Usage count = 4
[   64.943103] unregister_netdevice: waiting for lo to become free.
Usage count = 4

on upstream tree pulled ~12 hours ago.

Kernel does not detect this as any kind of BUG/WARNING, so
syzkaller/syzbot do not catch it as bug and do not try to reproduce,
localize and report.