netdev - Re: [PATCH net] net: mld: do not use system

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Fri, 22 Jul 2022 10:35:52 +0800
From:   Hangbin Liu <liuhangbin@...il.com>
To:     Taehee Yoo <ap420073@...il.com>
Cc:     Eric Dumazet <edumazet@...gle.com>,
        David Miller <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>,
        Paolo Abeni <pabeni@...hat.com>,
        Hideaki YOSHIFUJI <yoshfuji@...ux-ipv6.org>,
        David Ahern <dsahern@...nel.org>,
        netdev <netdev@...r.kernel.org>
Subject: Re: [PATCH net] net: mld: do not use system_wq in the mld

On Fri, Jul 22, 2022 at 04:01:13AM +0900, Taehee Yoo wrote:
> 
> 
> On 7/22/22 03:34, Eric Dumazet wrote:
> > On Thu, Jul 21, 2022 at 7:53 PM Taehee Yoo <ap420073@...il.com> wrote:
> >>
> >> Hi Eric,
> >> Thank you so much for your review!
> >>
> >
> > ...
> >
> >> I think your assumption is right.
> >> I tested the below scenario, which occurs the real issue.
> >> THREAD0                            THREAD1
> >> mld_report_work()
> >>                                      spin_lock_bh()
> >>                                      if (!mod_delayed_work()) <-- queued
> >>                                              in6_dev_hold();
> >>                                      spin_unlock_bh()
> >> spin_lock_bh()
> >> schedule_delayed_work() <-- return false, already queued by THREAD1
> >> spin_unlock_bh()
> >> return;
> >> //no in6_dev_put() regardless return value of schedule_delayed_work().
> >>
> >> In order to check, I added printk like below.
> >>           if (++cnt >= MLD_MAX_QUEUE) {
> >>
> >>                   rework = true;
> >>
> >>                   if (!schedule_delayed_work(&idev->mc_report_work, 0))
> >>                           printk("[TEST]%s %u \n", __func__, __LINE__);
> >>                   break;
> >>
> >>
> >> If the TEST log message is printed, work is already queued by other
> logic.
> >> So, it indicates a reference count is leaked.
> >> The result is that I can see log messages only when the reference count
> >> leak occurs.
> >> So, although I tested it only for 1 hour, I'm sure that this bug comes
> >> from missing check a return value of schedule_delayed_work().
> >>
> >> As you said, this changelog is not correct.
> >> system_wq and mld_wq are not related to this issue.
> >>
> >> I would like to send a v2 patch after some more tests.
> >> The v2 patch will change the commit message.
> >
> > Can you describe what kind of tests you are running ?
> > Was it a syzbot report ?
> 
> I found this bug while testing another syzbot report.
> (https://syzkaller.appspot.com/bug?id=ed41eaa4367b421d37aab5dee25e3f4c91ceae93)
> And I can't find the same case in the syzbot reports list.
> 
> I just use some command lines and many kernel debug options such as
> kmemleak, kasan, lockdep, and others.
> 

Hi Taehee,

I got a similar issue with yours after Eric's 2d3916f31891
("ipv6: fix skb drops in igmp6_event_query() and igmp6_event_report()").
I use force_mld_version=1 and adding a lot of IPv6 address to generate the
mld reports flood. Here is my reproducer:

[root@...tp-73-131-221 ~]# cat mld.sh
#!/bin/sh

ip netns add ns1
ip netns add ns2
ip netns exec ns1 sysctl -w net.ipv6.conf.all.force_mld_version=1
ip netns exec ns2 sysctl -w net.ipv6.conf.all.force_mld_version=1

ip -n ns1 link add veth0 type veth peer name veth0 netns ns2
ip -n ns1 link set veth0 up
ip -n ns2 link set veth0 up

for i in `seq 50`; do
        for j in `seq 100`; do
                ip -n ns1 addr add 2021:${i}::${j}/64 dev veth0
                ip -n ns2 addr add 2022:${i}::${j}/64 dev veth0
        done
done
modprobe -r veth
ip -a netns del

After `modprobe -r veth` we will the the ref leaker error:
[ 1382.683371] unregister_netdevice: waiting for veth0 to become free. Usage count = 2 
[ 1392.931397] unregister_netdevice: waiting for veth0 to become free. Usage count = 2 
[ 1402.939234] unregister_netdevice: waiting for veth0 to become free. Usage count = 2 
[ 1413.179296] unregister_netdevice: waiting for veth0 to become free. Usage count = 2 

I tried to debug the reason but didn't have much clue. Maybe this reproducer
could help you.

Thanks
Hangbin