netdev - Re: Crash in indirect block infra after unloading driver module

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <vbfk0zwy70f.fsf@mellanox.com>
Date:   Wed, 24 Jun 2020 15:37:20 +0300
From:   Vlad Buslov <vladbu@...lanox.com>
To:     Vlad Buslov <vladbu@...lanox.com>
Cc:     Pablo Neira Ayuso <pablo@...filter.org>,
        "netdev\@vger.kernel.org" <netdev@...r.kernel.org>,
        Jiri Pirko <jiri@...nulli.us>,
        Saeed Mahameed <saeedm@...lanox.com>,
        Roi Dayan <roid@...lanox.com>,
        Majd Dibbiny <majd@...lanox.com>,
        Maor Dickman <maord@...lanox.com>, wenxu <wenxu@...oud.cn>
Subject: Re: Crash in indirect block infra after unloading driver module


On Wed 24 Jun 2020 at 15:22, Vlad Buslov <vladbu@...lanox.com> wrote:
> On Wed 24 Jun 2020 at 13:30, Pablo Neira Ayuso <pablo@...filter.org> wrote:
>> On Wed, Jun 24, 2020 at 01:22:29PM +0300, Vlad Buslov wrote:
>>> Hi Pablo,
>>> 
>>> I've encountered a new issue with indirect offloads infrastructure. The
>>> issue is that on driver offload its indirect callbacks are not removed
>>> from blocks and any following offloads operations on block that has such
>>> callback in its offloads cb list causes call to unmapped address.
>>> 
>>> Steps to reproduce:
>>> 
>>> echo 1 >/sys/class/net/ens1f0/device/sriov_numvfs
>>> echo 0000:81:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
>>> devlink dev eswitch set pci/0000:81:00.0 mode switchdev
>>> 
>>> ip link add vxlan1 type vxlan dstport 4789 external
>>> ip addr add 192.168.1.1 dev ens1f0
>>> link set up dev ens1f0
>>> ip link set up dev ens1f0
>>> tc qdisc add dev vxlan1 ingress
>>> tc filter add dev vxlan1 protocol ip ingress flower enc_src_ip 192.168.1.2 enc_dst_ip 192.168.1.1 enc_key_id 42 enc_dst_port 4789 action tunnel_key unset action mirred egress redirect dev ens1f0_0
>>> tc -s filter show dev vxlan1 ingress
>>> 
>>> rmmod mlx5_ib
>>> rmmod mlx5_core
>>> tc -s filter show dev vxlan1 ingress
>>
>> On module removal, the representors are gone and the ->cleanup
>> callback should be exercised, this callback removes the flow_block and
>> removes the rules in the driver.
>>
>> Can you check if the ->cleanup callback is exercised?
>
> I added some traces. On module unload mlx5e_cleanup_rep_tx() and
> flow_indr_dev_unregister() are called, but not tc_block_indr_cleanup().
> Maybe this is the problem that wenxu fixed in one of his patches? I'll
> try to reproduce on net branch.

Indeed, on net branch tc_block_indr_cleanup() is called and crash is not
reproduced. It seems to be fixed by a1db217861f3 ("net: flow_offload:
fix flow_indr_dev_unregister path"). 

>
>>
>>> Resulting dmesg:
>>> 
>>> [  153.747853] BUG: unable to handle page fault for address: ffffffffc114cee0
>>> [  153.747975] #PF: supervisor instruction fetch in kernel mode
>>> [  153.748071] #PF: error_code(0x0010) - not-present page
>>> [  153.748189] PGD 5b6c12067 P4D 5b6c12067 PUD 5b6c14067 PMD 35b76b067 PTE 0
>>> [  153.748328] Oops: 0010 [#1] SMP KASAN PTI
>>> [  153.748403] CPU: 1 PID: 1909 Comm: tc Not tainted 5.8.0-rc1+ #1170
>>> [  153.748507] Hardware name: Supermicro SYS-2028TP-DECR/X10DRT-P, BIOS 2.0b 03/30/2017
>>> [  153.748638] RIP: 0010:0xffffffffc114cee0
>>> [  153.748709] Code: Bad RIP value.
>>> [  153.748767] RSP: 0018:ffff88834895ef00 EFLAGS: 00010246
>>> [  153.748858] RAX: 0000000000000000 RBX: ffff888330a30078 RCX: ffffffffb2da70ba
>>> [  153.748975] RDX: ffff888333635d80 RSI: ffff88834895efa0 RDI: 0000000000000002
>>> [  153.752948] RBP: 0000000000000002 R08: 0000000000000001 R09: ffffed106614600c
>>> [  153.759173] R10: ffff888330a3005f R11: ffffed106614600b R12: ffff88834895efa0
>>> [  153.765419] R13: 0000000000000000 R14: ffffffffc114cee0 R15: ffff8883470efe00
>>> [  153.771689] FS:  00007f6f6ac12480(0000) GS:ffff888362e40000(0000) knlGS:0000000000000000
>>> [  153.777983] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [  153.784187] CR2: ffffffffc114ceb6 CR3: 000000035eb9e005 CR4: 00000000001606e0
>>> [  153.790567] Call Trace:
>>> [  153.796844]  ? tc_setup_cb_call+0xd8/0x170
>>> [  153.803164]  ? fl_hw_update_stats+0x117/0x280 [cls_flower]
>>> [  153.809516]  ? 0xffffffffc1328000
>>> [  153.815766]  ? _find_next_bit.constprop.0+0x3e/0xf0
>>> [  153.822079]  ? __nla_reserve+0x4c/0x60
>> [...]
>>> 
>>> I can come up with something to fix mlx5 but it looks like all other
>>> drivers that support indirect devices are also susceptible to similar
>>> issue.
>>
>> How does the fix you have in mind look like?
>
> To call flow_indr_dev_unregister() from right path. But you are right,
> it is already called, so we just need to determine why it doesn't
> perform the proper cleanup.
>
>>
>> Thanks.