Message-ID: <f3cee80744acbad74fed1b653f2b3bd32856bfb0.camel@barracuda.com>
Date: Thu, 30 May 2024 16:00:22 +0000
From: Matthias Stocker <mstocker@...racuda.com>
To: "kuba@...nel.org" <kuba@...nel.org>
CC: "pv-drivers@...are.com" <pv-drivers@...are.com>, "netdev@...r.kernel.org"
	<netdev@...r.kernel.org>, "doshir@...are.com" <doshir@...are.com>
Subject: Re: Re: [PATCH] vmxnet3: disable rx data ring on dma allocation
 failure

Am Mittwoch, dem 29.05.2024 um 17:57 -0700 schrieb Jakub Kicinski:
> On Tue, 28 May 2024 12:06:15 +0200 Matthias Stocker wrote:
> > When vmxnet3_rq_create fails to allocate memory for the data ring,
> > vmxnet3_rq_destroy_all_rxdataring is called, but rq-
> > >data_ring.desc_size
> > is not zeroed, as is the case when adapter->rxdataring_enabled is
> > set to
> > false. This leads to the box crashing a short time later with a
> > NULL pointer dereference in memcpy.
>
> That's not much of an explanation, more of restating what the logs
> say.
> I can't spot the bug in the existing code after looking at this for
> 10min. Please provide a proper explanation.
>
You're right, on re-reading the description, it's worse than I thought.

> > [1101376.713751] vmxnet3 0000:13:00.0 dhcp: rx data ring will be
> > disabled
> > [1101376.719942] vmxnet3 0000:13:00.0 dhcp: intr type 3, mode 0, 3
> > vectors allocated
> > [1101376.721085] vmxnet3 0000:13:00.0 dhcp: NIC Link is Up 10000
> > Mbps
> > [1101377.020907] BUG: kernel NULL pointer dereference, address:
> > 0000000000000000
> > [1101377.023396] #PF: supervisor read access in kernel mode
> > [1101377.025172] #PF: error_code(0x0000) - not-present page
> > [1101377.026966] PGD 115a58067 P4D 115a58067 PUD 115a55067 PMD 0
> > [1101377.028930] Oops: 0000 [#1] SMP NOPTI
> > [1101377.033776] Hardware name: VMware, Inc. VMware Virtual
> > Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
> > [1101377.037316] RIP: 0010:__memcpy+0x12/0x20
>
> Grrr.. Looks like you hid the kernel version. Are you sure your
> kernel
That was definitely not my intention, pasting it once more here for
completeness:

[1101376.713751] vmxnet3 0000:13:00.0 dhcp: rx data ring will be
disabled
[1101376.719942] vmxnet3 0000:13:00.0 dhcp: intr type 3, mode 0, 3
vectors allocated
[1101376.721085] vmxnet3 0000:13:00.0 dhcp: NIC Link is Up 10000 Mbps
[1101376.721286] 8021q: adding VLAN 0 to HW filter on device dhcp
[1101377.020907] BUG: kernel NULL pointer dereference, address:
0000000000000000
[1101377.023396] #PF: supervisor read access in kernel mode
[1101377.025172] #PF: error_code(0x0000) - not-present page
[1101377.026966] PGD 115a58067 P4D 115a58067 PUD 115a55067 PMD 0
[1101377.028930] Oops: 0000 [#1] SMP NOPTI
[1101377.030223] CPU: 0 PID: 26630 Comm: boxinfo Kdump: loaded Tainted:
P           O      5.10.176-ph09.00.01.01.x86_64 #1
[1101377.033776] Hardware name: VMware, Inc. VMware Virtual
Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[1101377.037316] RIP: 0010:__memcpy+0x12/0x20

> has commit 6f4833383e85 ("net: vmxnet3: Fix NULL pointer dereference
> in
> vmxnet3_rq_rx_complete()") ?

No and you're right it didn't include this commit, which according to
the admittedly better description, addresses the same issue, but it
didn't seem to solve the problem. For my test I switched to a vanilla
kernel 6.9.3 which includes the commit and hardcoded this allocation to
fail with the following patch:


diff --git a/drivers/net/vmxnet3/vmxnet3_drv.c
b/drivers/net/vmxnet3/vmxnet3_drv.c
index b8b32a7ace92..9e40832f6a8f 100644
--- a/drivers/net/vmxnet3/vmxnet3_drv.c
+++ b/drivers/net/vmxnet3/vmxnet3_drv.c
@@ -2146,10 +2146,10 @@ vmxnet3_rq_create(struct vmxnet3_rx_queue *rq,
struct vmxnet3_adapter *adapter)

        if ((adapter->rxdataring_enabled) && (rq->data_ring.desc_size
!= 0)) {
                sz = rq->rx_ring[0].size * rq->data_ring.desc_size;
-               rq->data_ring.base =
-                       dma_alloc_coherent(&adapter->pdev->dev, sz,
+               rq->data_ring.base = NULL;
+                       /*dma_alloc_coherent(&adapter->pdev->dev, sz,
                                           &rq->data_ring.basePA,
-                                          GFP_KERNEL);
+                                          GFP_KERNEL);*/
                if (!rq->data_ring.base) {
                        netdev_err(adapter->netdev,
                                   "rx data ring will be disabled\n");

The result was, that I couldn't connect the the machine anymore via ssh
and after a few tries I triggered the following kernel bug in
vmxnet3_rq_rx_complete.


[   95.436876] kernel BUG at net/core/skbuff.c:207!
[   95.439074] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[   95.440411] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 6.9.3-dirty #1
[   95.441558] Hardware name: VMware, Inc. VMware Virtual
Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[   95.443481] RIP: 0010:skb_panic+0x4d/0x4f
[   95.444404] Code: 4f 70 50 8b 87 c0 00 00 00 50 8b 87 bc 00 00 00 50
ff b7 d0 00 00 00 4c 8b 8f c8 00 00 00 48 c7 c7 68 e8 be 9f e8 63 58 f9
ff <0f> 0b 48 8b 14 24 48 c7 c1 d0 73 65 9f e8 a1 ff ff ff 48 8b 14 24
[   95.447684] RSP: 0018:ffffa13340274dd0 EFLAGS: 00010246
[   95.448762] RAX: 0000000000000089 RBX: ffff8fbbc72b02d0 RCX:
000000000000083f
[   95.450148] RDX: 0000000000000000 RSI: 00000000000000f6 RDI:
000000000000083f
[   95.451520] RBP: 000000000000002d R08: 0000000000000000 R09:
ffffa13340274c60
[   95.452886] R10: ffffffffa04ed468 R11: 0000000000000002 R12:
0000000000000000
[   95.454293] R13: ffff8fbbdab3c2d0 R14: ffff8fbbdbd829e0 R15:
ffff8fbbdbd809e0
[   95.455682] FS:  0000000000000000(0000) GS:ffff8fbeefd80000(0000)
knlGS:0000000000000000
[   95.457178] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   95.458340] CR2: 00007fd0d1f650c8 CR3: 0000000115f28000 CR4:
00000000000406f0
[   95.459791] Call Trace:
[   95.460515]  <IRQ>
[   95.461180]  ? __die_body.cold+0x19/0x27
[   95.462150]  ? die+0x2e/0x50
[   95.462976]  ? do_trap+0xca/0x110
[   95.463973]  ? do_error_trap+0x6a/0x90
[   95.464966]  ? skb_panic+0x4d/0x4f
[   95.465901]  ? exc_invalid_op+0x50/0x70
[   95.466849]  ? skb_panic+0x4d/0x4f
[   95.467718]  ? asm_exc_invalid_op+0x1a/0x20
[   95.468758]  ? skb_panic+0x4d/0x4f
[   95.469655]  skb_put.cold+0x10/0x10
[   95.470573]  vmxnet3_rq_rx_complete+0x862/0x11e0 [vmxnet3]
[   95.471853]  vmxnet3_poll_rx_only+0x36/0xb0 [vmxnet3]
[   95.473185]  __napi_poll+0x2b/0x160
[   95.474145]  net_rx_action+0x2c6/0x3b0
[   95.475115]  handle_softirqs+0xe7/0x2a0
[   95.476122]  __irq_exit_rcu+0x97/0xb0
[   95.477109]  common_interrupt+0x85/0xa0
[   95.478102]  </IRQ>
[   95.478846]  <TASK>
[   95.479603]  asm_common_interrupt+0x26/0x40
[   95.480657] RIP: 0010:pv_native_safe_halt+0xf/0x20
[   95.481801] Code: 22 d7 e9 54 87 01 00 0f 1f 40 00 90 90 90 90 90 90
90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 93 ba 3b 00 fb
f4 <e9> 2c 87 01 00 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
[   95.485563] RSP: 0018:ffffa133400ffe58 EFLAGS: 00000246
[   95.486882] RAX: 0000000000004000 RBX: ffff8fbbc1d14064 RCX:
0000000000000000
[   95.488477] RDX: ffff8fbeefd80000 RSI: ffff8fbbc1d14000 RDI:
0000000000000001
[   95.490067] RBP: ffff8fbbc1d14064 R08: ffffffffa0652260 R09:
00000000000010d3
[   95.491683] R10: 0000000000000018 R11: ffff8fbeefdb4764 R12:
ffffffffa0652260
[   95.493389] R13: ffffffffa06522e0 R14: 0000000000000001 R15:
0000000000000000
[   95.495035]  acpi_safe_halt+0x14/0x20
[   95.496127]  acpi_idle_do_entry+0x2f/0x50
[   95.497221]  acpi_idle_enter+0x7f/0xd0
[   95.498272]  cpuidle_enter_state+0x81/0x420
[   95.499375]  cpuidle_enter+0x2d/0x40
[   95.500400]  do_idle+0x1e5/0x240
[   95.501385]  cpu_startup_entry+0x29/0x30
[   95.502422]  start_secondary+0x11c/0x140
[   95.503454]  common_startup_64+0x13e/0x141
[   95.504466]  </TASK>
[   95.505197] Modules linked in: nft_fib_inet nft_fib_ipv4
nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6
nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6
nf_defrag_ipv4 rfkill ip_set nf_tables vsock_loopback
vmw_vsock_virtio_transport_common qrtr vmw_vsock_vmci_transport vsock
sunrpc binfmt_misc pktcdvd vmw_balloon pcspkr vmw_vmci i2c_piix4 joydev
loop dm_multipath nfnetlink zram crct10dif_pclmul crc32_pclmul vmwgfx
crc32c_intel polyval_clmulni polyval_generic ghash_clmulni_intel
sha512_ssse3 sha256_ssse3 vmxnet3 sha1_ssse3 drm_ttm_helper vmw_pvscsi
ttm ata_generic pata_acpi serio_raw scsi_dh_rdac scsi_dh_emc
scsi_dh_alua ip6_tables ip_tables fuse
[   95.516536] ---[ end trace 0000000000000000 ]---

Whereas when I reverted the above mentioned commit 6f4833383e85 ("net:
vmxnet3: Fix NULL pointer dereference and applied the patch I've
submitted earlier, then everything seemed to work perfectly fine even
with those allocations failing.

I assume that setting rq->data_ring.desc_size to zero in
vmxnet3_rq_destroy_all_rxdataring is also needed for the data_ring that
faild to allocate its base. Otherwise it will later on be handed over
to Vmxnet3_RxQueueConf->rxDataRingDescSize in the function
vmxnet3_setup_driver_shared thus allowing to reference the data_ring of
this rx_queue, but I didn't have time to pursue it any further.




Get the 13 Email Threat Types eBook

https://www.barracuda.com/

This e-mail and any attachments to it contain confidential and proprietary material of Barracuda, its affiliates or agents, and is solely for the use of the intended recipient. Any review, use, disclosure, distribution or copying of this transmittal is prohibited except by or on behalf of the intended recipient. If you have received this transmittal in error, please notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed.

________________________________

Powered by blists - more mailing lists