Date:   Thu, 1 Sep 2022 09:56:49 +0000
From:   "Dubel, Helena Anna" <helena.anna.dubel@...el.com>
To:     ivecera <ivecera@...hat.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
CC:     "Piotrowski, Patryk" <patryk.piotrowski@...el.com>,
        "Brandeburg, Jesse" <jesse.brandeburg@...el.com>,
        "Nguyen, Anthony L" <anthony.l.nguyen@...el.com>,
        "David S. Miller" <davem@...emloft.net>,
        "Eric Dumazet" <edumazet@...gle.com>,
        Jakub Kicinski <kuba@...nel.org>,
        Paolo Abeni <pabeni@...hat.com>,
        "Williams, Mitch A" <mitch.a.williams@...el.com>,
        "Jeff Kirsher" <jeffrey.t.kirsher@...el.com>,
        "Keller, Jacob E" <jacob.e.keller@...el.com>,
        "moderated list:INTEL ETHERNET DRIVERS" 
        <intel-wired-lan@...ts.osuosl.org>,
        open list <linux-kernel@...r.kernel.org>
Subject: RE: [PATCH net] i40e: Fix kernel crash during module removal


> -----Original Message-----
> From: ivecera <ivecera@...hat.com>
> Sent: wtorek, 16 sierpnia 2022 18:23
> To: netdev@...r.kernel.org
> Cc: Piotrowski, Patryk <patryk.piotrowski@...el.com>; Brandeburg, Jesse
> <jesse.brandeburg@...el.com>; Nguyen, Anthony L
> <anthony.l.nguyen@...el.com>; David S. Miller <davem@...emloft.net>;
> Eric Dumazet <edumazet@...gle.com>; Jakub Kicinski <kuba@...nel.org>;
> Paolo Abeni <pabeni@...hat.com>; Williams, Mitch A
> <mitch.a.williams@...el.com>; Jeff Kirsher <jeffrey.t.kirsher@...el.com>;
> Keller, Jacob E <jacob.e.keller@...el.com>; moderated list:INTEL ETHERNET
> DRIVERS <intel-wired-lan@...ts.osuosl.org>; open list <linux-
> kernel@...r.kernel.org>
> Subject: [PATCH net] i40e: Fix kernel crash during module removal
> 
> The driver incorrectly frees client instance and subsequent i40e module
> removal leads to kernel crash.
> 
> Reproducer:
> 1. Do ethtool offline test followed immediately by another one host# ethtool
> -t eth0 offline; ethtool -t eth0 offline 2. Remove recursively irdma module
> that also removes i40e module host# modprobe -r irdma
> 
> Result:
> [ 8675.035651] i40e 0000:3d:00.0 eno1: offline testing starting [ 8675.193774]
> i40e 0000:3d:00.0 eno1: testing finished [ 8675.201316] i40e 0000:3d:00.0
> eno1: offline testing starting [ 8675.358921] i40e 0000:3d:00.0 eno1: testing
> finished [ 8675.496921] i40e 0000:3d:00.0: IRDMA hardware initialization
> FAILED init_state=2 status=-110 [ 8686.188955] i40e 0000:3d:00.1:
> i40e_ptp_stop: removed PHC on eno2 [ 8686.943890] i40e 0000:3d:00.1:
> Deleted LAN device PF1 bus=0x3d dev=0x00 func=0x01 [ 8686.952669] i40e
> 0000:3d:00.0: i40e_ptp_stop: removed PHC on eno1 [ 8687.761787] BUG:
> kernel NULL pointer dereference, address: 0000000000000030 [ 8687.768755]
> #PF: supervisor read access in kernel mode [ 8687.773895] #PF:
> error_code(0x0000) - not-present page [ 8687.779034] PGD 0 P4D 0 [
> 8687.781575] Oops: 0000 [#1] PREEMPT SMP NOPTI
> [ 8687.785935] CPU: 51 PID: 172891 Comm: rmmod Kdump: loaded Tainted: G
> W I        5.19.0+ #2
> [ 8687.794800] Hardware name: Intel Corporation S2600WFD/S2600WFD,
> BIOS SE5C620.86B.0X.02.0001.051420190324 05/14/2019 [ 8687.805222] RIP:
> 0010:i40e_lan_del_device+0x13/0xb0 [i40e] [ 8687.810719] Code: d4 84 c0 0f
> 84 b8 25 01 00 e9 9c 25 01 00 41 bc f4 ff ff ff eb 91 90 0f 1f 44 00 00 41 54 55 53
> 48 8b 87 58 08 00 00 48 89 fb <48> 8b 68 30 48 89 ef e8 21 8a 0f d5 48 89 ef e8
> a9 78 0f d5 48 8b [ 8687.829462] RSP: 0018:ffffa604072efce0 EFLAGS: 00010202
> [ 8687.834689] RAX: 0000000000000000 RBX: ffff8f43833b2000 RCX:
> 0000000000000000 [ 8687.841821] RDX: 0000000000000000 RSI:
> ffff8f4b0545b298 RDI: ffff8f43833b2000 [ 8687.848955] RBP: ffff8f43833b2000
> R08: 0000000000000001 R09: 0000000000000000 [ 8687.856086] R10:
> 0000000000000000 R11: 000ffffffffff000 R12: ffff8f43833b2ef0 [ 8687.863218]
> R13: ffff8f43833b2ef0 R14: ffff915103966000 R15: ffff8f43833b2008 [
> 8687.870342] FS:  00007f79501c3740(0000) GS:ffff8f4adffc0000(0000)
> knlGS:0000000000000000 [ 8687.878427] CS:  0010 DS: 0000 ES: 0000 CR0:
> 0000000080050033 [ 8687.884174] CR2: 0000000000000030 CR3:
> 000000014276e004 CR4: 00000000007706e0 [ 8687.891306] DR0:
> 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [
> 8687.898441] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400 [ 8687.905572] PKRU: 55555554 [ 8687.908286] Call Trace:
> [ 8687.910737]  <TASK>
> [ 8687.912843]  i40e_remove+0x2c0/0x330 [i40e] [ 8687.917040]
> pci_device_remove+0x33/0xa0 [ 8687.920962]
> device_release_driver_internal+0x1aa/0x230
> [ 8687.926188]  driver_detach+0x44/0x90
> [ 8687.929770]  bus_remove_driver+0x55/0xe0 [ 8687.933693]
> pci_unregister_driver+0x2a/0xb0 [ 8687.937967]
> i40e_exit_module+0xc/0xf48 [i40e]
> 
> Two offline tests cause IRDMA driver failure (ETIMEDOUT) and this failure is
> indicated back to i40e_client_subtask() that calls
> i40e_client_del_instance() to free client instance referenced by pf->cinst and
> sets this pointer to NULL. During the module removal i40e_remove() calls
> i40e_lan_del_device() that dereferences
> pf->cinst that is NULL -> crash.
> Do not remove client instance when client open callbacks fails and just clear
> __I40E_CLIENT_INSTANCE_OPENED bit. The driver also needs to take care
> about this situation (when netdev is up and client is NOT opened) in
> i40e_notify_client_of_netdev_close() and calls client close callback only
> when __I40E_CLIENT_INSTANCE_OPENED is set.
> 
> Fixes: 0ef2d5afb12d ("i40e: KISS the client interface")
> Signed-off-by: Ivan Vecera <ivecera@...hat.com>
> ---
>  drivers/net/ethernet/intel/i40e/i40e_client.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 

Tested-by: Helena Anna Dubel <helena.anna.dubel@...el.com>
-------------------------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173, 80-298 Gdansk
KRS 101882, NIP 957-07-52-316

Powered by blists - more mailing lists