| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Tue, 4 Jun 2024 09:44:07 +0000
From: "Berger, Michal" <michal.berger@...el.com>
To: Shay Drori <shayd@...dia.com>, "netdev@...r.kernel.org"
<netdev@...r.kernel.org>, "moshe@...dia.com" <moshe@...dia.com>,
"linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
"phaddad@...dia.com" <phaddad@...dia.com>
Subject: Re: Kernel panic triggered while removing mlx5_core devices from the
pci bus
Hi Shay,
I am afraid that the suggested change didn't help (I applied both patches now on top of 6.8.9), kernel still crashes with very similar (if not the same) traces, please see attached.
Regards,
Michal
________________________________
From: Shay Drori <shayd@...dia.com>
Sent: Thursday, May 30, 2024 10:09 AM
To: Berger, Michal <michal.berger@...el.com>; netdev@...r.kernel.org <netdev@...r.kernel.org>; moshe@...dia.com <moshe@...dia.com>; linux-rdma@...r.kernel.org <linux-rdma@...r.kernel.org>; phaddad@...dia.com <phaddad@...dia.com>
Subject: Re: Kernel panic triggered while removing mlx5_core devices from the pci bus
Hi Michal
can you please try the bellow change[1]?
In addition, the bug/trace is in mlx5_ib driver code, so I CC rdma ML
(linux-rdma@...r.kernel.org).
thanks
Shay Drory
[1]
--- a/drivers/infiniband/hw/mlx5/cq.c
+++ b/drivers/infiniband/hw/mlx5/cq.c
@@ -614,7 +614,6 @@ int mlx5_ib_poll_cq(struct ib_cq *ibcq, int
num_entries, struct ib_wc *wc)
int soft_polled = 0;
int npolled;
- spin_lock_irqsave(&cq->lock, flags);
if (mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR) {
/* make sure no soft wqe's are waiting */
if (unlikely(!list_empty(&cq->wc_list)))
@@ -625,6 +624,7 @@ int mlx5_ib_poll_cq(struct ib_cq *ibcq, int
num_entries, struct ib_wc *wc)
goto out;
}
+ spin_lock_irqsave(&cq->lock, flags);
if (unlikely(!list_empty(&cq->wc_list)))
soft_polled = poll_soft_wc(cq, num_entries, wc, false);
@@ -635,9 +635,9 @@ int mlx5_ib_poll_cq(struct ib_cq *ibcq, int
num_entries, struct ib_wc *wc)
if (npolled)
mlx5_cq_set_ci(&cq->mcq);
-out:
spin_unlock_irqrestore(&cq->lock, flags);
+out:
return soft_polled + npolled;
}
On 28/05/2024 9:18, Berger, Michal wrote:
> *External email: Use caution opening links or attachments*
>
>
> Hi Shay,
>
> Appreciate your feedback. I applied the suggested change on top of our
> 6.8.9 kernel build, but I am afraid it didn't solve the problem.
> Granted, the stacktrace doesn't point at the mlx5_health* anymore, but
> the panic happens exactly at the same time - it takes couple dozen of
> tries to trigger it, but it's still there. Attaching latest trace.
>
> Michal Berger
>
>
> Intel Technology Poland sp. z o.o. - ul. Slowackiego 173, 80-298 Gdansk
>
> KRS 101882
>
> NIP 957-07-52-316
>
> ------------------------------------------------------------------------
> *From:* Shay Drori <shayd@...dia.com>
> *Sent:* Sunday, May 26, 2024 2:35 PM
> *To:* Berger, Michal <michal.berger@...el.com>; netdev@...r.kernel.org
> <netdev@...r.kernel.org>; moshe@...dia.com <moshe@...dia.com>
> *Subject:* Re: Kernel panic triggered while removing mlx5_core devices
> from the pci bus
> Hi Michal.
>
> can you please try the bellow change[1]?
> we try it locally and it seems to solve the issue.
>
> thanks
> Shay Drory
>
> [1]
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c
> b/drivers/net/ethernet/mellanox/mlx5/core/main.c
> index 6574c145dc1e..459a836a5d9c 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
> @@ -1298,6 +1298,9 @@ static int mlx5_function_teardown(struct
> mlx5_core_dev *dev, bool boot)
> if (!err)
> mlx5_function_disable(dev, boot);
> + else
> + mlx5_stop_health_poll(dev, boot);
> +
> return err;
> }
>
>
>
> On 24/05/2024 11:07, Berger, Michal wrote:
>> Kernel: 6.7.0, 6.8.8 (fedora builds)
>> Devices: MT27710 Family [ConnectX-4 Lx] (0x1015), fw_ver: 14.23.1020
>> rdma-core: 44.0
>>
>> We have a small test which performs a somewhat controlled hotplug of the net device on the pci bus (via sysfs). The affected device is part of the nvmf-rdma setup running in SPDK context (i.e. https://github.com/spdk/spdk/blob/master/test/nvmf/target/device_removal.sh) <https://github.com/spdk/spdk/blob/master/test/nvmf/target/device_removal.sh)> Sometimes (it's not reproducible at each run unfortunately) when the device is removed, kernel hits
>> Oops - with our panic setup it's then followed by a kernel reboot, but if we allow the kernel to continue it eventually deadlocks itself.
>>
>> This happens across different systems using the same set of NICs. Example of these oops attached.
>>
>> Just to note, we previously had the same issue under older kernels (e.g. 6.1), all reported here https://bugzilla.kernel.org/show_bug.cgi?id=218288
> <https://bugzilla.kernel.org/show_bug.cgi?id=218288>. Bump to 6.7.0
> helped to reduce the frequency
>> of this issue but unfortunately it's still there.
>>
>> Any hints on how to tackle this issue would be appreciated.
>>
>> Regards,
>> Michal
>> ---------------------------------------------------------------------
>> Intel Technology Poland sp. z o.o.
>> ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
>> Spolka oswiadcza, ze posiada status duzego przedsiebiorcy w rozumieniu ustawy z dnia 8 marca 2013 r. o przeciwdzialaniu nadmiernym opoznieniom w transakcjach handlowych.
>>
>> Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione.
>> This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.
Content of type "text/html" skipped
Download attachment "oops1.log" of type "application/octet-stream" (8647 bytes)
Powered by blists - more mailing lists