netdev - Re: EEH recovery failing on mlx5 card

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <5c8e2eba-88c9-5913-ad18-db272484adf5@nvidia.com>
Date: Mon, 17 Jul 2023 19:18:50 +0300
From: Moshe Shemesh <moshe@...dia.com>
To: Leon Romanovsky <leon@...nel.org>, Ganesh G R <ganeshgr@...ux.ibm.com>
CC: <saeedm@...dia.com>, <netdev@...r.kernel.org>, <oohall@...il.com>, "Mahesh
 Salgaonkar" <mahesh@...ux.ibm.com>
Subject: Re: EEH recovery failing on mlx5 card



On 7/17/2023 4:10 PM, Leon Romanovsky wrote:
> 
> + Moshe
> 
> On Mon, Jul 17, 2023 at 12:48:37PM +0530, Ganesh G R wrote:
>> Hi,
>>
>> mlx5 cards are failing to recover from PCI errors, Upon investigation we found that the
>> driver is trying to do MMIO in the middle of EEH error handling.
>> The following fix in mlx5_pci_err_detected() is fixing the issue, Do you think its the right fix?
>>
>> @@ -1847,6 +1847,7 @@ static pci_ers_result_t mlx5_pci_err_detected(struct pci_dev *pdev,
>>          mlx5_unload_one(dev, true);
>>          mlx5_drain_health_wq(dev);
>>          mlx5_pci_disable_device(dev);
>> +       cancel_delayed_work_sync(&clock->timer.overflow_work);
>>          res = state == pci_channel_io_perm_failure ?
>>                  PCI_ERS_RESULT_DISCONNECT : PCI_ERS_RESULT_NEED_RESET;
>>

Hi Ganesh,
Thanks for pointing to this issue and its solution!
I would rather fix it in the work itself.
Please test this fix instead :

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c 
b/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c
index 973babfaff25..2ad0bcc0f1b1 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c
@@ -227,10 +227,15 @@ static void mlx5_timestamp_overflow(struct 
work_struct *work)
         clock = container_of(timer, struct mlx5_clock, timer);
         mdev = container_of(clock, struct mlx5_core_dev, clock);

+       if (mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR)
+              goto out;
+
         write_seqlock_irqsave(&clock->lock, flags);
         timecounter_read(&timer->tc);
         mlx5_update_clock_info_page(mdev);
         write_sequnlock_irqrestore(&clock->lock, flags);
+
+out:
         schedule_delayed_work(&timer->overflow_work, 
timer->overflow_period);
  }


>> Regards
>> Ganesh