netdev - Re: EEH recovery failing on mlx5 card

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <9bdb9b9d-140a-7a28-f0de-2e64e873c068@nvidia.com>
Date: Wed, 19 Jul 2023 10:46:09 +0300
From: Moshe Shemesh <moshe@...dia.com>
To: Ganesh G R <ganeshgr@...ux.ibm.com>, Leon Romanovsky <leon@...nel.org>
CC: <saeedm@...dia.com>, <netdev@...r.kernel.org>, <oohall@...il.com>, "Mahesh
 Salgaonkar" <mahesh@...ux.ibm.com>
Subject: Re: EEH recovery failing on mlx5 card



On 7/18/2023 9:47 PM, Ganesh G R wrote:
> 
> On 7/17/23 9:48 PM, Moshe Shemesh wrote:
> 
>> On 7/17/2023 4:10 PM, Leon Romanovsky wrote:
>>> + Moshe
>>> On Mon, Jul 17, 2023 at 12:48:37PM +0530, Ganesh G R wrote:
>>>> Hi,
>>>> mlx5 cards are failing to recover from PCI errors, Upon investigation we found that the
>>>> driver is trying to do MMIO in the middle of EEH error handling.
>>>> The following fix in mlx5_pci_err_detected() is fixing the issue, Do you think its the right fix?
>>>> @@ -1847,6 +1847,7 @@ static pci_ers_result_t mlx5_pci_err_detected(struct pci_dev *pdev,
>>>>           mlx5_unload_one(dev, true);
>>>>           mlx5_drain_health_wq(dev);
>>>>           mlx5_pci_disable_device(dev);
>>>> +       cancel_delayed_work_sync(&clock->timer.overflow_work);
>>>>           res = state == pci_channel_io_perm_failure ?
>>>>                   PCI_ERS_RESULT_DISCONNECT : PCI_ERS_RESULT_NEED_RESET;
>> Hi Ganesh,
>> Thanks for pointing to this issue and its solution!
>> I would rather fix it in the work itself.
>> Please test this fix instead :
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c
>> b/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c
>> index 973babfaff25..2ad0bcc0f1b1 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c
>> @@ -227,10 +227,15 @@ static void mlx5_timestamp_overflow(struct
>> work_struct *work)
>>          clock = container_of(timer, struct mlx5_clock, timer);
>>          mdev = container_of(clock, struct mlx5_core_dev, clock);
>> +       if (mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR)
>> +              goto out;
>> +
>>          write_seqlock_irqsave(&clock->lock, flags);
>>          timecounter_read(&timer->tc);
>>          mlx5_update_clock_info_page(mdev);
>>          write_sequnlock_irqrestore(&clock->lock, flags);
>> +
>> +out:
>>          schedule_delayed_work(&timer->overflow_work,
>> timer->overflow_period);
>>   }
> Thanks Moshe, The fix looks clean and It worked.
> 

Great, thank you !