Message-ID: <dd2da83e-6749-dc85-77dc-24864ac9a887@amd.com>
Date: Wed, 7 Feb 2024 22:42:35 +0530
From: Smita Koralahalli <Smita.KoralahalliChannabasappa@....com>
To: Ilpo Järvinen <ilpo.jarvinen@...ux.intel.com>
Cc: linux-pci@...r.kernel.org, LKML <linux-kernel@...r.kernel.org>,
 Bjorn Helgaas <bhelgaas@...gle.com>,
 Mahesh J Salgaonkar <mahesh@...ux.ibm.com>, Lukas Wunner <lukas@...ner.de>,
 Yazen Ghannam <yazen.ghannam@....com>
Subject: Re: [PATCH v8] PCI/DPC: Ignore Surprise Down error on hot removal

On 2/7/2024 5:05 PM, Ilpo Järvinen wrote:
> On Wed, 7 Feb 2024, Smita Koralahalli wrote:
> 
>> According to PCIe r6.0 sec 6.7.6 [1], async removal with DPC may result in
>> surprise down error. This error is expected and is just a side-effect of
>> async remove.
>>
>> Ignore surprise down error generated as a side-effect of async remove.
>> Typically, this error is benign as the pciehp handler invoked by PDC
>> or/and DLLSC alongside DPC, de-enumerates and brings down the device
>> appropriately. But the error messages might confuse users. Get rid of
>> these irritating log messages with a 1s delay while pciehp waits for
>> dpc recovery.
> 
> dpc -> DPC.

Will fix.

> 
>> The implementation is as follows: On an async remove a DPC is triggered
>> along with a Presence Detect State change and/or DLL State Change.
>> Determine it's an async remove by checking for DPC Trigger Status in DPC
>> Status Register and Surprise Down Error Status in AER Uncorrected Error
>> Status to be non-zero. If true, treat the DPC event as a side-effect of
>> async remove, clear the error status registers and continue with hot-plug
>> tear down routines. If not, follow the existing routine to handle AER and
>> DPC errors.
>>
>> Please note that, masking Surprise Down Errors was explored as an
>> alternative approach, but left due to the odd behavior that masking only
>> avoids the interrupt, but still records an error per PCIe r6.0.1 Section
>> 6.2.3.2.2. That stale error is going to be reported the next time some
>> error other than Surprise Down is handled.
>>
>> Dmesg before:
>>
>>    pcieport 0000:00:01.4: DPC: containment event, status:0x1f01 source:0x0000
>>    pcieport 0000:00:01.4: DPC: unmasked uncorrectable error detected
>>    pcieport 0000:00:01.4: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID)
>>    pcieport 0000:00:01.4:   device [1022:14ab] error status/mask=00000020/04004000
>>    pcieport 0000:00:01.4:    [ 5] SDES (First)
>>    nvme nvme2: frozen state error detected, reset controller
>>    pcieport 0000:00:01.4: DPC: Data Link Layer Link Active not set in 1000 msec
>>    pcieport 0000:00:01.4: AER: subordinate device reset failed
>>    pcieport 0000:00:01.4: AER: device recovery failed
>>    pcieport 0000:00:01.4: pciehp: Slot(16): Link Down
>>    nvme2n1: detected capacity change from 1953525168 to 0
>>    pci 0000:04:00.0: Removing from iommu group 49
>>
>> Dmesg after:
>>
>>   pcieport 0000:00:01.4: pciehp: Slot(16): Link Down
>>   nvme1n1: detected capacity change from 1953525168 to 0
>>   pci 0000:04:00.0: Removing from iommu group 37
>>
>> [1] PCI Express Base Specification Revision 6.0, Dec 16 2021.
>>      https://members.pcisig.com/wg/PCI-SIG/document/16609
>>
>> Signed-off-by: Smita Koralahalli <Smita.KoralahalliChannabasappa@....com>
>> Reviewed-by: Lukas Wunner <lukas@...ner.de>
>> Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@...ux.intel.com>
>> ---
>> v2:
>> 	Indentation is taken care. (Bjorn)
>> 	Unrelevant dmesg logs are removed. (Bjorn)
>> 	Rephrased commit message, to be clear on native vs FW-First
>> 	handling. (Bjorn and Sathyanarayanan)
>> 	Prefix changed from pciehp_ to dpc_. (Lukas)
>> 	Clearing ARI and AtomicOp Requester are performed as a part of
>> 	(de-)enumeration in pciehp_unconfigure_device(). (Lukas)
>> 	Changed to clearing all optional capabilities in DEVCTL2.
>> 	OS-First -> native. (Sathyanarayanan)
>>
>> v3:
>> 	Added error message when root port become inactive.
>> 	Modified commit description to add more details.
>> 	Rearranged code comments and function calls with no functional
>> 	change.
>> 	Additional check for is_hotplug_bridge.
>> 	dpc_completed_waitqueue to wakeup pciehp handler.
>> 	Cleared only Fatal error detected in DEVSTA.
>>
>> v4:
>> 	Made read+write conditional on "if (pdev->dpc_rp_extensions)"
>> 	for DPC_RP_PIO_STATUS.
>> 	Wrapped to 80 chars.
>> 	Code comment for clearing PCI_STATUS and PCI_EXP_DEVSTA.
>> 	Added pcie_wait_for_link() check.
>> 	Removed error message for root port inactive as the message
>> 	already existed.
>> 	Check for is_hotplug_bridge before registers read.
>> 	Section 6.7.6 of the PCIe Base Spec 6.0 -> PCIe r6.0 sec 6.7.6.
>> 	Made code comment more meaningful.
>>
>> v5:
>> 	$SUBJECT correction.
>> 	Added "Reviewed-by" tag.
>> 	No code changes. Re-spin on latest base to get Bjorn's
>> 	attention.
>>
>> v6:
>> 	Change to write 1's to clear error. (Sathyanarayanan)
>>
>> v7:
>> 	No changes. Rebasing on pci main branch as per Bjorn comments.
>>
>> v8:
>> 	Just return "status & PCI_ERR_UNC_SURPDN" instead of true and
>> 	false and allow C to handle the conversion to bool. (Ilpo)
>> ---
>>   drivers/pci/pcie/dpc.c | 64 ++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 64 insertions(+)
>>
>> diff --git a/drivers/pci/pcie/dpc.c b/drivers/pci/pcie/dpc.c
>> index 94111e438241..ba7240a2ba2f 100644
>> --- a/drivers/pci/pcie/dpc.c
>> +++ b/drivers/pci/pcie/dpc.c
>> @@ -303,10 +303,74 @@ void dpc_process_error(struct pci_dev *pdev)
>>   	}
>>   }
>>   
>> +static void pci_clear_surpdn_errors(struct pci_dev *pdev)
>> +{
>> +	u32 reg32;
>> +
>> +	if (pdev->dpc_rp_extensions) {
>> +		pci_read_config_dword(pdev, pdev->dpc_cap + PCI_EXP_DPC_RP_PIO_STATUS,
>> +				      &reg32);
>> +		pci_write_config_dword(pdev, pdev->dpc_cap + PCI_EXP_DPC_RP_PIO_STATUS,
>> +				       reg32);
>> +	}
>> +
>> +	/*
>> +	 * In practice, Surprise Down errors have been observed to also set
>> +	 * error bits in the Status Register as well as the Fatal Error
>> +	 * Detected bit in the Device Status Register.
>> +	 */
>> +	pci_write_config_word(pdev, PCI_STATUS, 0xffff);
>> +
>> +	pcie_capability_write_word(pdev, PCI_EXP_DEVSTA, PCI_EXP_DEVSTA_FED);
>> +}
>> +
>> +static void dpc_handle_surprise_removal(struct pci_dev *pdev)
>> +{
>> +	if (!pcie_wait_for_link(pdev, false)) {
>> +		pci_info(pdev, "Data Link Layer Link Active not cleared in 1000 msec\n");
>> +		goto out;
>> +	}
>> +
>> +	if (pdev->dpc_rp_extensions && dpc_wait_rp_inactive(pdev))
>> +		goto out;
>> +
>> +	pci_aer_raw_clear_status(pdev);
>> +	pci_clear_surpdn_errors(pdev);
>> +
>> +	pci_write_config_word(pdev, pdev->dpc_cap + PCI_EXP_DPC_STATUS,
>> +			      PCI_EXP_DPC_STATUS_TRIGGER);
>> +
>> +out:
>> +	clear_bit(PCI_DPC_RECOVERED, &pdev->priv_flags);
>> +	wake_up_all(&dpc_completed_waitqueue);
>> +}
>> +
>> +static bool dpc_is_surprise_removal(struct pci_dev *pdev)
>> +{
>> +	u16 status;
>> +
>> +	if (!pdev->is_hotplug_bridge)
>> +		return false;
>> +
>> +	pci_read_config_word(pdev, pdev->aer_cap + PCI_ERR_UNCOR_STATUS,
>> +			     &status);
>> +
>> +	return status & PCI_ERR_UNC_SURPDN;
> 
> Thanks for the update, looks more readable now.
> 
> While staring this now again, I think checking for error from
> pci_read_config_word() would be useful so that it cannot confused with
> surprise down?
> 

You are right. I should have it like this:
.
if (pci_read_config_word(pdev, pdev->aer_cap + PCI_ERR_UNCOR_STATUS,
			 &status)
	return false;

return status & PCI_ERR_UNC_SURPDN;

Let me change and send v9.

Thanks
Smita

Powered by blists - more mailing lists