Message-ID: <85cebcb9-ce97-43f2-8da5-01c3a745fe2c@kernel.org>
Date: Mon, 1 Jul 2024 12:03:02 +0900
From: Damien Le Moal <dlemoal@...nel.org>
To: Yihang Li <liyihang9@...wei.com>, cassel@...nel.org
Cc: James.Bottomley@...senPartnership.com, martin.petersen@...cle.com,
 john.g.garry@...cle.com, yanaijie@...wei.com, linux-kernel@...r.kernel.org,
 linux-scsi@...r.kernel.org, linuxarm@...wei.com, chenxiang66@...ilicon.com,
 prime.zeng@...wei.com, "linux-pci@...r.kernel.org"
 <linux-pci@...r.kernel.org>, Bjorn Helgaas <bhelgaas@...gle.com>
Subject: Re: [bug report] scsi: SATA devices missing after FLR is triggered
 during HBA suspended

On 6/24/24 21:10, Yihang Li wrote:
>> Thank you for the explanation, but as Niklas said, it would be a lot easier for
>> me to recreate the issue if you send the exact commands you execute to trigger
>> the issue. E.g. "suspend all disks" in step a can have a lot of different
>> meaning depending on which type os suspend you are using... So please send the
>> exact commands you use.
>> is what exactly ? autosuspend ? or something else ?

I am failing to recreate the exact same issue. I do see a lot of bad things
happening though, but that is not looking like what you sent. I do endup with
the 4 drives connected on my HBA being disabled by libata as revalidate/IDENTIFY
fails. And even worse: I hit a deadlock on dev->mutex when I try to do "rmmod
pm80xx" after running your test.

I am using a pm80xx adapter as that is the only libsas adapter I have.

I think your test just kicked a big can of worms... There seem to be a lot of
wrong things going on, but I now need to sort out if the problems are with the
pm80xx driver, libsas, libata or sd. Probably a combination of all.

ATA device suspend/resume has been a constant source of issues since scsi layer
switched to doing PM operations asynchronouly. Your issue is latest one.
This will take a while to debug.

> In step a, I suspend all disks by issuing the following command to all disks
> attached to the SAS controller 0000:b4:02.0:
> [root@...alhost ~]# echo auto > /sys/devices/pci0000:b4/0000:b4:02.0/host6/port-6:0/end_device-6:0/target6:0:0/6:0:0:0/power/control
> [root@...alhost ~]# echo 5000 > /sys/devices/pci0000:b4/0000:b4:02.0/host6/port-6:0/end_device-6:0/target6:0:0/6:0:0:0/power/autosuspend_delay_ms
> ...
> [root@...alhost ~]# echo auto > /sys/devices/pci0000:b4/0000:b4:02.0/host6/port-6:6/end_device-6:6/target6:0:6/6:0:6:0/power/control
> [root@...alhost ~]# echo 5000 > /sys/devices/pci0000:b4/0000:b4:02.0/host6/port-6:6/end_device-6:6/target6:0:6/6:0:6:0/power/autosuspend_delay_ms

This works as expected on my system and I see my drives going to sleep after 5s.

> Step b, Suspend the SAS controller:
> [root@...alhost ~]# echo auto > /sys/devices/pci0000:b4/0000:b4:02.0/power/control

This has no effect for me. Can you confirm that your controller is actually
sleeping ? I.e., what do the following show ?

cat /sys/devices/pci0000:b4/0000:b4:02.0/power/runtime_active_kids
cat /sys/devices/pci0000:b4/0000:b4:02.0/power/runtime_status

?

> At this point, the SAS controller is suspended. Next step c is trigger PCI FLR.
> [root@...alhost ~]# echo 1 > /sys/bus/pci/devices/0000:b4:02.0/reset

What does

cat /sys/bus/pci/devices/0000:b4:02.0/reset_method

is on your system ?

Mine is "bus" only.

>>> The issue 2:
>>> a. Suspend all disks on controller B.
>>> b. Suspend controller B.
>>> c. Resuming all disks on controller B.
>>> d. Run the "lsmod" command to check the driver reference counting.

What is the reference count before you do step (a), after you run step (b) and
at step (d) ?

For my system using the pm80xx driver, I get:

pm80xx                352256  0
libsas                155648  1 pm80xx

before and after, and that is all normal. But there is the difference that
suspending the pm80xx controller does not seem to be supported and does nothing.

-- 
Damien Le Moal
Western Digital Research

Powered by blists - more mailing lists