linux-kernel - Re: [REQUEST DISCUSS]: speed up SCSI error handle for host with massive devices

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1dd69d03-b4f6-ab20-4923-0995b40f045d@suse.de>
Date:   Wed, 30 Mar 2022 11:32:03 +0200
From:   Hannes Reinecke <hare@...e.de>
To:     Wenchao Hao <haowenchao@...wei.com>,
        Steffen Maier <maier@...ux.ibm.com>,
        linux-scsi@...r.kernel.org,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "James E.J. Bottomley" <jejb@...ux.ibm.com>,
        "Martin K. Petersen" <martin.petersen@...cle.com>,
        Mike Christie <michael.christie@...cle.com>,
        Lee Duncan <lduncan@...e.com>,
        John Garry <john.garry@...wei.com>
Cc:     Wu Bo <wubo40@...wei.com>, Feilong Lin <linfeilong@...wei.com>,
        zhangjian013@...wei.com
Subject: Re: [REQUEST DISCUSS]: speed up SCSI error handle for host with
 massive devices

On 3/30/22 11:11, Wenchao Hao wrote:
> On 2022/3/30 2:56, Hannes Reinecke wrote:
>> On 3/29/22 14:40, Wenchao Hao wrote:
>>> On 2022/3/29 18:56, Steffen Maier wrote:
>>>> On 3/29/22 11:06, Wenchao Hao wrote:
>>>>> SCSI timeout would call scsi_eh_scmd_add() on some conditions, host would be set
>>>>> to SHOST_RECOVERY state. Once host enter SHOST_RECOVERY, IOs submitted to all
>>>>> devices in this host would not succeed until the scsi_error_handler() finished.
>>>>> The scsi_error_handler() might takes long time to be done, it's unbearable when
>>>>> host has massive devices.
>>>>>
>>>>> I want to ask is anyone applying another error handler flow to address this
>>>>> phenomenon?
>>>>>
>>>>> I think we can move some operations(like scsi get sense, scsi send startunit
>>>>> and scsi device reset) out of scsi_unjam_host(), to perform these operations
>>>>> without setting host to SHOST_RECOVERY? It would reduce the time of block the
>>>>> whole host.
>>>>>
>>>>> Waiting for your discussion.
>>>>
>>>> We already have "async" aborts before even entering scsi_eh. So your use case seems to imply that those aborts fail and we enter scsi_eh?
>>>>
>>>
>>> Yes, I mean when scsi_abort_command() failed and scsi_eh_scmd_add() is called.
>>>
>>>> There's eh_deadline for limiting the time spent in escalation of scsi_eh, and instead directly go to host reset. Would this help?
>>>>
>>>>
>>>
>>> The deadline seems not helpful. What we want to see is a single LUN's command error
>>> would not stop other LUNs which share the same host. So my plan is to move reset LUN out
>>> from scsi_unjam_host() which run with host set to SHOST_RECOVERY.
>>
>> Nope. One of the key points of scsi_unjam_host() is that is has to stop all I/O before proceeding. Without doing so basically all SCSI parallel HBAs will fail EH as they _require_ I/O to be stopped.
>>
> 
> I still can not understand why we must stop all I/O. In my comprehension, stopping all I/O
> is because we might reset host during scsi_error_handler() and we must wait host's number of
> failed command equal to number of busy command then we can wake up scsi_error_handler().
> 
> If move reset LUN out of scsi_error_handler(), and perform single LUN reset, we only need
> stop I/O of this single LUN, this would not affect other LUNs. If single LUN reset failed,
> we can then call in the large scale error handle.
> 
I know the EH flow.

Problem here is the way parallel SCSI operates. Remember, parallel SCSI 
is a _bus_, and there can be only one command at a time on the bus.
So if one command on the bus misfires and you have to start EH you have 
to stop all I/O on the bus to ensure that your EH command is the only 
one active on the bus.

For modern HBAs we sure can device other ways and means of error 
recovery, but I can't really see how we would do that on legacy HBAs.

> Here is a brief flow:
> 
> abort command
>     ||
>     || failed
>     ||
>     \/
> stop single LUN's I/O (need to wait LUN's failed command number equal to busy command  number)
>     ||
>     || failed  (according to our statistic, 90% reset LUN would succeed)
>     ||
>     \/

Interesting. This does not match up with my experience, where 99% of the 
errors were due to a command timeout.

So which errors do you see here? What are the causes?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@...e.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer