linux-kernel - [PATCH 0/9] scsi_debug: Fix shost command overloading issue

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-Id: <20230313111019.1521680-1-john.g.garry@oracle.com>
Date:   Mon, 13 Mar 2023 11:10:10 +0000
From:   John Garry <john.g.garry@...cle.com>
To:     jejb@...ux.ibm.com, martin.petersen@...cle.com, hch@....de,
        dgilbert@...erlog.com, hare@...e.com, ming.lei@...hat.com,
        bvanassche@....org
Cc:     linux-scsi@...r.kernel.org, linux-kernel@...r.kernel.org,
        John Garry <john.g.garry@...cle.com>
Subject: [PATCH 0/9] scsi_debug: Fix shost command overloading issue

It's easy to get scsi_debug to error on throughput testing when we have
multiple hosts:

#lsscsi
[7:0:0:0]       disk    Linux   scsi_debug      0191
[0:0:0:0]       disk    Linux   scsi_debug      0191

# fio --filename=/dev/sda --filename=/dev/sdb --direct=1 --rw=read --bs=4k --iodepth=256 --runtime=60 --numjobs=40 --time_based --name=jpg --eta-newline=1 --readonly --ioengine=io_uring --hipri --exitall_on_error
jpg: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=256
...
fio-3.28
Starting 40 processes
[   27.521809] hrtimer: interrupt took 33067 ns
[   27.904660] sd 7:0:0:0: [sdb] tag#171 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK cmd_age=0s
[   27.904660] sd 0:0:0:0: [sda] tag#58 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK cmd_age=0s
fio: io_u error [   27.904667] sd 0:0:0:0: [sda] tag#58 CDB: Read(10) 28 00 00 00 27 00 00 01 18 00
on file /dev/sda[   27.904670] sd 0:0:0:0: [sda] tag#62 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK cmd_age=0s

The issue is related to how the driver manages submit queues and tags. A
single array of submit queues - sdebug_q_arr - with its own set of tags is
shared among all hosts. As such, for occasions when we have more than one
host it is possible to overload the submit queues and run out of tags.
Running out of tags makes the driver error in the submission path. We could
change the driver error code in this scenario to request SCSI ml to
requeue, but that is not a proper fix.

A solution to solving this multi-host problem could be by making the submit
queues per host. A downside there is that we make the driver considerably
more complex again.

There is another separate issue that we may reduce the host submit queue
depth, sdebug_max_queue, dynamically causing the host to be overloaded.
How many IOs which the host may be sent is fixed at can_queue at init
time, which is the same initial value for sdebug_max_queue. So reducing
sdebug_max_queue means that the host may be sent more IOs than it is
configured to handle, causing overloading.

About submit queues, they are not really required in the driver at all.
blk-mq and the SCSI ml already manages tags for us, so that the LLD does
not have to. The submit queues hold per-IO deferred command info in
sdebug_defer, but this could be dynamically allocated and assigned on a
per-IO basis.

This series removes the scsi_debug submit queue concept and uses
pre-existing APIs to manage and examine tags, like scsi_block_requests()
and blk_mq_tagset_busy_iter(). Using standard APIs makes the driver more
maintainable and extensible in future.

The last patch is an RFC as the LLD should not be calling
sbitmap_queue_resize() for the host tags.

This series is based on my earlier series
https://lore.kernel.org/linux-scsi/20230313093114.1498305-1-john.g.garry@oracle.com/T/#m80f1de854ac590ce79c577e307ee2ba94a3534dd

John Garry (9):
  scsi: scsi_debug: Don't iter all hosts in
    clear_luns_changed_on_target()
  scsi: scsi_debug: Change host list lock to a mutex
  scsi: scsi_debug: Protect block_unblock_all_queues() with mutex
  scsi: scsi_debug: Use scsi_block_requests() to block queues
  scsi: scsi_debug: Dynamically allocate sdebug_queued_cmd
  scsi: scsi_debug: Use blk_mq_tagset_busy_iter() in
    sdebug_blk_mq_poll()
  scsi: scsi_debug: Use blk_mq_tagset_busy_iter() in stop_all_queued()
  scsi: scsi_debug: Use scsi_host_busy() in delay_store() and
    ndelay_store()
  scsi: scsi_debug: Drop sdebug_queue concept

 drivers/scsi/scsi_debug.c | 767 +++++++++++++++++---------------------
 1 file changed, 342 insertions(+), 425 deletions(-)

-- 
2.35.3