lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Mon, 14 Jul 2014 12:13:26 +0300
From:	Sagi Grimberg <>
To:	Christoph Hellwig <>,
	James Bottomley <>,
	Jens Axboe <>,
	Bart Van Assche <>,
	Robert Elliott <>,,
CC:	Or Gerlitz <>, Oren Duer <>,
	"Nicholas A. Bellinger" <>,
	Mike Christie <>,
	Bart Van Assche <>
Subject: Re: scsi-mq V2

On 7/8/2014 5:48 PM, Christoph Hellwig wrote:
> I've pushed out a new scsi-mq.3 branch, which has been rebased on the
> latest core-for-3.17 tree + the "RFC: clean up command setup" series
> from June 29th.  Robert Elliot found a problem with not fully zeroed
> out UNMAP CDBs, which is fixed by the saner discard handling in that
> series.
> There is a new patch to factor the code from the above series for
> blk-mq use, which I've attached below.  Besides that the only changes
> are minor merge fixups in the main blk-mq usage patch.

Hey Christoph & Co,

I'd like to share some benchmarks I took on this patch set using iSER 
initiator (+2 pre-submitted performance improvements) vs LIO iSER target.
I ran workloads I think are interesting use-cases (single LUN with 1,2,4 
IO threads up to a fully occupied system doing IO to multiple LUNs).
Overall (except 2 strange anomalies) seems that scsi-mq patches 
(use_blk_mq=N) roughly sustains traditional scsi performance.
On the other hand scsi-mq code path (use_blk_mq=Y) on its own clearly 
shows better performance (tables below).

At first I too hit the aio issues discussed in this thread and converted 
to scsi-mq.3-no-rebase for testing (thanks Doug & Rob for raising it).
I must say that for some reason I get very low numbers for writes vs. 
reads (writes perf stuck at ~20K IOPs per thread), this happens
on 3.16-rc2 even before scsi-mq patches. Did anyone step on this as well 
or is it just a weird problem I'm having in my setup?
Anyway this is why my benchmarks shows only randread IO pattern (getting 
familiar numbers). I need to figure out whats wrong
with IO writes - I'll start bisecting on this.

I also reviewed the patch set and at this point, I don't have any 
comments. So you can add to the series:
Reviewed-by: Sagi Grimberg '<>' (or Tested-by - 
whatever you choose).

I want to state that I tested a traditional iSER initiator - no scsi-mq 
adoption at all.
I started looking into adopting scsi-mq to iSCSI/iSER recently and I 
must that say the scsi-mq adoption is not so
trivial due to iSCSI session-wide CmdSN/StatSN ordering constraints 
(can't just use more RDMA channels per connection...)
I'll be on vacation for the next couple of weeks, so I'll start a 
separate thread to get the community input on this matter.

Results: table entries are KIOPS(CPU%)
3.16-rc2 (scsi-mq patches reverted)
    Threads/LUN   1           2            4
  1             231(6.5%)   355(18.5%)   337(31.1%)
  2             446(13.6%)  673(37.2%)   654(49.8%)
  4             594(25%)    960(49.41%) 1165(99.3%)
  8            1018(50.3%) 1563(99.6%)  1696(99.9%)
  16           1660(86.5%) 1731(99.6%)  1710(100%)

3.16-rc2 (scsi-mq included, use_blk_mq=N)
    Threads/LUN   1           2            4
  1             231(6.5%)   351(18.5%)   337(31.4%)
  2             446(13.6%)  660(37.3%)   647(50%)
  4             591(25%)    967(49.7%)  1136(98.1%)
  8            1014(52.1%) 1296(100%)   1470(100%)
  16           1741(100%)  1761(100%)   1853(100%)

3.16-rc2 (scsi-mq included, use_blk_mq=Y)
    Threads/LUN   1           2            4
  1             265(6.4%)   465(13.4%)   572(27.9%)
  2             507(13.4%)  902(27.8%)  1034(45.9%)
  4             697(25%)   1197(49.5%)  1477(98.6%)
  8            1257(53.6%) 1856(98.7%)  1906(100%)
  16           1991(100%)  2021(100%)   2020(100%)

     - IOPs measurements are the average of a 60 seconds runs.
     - The CPU measurement is the total usage across all CPUs, In order
       to understand per-CPU utilization value should be normalized to 16
     - scsi_mq (use_blk_mq=N) has roughly the same performance as
       traditional scsi IO path but I see an anomaly in test cases
       {8 LUNs, 2/4 threads per LUN}. This may result in NUMA
       misalignment for threads/interrupts – requires further
     - iSER initiator has no Multi-Queue awareness.

Testing environment:
     - Initiator and target systems of 16 (8x2) cores (Hyperthreading
     - CPU model: Intel(R) Xeon(R) @ 2.60GHz
     - Block Layer settings:
     - scheduler=noop
         - rq_affinity=1
         - add_random=0
         - nomerges=1
     - Single FDR link between the target and initiator.
     - Device model: Mellanox ConnectIB (the numbers are also familiar
       with Mellanox ConnectX-3).
     - MSIX interrupt vectors were spread across system cores.
     - irqbalancer was disabled.
     - scsi_host settings:
         - cmd_per_lun=32 (default)
         - can_queue=113 (default)
     - In the multi-LUN test cases, each LUN exposed via different
       scsi_host (iSCSI session).

     - fio version: 2.0.13
     - LIO iSER target (target-pending for-next)
     - Null backing devices (NULLIO)
     - Upstream based iSER initiator + internal pre-submitted
       performance enhancements.

fio configuration:

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to
More majordomo info at
Please read the FAQ at

Powered by blists - more mailing lists