linux-kernel - Re: blk-mq: bitmap tag: performance degradation?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CACVXFVO-1shNB=tSpNBp_tpx+9imx5ycxfaSwofg-VeR_4S+MA@mail.gmail.com>
Date:	Fri, 6 Jun 2014 07:33:23 +0800
From:	Ming Lei <tom.leiming@...il.com>
To:	Jens Axboe <axboe@...nel.dk>
Cc:	Alexander Gordeev <agordeev@...hat.com>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: blk-mq: bitmap tag: performance degradation?

On Fri, Jun 6, 2014 at 1:17 AM, Jens Axboe <axboe@...nel.dk> wrote:
> On 06/05/2014 08:16 AM, Ming Lei wrote:
>> On Thu, Jun 5, 2014 at 10:03 PM, Jens Axboe <axboe@...nel.dk> wrote:
>>> On 2014-06-05 08:01, Alexander Gordeev wrote:
>>>>
>>>> On Wed, Jun 04, 2014 at 08:18:42AM -0600, Jens Axboe wrote:
>>>>>
>>>>> A null_blk test is the absolute best case for percpu_ida, since
>>>>> there are enough tags and everything is localized. The above test is
>>>>> more useful for testing blk-mq than any real world application of
>>>>> the tagging.
>>>>>
>>>>> I've done considerable testing on both 2 and 4 socket (32 and 64
>>>>> CPUs) and bitmap tagging is better in a much wider range of
>>>>> applications. This includes even high tag depth devices like nvme,
>>>>> and more normal ranges like mtip32xx and scsi-mq setups.
>>>>
>>>>
>>>> Just for the record: bitmap tags on a 48 CPU box with NVMe device
>>>> indeed shows almost the same performance/cache rate as the stock
>>>> kernel.
>>>
>>>
>>> Thanks for confirming. It's one of the dangers of null_blk, it's not always
>>> a very accurate simulation of what a real device will do. I think it's
>>> mostly a completion side thing, would be great with a small device that
>>> supported msi-x and could be used as an irq trigger :-)
>>
>> Maybe null_blk at IRQ_TIMER mode is more close to
>> a real device, and I guess the result may be different with
>> mode IRQ_NONE/IRQ_SOFTIRQ.
>
> It'd be closer in behavior, but the results might then be skewed by
> hitting the timer way too hard. And it'd be a general slowdown, again
> possibly skewing it. But I haven't tried with the timer completion, to
> see if that yields more accurate modelling for this test, so it might
> actually be a lot better.

My test on a 16core VM(host: 2 sockets, 16core):

1, bitmap tag allocation(3.15-rc7-next):
- softirq mode: 759K IOPS
- timer mode: 409K IOPS

2, percpu_ida allocation(3.15-rc7)
- softirq mode: 1116K IOPS
- timer mode: 411K IOPS

Also on real hardware, I remember there is no such big difference
between softirq mode and timer mode.

[global]
direct=1
size=128G
bsrange=4k-4k
timeout=20
numjobs=16
ioengine=libaio
iodepth=64
filename=/dev/nullb0
group_reporting=1

[f2]
stonewall
rw=randread


Thanks,
-- 
Ming Lei
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/