linux-kernel - Re: [PATCH v7] NVMe: conversion to blk-mq

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <539B3F75.7040700@fb.com>
Date:	Fri, 13 Jun 2014 12:14:13 -0600
From:	Jens Axboe <axboe@...com>
To:	Keith Busch <keith.busch@...el.com>
CC:	Matias Bjørling <m@...rling.me>,
	Matthew Wilcox <willy@...ux.intel.com>,
	"sbradshaw@...ron.com" <sbradshaw@...ron.com>,
	"tom.leiming@...il.com" <tom.leiming@...il.com>,
	"hch@...radead.org" <hch@...radead.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"linux-nvme@...ts.infradead.org" <linux-nvme@...ts.infradead.org>
Subject: Re: [PATCH v7] NVMe: conversion to blk-mq

On 06/13/2014 09:16 AM, Keith Busch wrote:
> On Fri, 13 Jun 2014, Jens Axboe wrote:
>> On 06/13/2014 09:05 AM, Keith Busch wrote:
>>> Here are the performance drops observed with blk-mq with the existing
>>> driver as baseline:
>>>
>>>  CPU : Drop
>>>  ....:.....
>>>    0 : -6%
>>>    8 : -36%
>>>   16 : -12%
>>
>> We need the hints back for sure, I'll run some of the same tests and
>> verify to be sure. Out of curiousity, what is the topology like on your
>> box? Are 0/1 siblings, and 0..7 one node?
> 
> 0-7 are different cores on node 0, with 16-23 being their thread
> siblings. Similiar setup with 8-15 and 24-32 on node 1.

OK, same setup as mine. The affinity hint is really screwing us over, no
question about it. We just need a:

irq_set_affinity_hint(dev->entry[nvmeq->cq_vector].vector, hctx->cpumask);

in the ->init_hctx() methods to fix that up.

That brings us to roughly the same performance, except for the cases
where the dd is run on the thread sibling of the core handling the
interrupt. And granted, with the 16 queues used, that'll happen on
blk-mq. But since you have 32 threads and just 31 IO queues, the non
blk-mq driver must end up sharing for some cases, too.

So what do we care most about here? Consistency, or using all queues at
all costs?


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/