| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Tue, 21 Dec 2021 19:23:53 +0530
From: Kashyap Desai <kashyap.desai@...adcom.com>
To: John Garry <john.garry@...wei.com>, axboe@...nel.dk
Cc: linux-block@...r.kernel.org, linux-kernel@...r.kernel.org,
ming.lei@...hat.com,
Sathya Prakash Veerichetty <sathya.prakash@...adcom.com>
Subject: RE: [PATCH RFT] blk-mq: optimize queue tag busy iter for shared_tags
>
> On 21/12/2021 12:31, Kashyap Desai wrote:
>
> Hi Kashyap,
>
> What kernel is this for? 5.17 or 5.16 + stable? Your intention is not
> clear to
> me.
Hi John
This is for current/5.17. This patch is meaningfully only on top of [1].
[1] " blk-mq: Use shared tags for shared sbitmap support" Commit -
e155b0c238b20f0a866f4334d292656665836c8a
While doing additional testing for [1], I noticed some performance issue.
Along with the performance issue, I noticed CPU lockup as well. Lockup
trace -
_raw_spin_lock_irqsave+0x42/0x50
blk_mq_find_and_get_req+0x20/0xa0
bt_iter+0x2d/0x80
blk_mq_queue_tag_busy_iter+0x1aa/0x2f0
? blk_mq_complete_request+0x30/0x30
? blk_mq_complete_request+0x30/0x30
? __schedule+0x360/0x850
blk_mq_timeout_work+0x5e/0x120
process_one_work+0x1a8/0x380
worker_thread+0x30/0x380
? wq_calc_node_cpumask.isra.30+0x100/0x100
kthread+0x167/0x190
? set_kthread_struct+0x40/0x40
ret_from_fork+0x22/0x30
It is a generic performance issue if driver use " shost->host_tagset = 1".
In fact, I found that [1] is useful to fix performance issue and provided
this additional patch.
I changed my setup to have 64 scsi_devices (earlier I just kept 16 or 24
drives, so did not noticed this issue). Performance/cpu lockup issue is not
due to [1].
More number of scsi device, hardware context per host and high queue depth
will increase the chances of lockup and performance drop.
Do you think, it is good to have changes in 5.16 + stable ?
I don't know if this patch will create any side effect. Can you review and
let me know your feedback. ?
Kashyap
>
>
> > In [0], CPU usage for blk_mq_queue_tag_busy_iter() was optimized, but
> > there are still periodic call of blk_mq_queue_tag_busy_iter() from
> > below context. Below context is used for block layer timer to find out
> > potential expired command (per request queue) which requires tag
> > iteration
> > almost every 5 seconds(defined BLK_MAX_TIMEOUT) for each request
> queue.
> >
> > kthread
> > worker_thread
> > process_one_work
> > blk_mq_timeout_work
> > blk_mq_queue_tag_busy_iter
> > bt_iter
> > blk_mq_find_and_get_req
> > _raw_spin_lock_irqsave
> > native_queued_spin_lock_slowpath
> >
> > Changes in this patch optimize extra iterations of tags in case of
> > shared_tags. One iteration of shared_tags can give expected results for
> > iterate function.
> >
> > Setup - AMD64 Gen-4.0 Server.
> > 64 Virtual Drive created using 16 Nvme drives + mpi3mr driver (in
> > shared_tags mode)
> >
> > Test command -
> > fio 64.fio --rw=randread --bs=4K --iodepth=32 --numjobs=2 --
> ioscheduler=mq-deadline --disk_util=0
> >
> > Without this patch on 5.16.0-rc5, mpi3mr driver in shared_tags mode can
> > give 4.0M IOPs vs expected to get ~6.0M.
> > Snippet of perf top
> >
> > 25.42% [kernel] [k]
> > native_queued_spin_lock_slowpath
> > 3.95% [kernel] [k] cpupri_set
> > 2.05% [kernel] [k]
> > __blk_mq_get_driver_tag
> > 1.67% [kernel] [k] __rcu_read_unlock
> > 1.63% [kernel] [k]
> > check_preemption_disabled
> >
> > After applying this patch on 5.16.0-rc5, mpi3mr driver in shared_tags
> > mode reach up to 5.8M IOPs.
> >
> > Snippet of perf top
> >
> > 7.95% [kernel] [k]
> > native_queued_spin_lock_slowpath
> > 5.61% [kernel] [k] cpupri_set
> > 2.98% [kernel] [k]
> > acpi_processor_ffh_cstate_enter
> > 2.49% [kernel] [k] read_tsc
> > 2.15% [kernel] [k]
> > check_preemption_disabled
> >
> >
> > [0]
> https://lore.kernel.org/all/9b092ca49e9b5415772cd950a3c12584@mail.gma
> il.com/
> >
> >
> > Cc: linux-block@...r.kernel.org
> > Cc: linux-kernel@...r.kernel.org
> > Cc: john.garry@...wei.com
> > Cc: ming.lei@...hat.com
> > Cc: sathya.prakash@...adcom.com
> > Signed-off-by: Kashyap Desai <kashyap.desai@...adcom.com>
> > ---
> > block/blk-mq-tag.c | 11 ++++++++++-
> > 1 file changed, 10 insertions(+), 1 deletion(-)
> >
> > diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
> > index 995336abee33..3e0a8e79f966 100644
> > --- a/block/blk-mq-tag.c
> > +++ b/block/blk-mq-tag.c
> > @@ -253,7 +253,8 @@ static bool bt_iter(struct sbitmap *bitmap, unsigned
> int bitnr, void *data)
> > if (!rq)
> > return true;
> >
> > - if (rq->q == hctx->queue && rq->mq_hctx == hctx)
> > + if (rq->q == hctx->queue && (rq->mq_hctx == hctx ||
> > + blk_mq_is_shared_tags(hctx->flags)))
> > ret = iter_data->fn(hctx, rq, iter_data->data, reserved);
> > blk_mq_put_rq_ref(rq);
> > return ret;
> > @@ -484,6 +485,14 @@ void blk_mq_queue_tag_busy_iter(struct
> request_queue *q, busy_iter_fn *fn,
> > if (tags->nr_reserved_tags)
> > bt_for_each(hctx, &tags->breserved_tags, fn, priv,
> true);
> > bt_for_each(hctx, &tags->bitmap_tags, fn, priv, false);
> > +
> > + /* In case of shared bitmap if shared_tags is allocated, it is not
> required
> > + * to iterate all the hctx. Looping one hctx is good enough.
> > + */
> > + if (blk_mq_is_shared_tags(hctx->flags)) {
> > + blk_queue_exit(q);
> > + return;
>
> this looks like v5.16-rc6 code
>
> > + }
> > }
> > blk_queue_exit(q);
> > }
> >
>
>
>
> Thanks,
> John
Download attachment "smime.p7s" of type "application/pkcs7-signature" (4212 bytes)
Powered by blists - more mailing lists