linux-kernel - [RFC PATCH 0/3] softirq/blk-mq: implement interrupt flood detection for avoiding cpu lockup

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 18 Dec 2019 15:19:39 +0800
From:   Ming Lei <ming.lei@...hat.com>
To:     Thomas Gleixner <tglx@...utronix.de>, Jens Axboe <axboe@...nel.dk>
Cc:     linux-kernel@...r.kernel.org, linux-block@...r.kernel.org,
        Ming Lei <ming.lei@...hat.com>, Long Li <longli@...rosoft.com>,
        Ingo Molnar <mingo@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Keith Busch <keith.busch@...el.com>,
        Christoph Hellwig <hch@....de>,
        Sagi Grimberg <sagi@...mberg.me>,
        John Garry <john.garry@...wei.com>,
        Hannes Reinecke <hare@...e.com>
Subject: [RFC PATCH 0/3] softirq/blk-mq: implement interrupt flood detection for avoiding cpu lockup

Hi Guys,

For minimizing IO latency, blk-mq often completes IO in the hardware
interrupt context. However, in case of multiple HBAs, or multiple
storage devices attached to same HBA, if one CPU core has to handle
interrupts of IOs submitted from multile CPU , there is risk to lock
up CPUs[1][2].

Follows the idea when there is mutliple IO submitter and single
CPU for completing these IOs:

1) in case of multiple storages attached to one single HBA, these
storages may handle IO more quickly than single CPU core

2) in case of multiple HBAs in one system, one single effective CPU
can be selected for handling interrupts from several queues, then
multiple storages still may handle IO more quickly than single CPU
core.

When handling IO completion in interrupt context, IO latency is good,
but there is risk to lock up CPU. When moving IO completion to process
context via NAPI or threaded interrupt handler, CPU lockup can be
avoided. So people try to move part of IO completion into process
context for avoiding CPU lockup, meantime trying to not hurt IO
performance, such as Keith's work[3].

However, it is hard to partition IO completion in the two contexts, if
less work is moved to process context, risk of locking up CPU can't be
eliminated; if more work is moved to process context, extra IO latency is
introduced, then IO performance is hurt.

Interrupt flood information can be one useful hint for partitioning IO
completion work into the two contexts effectively.

The 1st two patches implement interrpupt flood detection, and the 3rd
patch applies the hint to complet IO in process context when interrupt
flood happens. This way avoids CPU lockup, meantime IO performance isn't
hurt obviously.

[1] https://lore.kernel.org/lkml/1566281669-48212-1-git-send-email-longli@linuxonhyperv.com/
[2] https://lore.kernel.org/lkml/a7ef3810-31af-013a-6d18-ceb6154aa2ef@huawei.com/
[3] https://lore.kernel.org/linux-nvme/20191209175622.1964-1-kbusch@kernel.org/T/#t

Ming Lei (3):
  sched/core: add API for exporting runqueue clock
  softirq: implement interrupt flood detection
  blk-mq: complete request in rescuer process context in case of irq
    flood

 block/blk-mq.c          |  68 ++++++++++++++++-
 drivers/base/cpu.c      |  23 ++++++
 include/linux/hardirq.h |   2 +
 include/linux/sched.h   |   2 +
 kernel/sched/core.c     |   5 ++
 kernel/softirq.c        | 161 ++++++++++++++++++++++++++++++++++++++++
 6 files changed, 260 insertions(+), 1 deletion(-)

Cc: Long Li <longli@...rosoft.com>
Cc: Ingo Molnar <mingo@...hat.com>,
Cc: Peter Zijlstra <peterz@...radead.org>
Cc: Keith Busch <keith.busch@...el.com>
Cc: Christoph Hellwig <hch@....de>
Cc: Sagi Grimberg <sagi@...mberg.me>
Cc: John Garry <john.garry@...wei.com>
Cc: Thomas Gleixner <tglx@...utronix.de>
Cc: Hannes Reinecke <hare@...e.com>
-- 
2.20.1