linux-kernel - Re: [PATCH v2] blk-throtl: Introduce sync and async queues for blk-throtl

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20221222133912.GA20830@blackbody.suse.cz>
Date:   Thu, 22 Dec 2022 14:39:12 +0100
From:   Michal Koutný <mkoutny@...e.com>
To:     Jinke Han <hanjinke.666@...edance.com>
Cc:     tj@...nel.org, josef@...icpanda.com, axboe@...nel.dk,
        cgroups@...r.kernel.org, linux-block@...r.kernel.org,
        linux-kernel@...r.kernel.org, yinxin.x@...edance.com
Subject: Re: [PATCH v2] blk-throtl: Introduce sync and async queues for
 blk-throtl

Hello Jinke.

On Wed, Dec 21, 2022 at 06:42:46PM +0800, Jinke Han <hanjinke.666@...edance.com> wrote:
> In our test, fio writes a 100g file in sequential 4k blocksize in
> a container with low bps limit configured (wbps=10M). More than 1200
> ios were throttled in blk-throtl queue and the avarage throtle time
> of each io is 140s. At the same time, the operation of saving a small
> file by vim will be blocked amolst 140s. As a fsync will be send by vim,
> the sync ios of fsync will be blocked by a huge amount of buffer write
> ios ahead. This is also a priority inversion problem within one cgroup.
> In the database scene, things got really bad with blk-throtle enabled
> as fsync is called very often.

I'm trying to make sense of the numbers:
- at 10 MB/s, it's 0.4 ms per 4k block
- there are 1.2k throttled bios that gives waiting time of roughly 0.5s
  ~ 0.4ms * 1200
- you say that you observe 280 times longer throttling time,
- that'd mean there should be 340k queued bios 
  - or cummulative dispatch of ~1400 MB of data

So what are the queued quantities? Are there more than 1200 bios or are
they bigger than the 4k you mention?

Thanks for clarification.

(I acknowledge the possible problem with a large population of async
writes delaying scarce sync writes.)

Michal

Download attachment "signature.asc" of type "application/pgp-signature" (229 bytes)