[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAK1f24=R9tgxGB3T3JPsHAhg+hAgvRg+-K-gWT_+WYVOTRtSpA@mail.gmail.com>
Date: Tue, 13 Aug 2024 14:17:30 +0800
From: Lance Yang <ioworker0@...il.com>
To: Yu Kuai <yukuai1@...weicloud.com>
Cc: Michal Koutný <mkoutny@...e.com>, linux-mm@...ck.org,
linux-kernel@...r.kernel.org, linux-block@...r.kernel.org,
cgroups@...r.kernel.org, josef@...icpanda.com, tj@...nel.org,
fujita.tomonori@....ntt.co.jp, boqun.feng@...il.com, a.hindborg@...sung.com,
paolo.valente@...more.it, axboe@...nel.dk, vbabka@...nel.org,
david@...hat.com, 21cnbao@...il.com, baolin.wang@...ux.alibaba.com,
libang.li@...group.com, "yukuai (C)" <yukuai3@...wei.com>
Subject: Re: [BUG] cgroupv2/blk: inconsistent I/O behavior in Cgroup v2 with
set device wbps and wiops
I just realized that bursts appear to depend on the maximum values of wbps and
wiops, when wiops is not `max`?
Thanks,
Lance
On Tue, Aug 13, 2024 at 1:00 PM Lance Yang <ioworker0@...il.com> wrote:
>
> Hi Kuai,
>
> Thanks a lot for jumping in!
>
> On Tue, Aug 13, 2024 at 9:37 AM Yu Kuai <yukuai1@...weicloud.com> wrote:
> >
> > Hi,
> >
> > 在 2024/08/12 23:43, Michal Koutný 写道:
> > > +Cc Kuai
> > >
> > > On Mon, Aug 12, 2024 at 11:00:30PM GMT, Lance Yang <ioworker0@...il.com> wrote:
> > >> Hi all,
> > >>
> > >> I've run into a problem with Cgroup v2 where it doesn't seem to correctly limit
> > >> I/O operations when I set both wbps and wiops for a device. However, if I only
> > >> set wbps, then everything works as expected.
> > >>
> > >> To reproduce the problem, we can follow these command-based steps:
> > >>
> > >> 1. **System Information:**
> > >> - Kernel Version and OS Release:
> > >> ```
> > >> $ uname -r
> > >> 6.10.0-rc5+
> > >>
> > >> $ cat /etc/os-release
> > >> PRETTY_NAME="Ubuntu 24.04 LTS"
> > >> NAME="Ubuntu"
> > >> VERSION_ID="24.04"
> > >> VERSION="24.04 LTS (Noble Numbat)"
> > >> VERSION_CODENAME=noble
> > >> ID=ubuntu
> > >> ID_LIKE=debian
> > >> HOME_URL="https://www.ubuntu.com/"
> > >> SUPPORT_URL="https://help.ubuntu.com/"
> > >> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
> > >> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
> > >> UBUNTU_CODENAME=noble
> > >> LOGO=ubuntu-logo
> > >> ```
> > >>
> > >> 2. **Device Information and Settings:**
> > >> - List Block Devices and Scheduler:
> > >> ```
> > >> $ lsblk
> > >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
> > >> sda 8:0 0 4.4T 0 disk
> > >> └─sda1 8:1 0 4.4T 0 part /data
> > >> ...
> > >>
> > >> $ cat /sys/block/sda/queue/scheduler
> > >> none [mq-deadline] kyber bfq
> > >>
> > >> $ cat /sys/block/sda/queue/rotational
> > >> 1
> > >> ```
> > >>
> > >> 3. **Reproducing the problem:**
> > >> - Navigate to the cgroup v2 filesystem and configure I/O settings:
> > >> ```
> > >> $ cd /sys/fs/cgroup/
> > >> $ stat -fc %T /sys/fs/cgroup
> > >> cgroup2fs
> > >> $ mkdir test
> > >> $ echo "8:0 wbps=10485760 wiops=100000" > io.max
> > >> ```
> > >> In this setup:
> > >> wbps=10485760 sets the write bytes per second limit to 10 MB/s.
> > >> wiops=100000 sets the write I/O operations per second limit to 100,000.
> > >>
> > >> - Add process to the cgroup and verify:
> > >> ```
> > >> $ echo $$ > cgroup.procs
> > >> $ cat cgroup.procs
> > >> 3826771
> > >> 3828513
> > >> $ ps -ef|grep 3826771
> > >> root 3826771 3826768 0 22:04 pts/1 00:00:00 -bash
> > >> root 3828761 3826771 0 22:06 pts/1 00:00:00 ps -ef
> > >> root 3828762 3826771 0 22:06 pts/1 00:00:00 grep --color=auto 3826771
> > >> ```
> > >>
> > >> - Observe I/O performance using `dd` commands and `iostat`:
> > >> ```
> > >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 &
> > >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 &
> >
> > You're testing buffer IO here, and I don't see that write back cgroup is
> > enabled. Is this test intentional? Why not test direct IO?
>
> Yes, I was testing buffered I/O and can confirm that CONFIG_CGROUP_WRITEBACK
> was enabled.
>
> $ cat /boot/config-6.10.0-rc5+ |grep CONFIG_CGROUP_WRITEBACK
> CONFIG_CGROUP_WRITEBACK=y
>
> We intend to configure both wbps (write bytes per second) and wiops
> (write I/O operations
> per second) for the containers. IIUC, this setup will effectively
> restrict both their block device
> I/Os and buffered I/Os.
>
> > Why not test direct IO?
>
> I was testing direct IO as well. However it did not work as expected with
> `echo "8:0 wbps=10485760 wiops=100000" > io.max`.
>
> $ time dd if=/dev/zero of=/data/file7 bs=512M count=1 oflag=direct
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB, 512 MiB) copied, 51.5962 s, 10.4 MB/s
>
> real 0m51.637s
> user 0m0.000s
> sys 0m0.313s
>
> $ iostat -d 1 -h -y -p sda
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda
> 11.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 55.00 0.0k 1.8M 0.0k 0.0k 1.8M
> 0.0k sda
> 55.00 0.0k 1.8M 0.0k 0.0k 1.8M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 14.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 14.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 14.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 14.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 13.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda
> 13.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 13.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda
> 13.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 18.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda
> 18.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 12.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda
> 12.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 1804.00 0.0k 445.8M 0.0k 0.0k 445.8M
> 0.0k sda
> 1804.00 0.0k 445.8M 0.0k 0.0k 445.8M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 4.00 0.0k 24.0k 0.0k 0.0k 24.0k
> 0.0k sda
> 4.00 0.0k 24.0k 0.0k 0.0k 24.0k
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 0.00 0.0k 0.0k 0.0k 0.0k 0.0k
> 0.0k sda
> 0.00 0.0k 0.0k 0.0k 0.0k 0.0k
> 0.0k sda1
>
> There are two things that confuse me. First, initially, neither the
> wbps nor the wiops
> reached their limits. Second, in the last second, the wbps far
> exceeded the limit.
>
> But if I only set wbps, then everything works as expected with
> `echo "8:0 wbps=10485760 wiops=max" > io.max`
>
> > >> ```
> > >> ```
> > >> $ iostat -d 1 -h -y -p sda
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda
> > >> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda
> > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 21.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda
> > >> 21.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda
> > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda
> > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M 0.0k sda
> > >> 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M 0.0k sda1
> >
> > Looks like all dirty buffer got flushed to disk at the last second while
> > the file is closed, this is expected.
>
> The dd command completed in less than a second, but flushing all the
> dirty buffers to
> disk took a much longer time. By the time the flushing was completed,
> the file had
> already been closed, IIUC.
>
> $ time dd if=/dev/zero of=/data/file5 bs=512M count=1
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB, 512 MiB) copied, 0.531944 s, 1.0 GB/s
>
> real 0m0.578s
> user 0m0.000s
> sys 0m0.576s
>
> $ iostat -d 1 -h -y -p sda
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 0.00 0.0k 0.0k 0.0k 0.0k 0.0k
> 0.0k sda
> 0.00 0.0k 0.0k 0.0k 0.0k 0.0k
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 74.00 0.0k 664.0k 0.0k 0.0k 664.0k
> 0.0k sda
> 74.00 0.0k 664.0k 0.0k 0.0k 664.0k
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 15.00 0.0k 1.1M 0.0k 0.0k 1.1M
> 0.0k sda
> 15.00 0.0k 1.1M 0.0k 0.0k 1.1M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 13.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 13.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 12.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 9.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 46.00 0.0k 1.7M 0.0k 0.0k 1.7M
> 0.0k sda
> 46.00 0.0k 1.7M 0.0k 0.0k 1.7M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 11.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda
> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 6.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 11.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda
> 11.00 0.0k 1.4M 0.0k 0.0k 1.4M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 15.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 15.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 8.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 15.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 15.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda
> 10.00 0.0k 1.3M 0.0k 0.0k 1.3M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 49.00 0.0k 1.6M 0.0k 0.0k 1.6M
> 0.0k sda
> 49.00 0.0k 1.6M 0.0k 0.0k 1.6M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 53.00 0.0k 1.6M 0.0k 0.0k 1.6M
> 0.0k sda
> 53.00 0.0k 1.6M 0.0k 0.0k 1.6M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 1805.00 0.0k 448.4M 0.0k 0.0k 448.4M
> 0.0k sda
> 1805.00 0.0k 448.4M 0.0k 0.0k 448.4M
> 0.0k sda1
>
>
> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
> kB_dscd Device
> 0.00 0.0k 0.0k 0.0k 0.0k 0.0k
> 0.0k sda
> 0.00 0.0k 0.0k 0.0k 0.0k 0.0k
> 0.0k sda1
>
> > >> ```
> > >> Initially, the write speed is slow (<2MB/s) then suddenly bursts to several
> > >> hundreds of MB/s.
> > >
> > > What it would be on average?
> > > IOW how long would the whole operation in throttled cgroup take?
> > >
> > >>
> > >> - Testing with wiops set to max:
> > >> ```
> > >> echo "8:0 wbps=10485760 wiops=max" > io.max
> > >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 &
> > >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 &
> > >> ```
> > >> ```
> > >> $ iostat -d 1 -h -y -p sda
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 48.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda
> > >> 48.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 40.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda
> > >> 40.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 41.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda
> > >> 41.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 46.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda
> > >> 46.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1
> > >>
> > >>
> > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> > >> 55.00 0.0k 10.2M 0.0k 0.0k 10.2M 0.0k sda
> > >> 55.00 0.0k 10.2M 0.0k 0.0k 10.2M 0.0k sda1
> >
> > And I don't this wiops=max is the reason, what need to explain is that
> > why dirty buffer got flushed to disk synchronously before the dd finish
> > and close the file?
>
> The dd command operates in the background, and it seems that the dirty
> buffers begin
> to flush after the command has completed.
>
> >
> > >> ```
> > >> The iostat output shows the write operations as stabilizing at around 10 MB/s,
> > >> which aligns with the defined limit of 10 MB/s. After setting wiops to max, the
> > >> I/O limits appear to work as expected.
> >
> > Can you give the direct IO a test? And also enable write back cgroup for
> > buffer IO.
> >
> > Thanks,
> > Kuai
>
>
> Thanks a lot again for your time!
> Lance
>
> >
> > >>
> > >>
> > >> Thanks,
> > >> Lance
> > >
> > > Thanks for the report Lance. Is this something you started seeing after
> > > a kernel update or switch to cgroup v2? (Or you simply noticed with this
> > > setup only?)
> > >
> > >
> > > Michal
> > >
> >
Powered by blists - more mailing lists