| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAK1f24mwzXa8Az5WFYu+1UopTCStDWx3yDr1RugLwphS-hWizw@mail.gmail.com>
Date: Tue, 13 Aug 2024 13:00:53 +0800
From: Lance Yang <ioworker0@...il.com>
To: Yu Kuai <yukuai1@...weicloud.com>
Cc: Michal Koutný <mkoutny@...e.com>, linux-mm@...ck.org,
linux-kernel@...r.kernel.org, linux-block@...r.kernel.org,
cgroups@...r.kernel.org, josef@...icpanda.com, tj@...nel.org,
fujita.tomonori@....ntt.co.jp, boqun.feng@...il.com, a.hindborg@...sung.com,
paolo.valente@...more.it, axboe@...nel.dk, vbabka@...nel.org,
david@...hat.com, 21cnbao@...il.com, baolin.wang@...ux.alibaba.com,
libang.li@...group.com, "yukuai (C)" <yukuai3@...wei.com>
Subject: Re: [BUG] cgroupv2/blk: inconsistent I/O behavior in Cgroup v2 with
set device wbps and wiops
Hi Kuai,
Thanks a lot for jumping in!
On Tue, Aug 13, 2024 at 9:37 AM Yu Kuai <yukuai1@...weicloud.com> wrote:
>
> Hi,
>
> 在 2024/08/12 23:43, Michal Koutný 写道:
> > +Cc Kuai
> >
> > On Mon, Aug 12, 2024 at 11:00:30PM GMT, Lance Yang <ioworker0@...il.com> wrote:
> >> Hi all,
> >>
> >> I've run into a problem with Cgroup v2 where it doesn't seem to correctly limit
> >> I/O operations when I set both wbps and wiops for a device. However, if I only
> >> set wbps, then everything works as expected.
> >>
> >> To reproduce the problem, we can follow these command-based steps:
> >>
> >> 1. **System Information:**
> >> - Kernel Version and OS Release:
> >> ```
> >> $ uname -r
> >> 6.10.0-rc5+
> >>
> >> $ cat /etc/os-release
> >> PRETTY_NAME="Ubuntu 24.04 LTS"
> >> NAME="Ubuntu"
> >> VERSION_ID="24.04"
> >> VERSION="24.04 LTS (Noble Numbat)"
> >> VERSION_CODENAME=noble
> >> ID=ubuntu
> >> ID_LIKE=debian
> >> HOME_URL="https://www.ubuntu.com/"
> >> SUPPORT_URL="https://help.ubuntu.com/"
> >> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
> >> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
> >> UBUNTU_CODENAME=noble
> >> LOGO=ubuntu-logo
> >> ```
> >>
> >> 2. **Device Information and Settings:**
> >> - List Block Devices and Scheduler:
> >> ```
> >> $ lsblk
> >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
> >> sda 8:0 0 4.4T 0 disk
> >> └─sda1 8:1 0 4.4T 0 part /data
> >> ...
> >>
> >> $ cat /sys/block/sda/queue/scheduler
> >> none [mq-deadline] kyber bfq
> >>
> >> $ cat /sys/block/sda/queue/rotational
> >> 1
> >> ```
> >>
> >> 3. **Reproducing the problem:**
> >> - Navigate to the cgroup v2 filesystem and configure I/O settings:
> >> ```
> >> $ cd /sys/fs/cgroup/
> >> $ stat -fc %T /sys/fs/cgroup
> >> cgroup2fs
> >> $ mkdir test
> >> $ echo "8:0 wbps=10485760 wiops=100000" > io.max
> >> ```
> >> In this setup:
> >> wbps=10485760 sets the write bytes per second limit to 10 MB/s.
> >> wiops=100000 sets the write I/O operations per second limit to 100,000.
> >>
> >> - Add process to the cgroup and verify:
> >> ```
> >> $ echo $$ > cgroup.procs
> >> $ cat cgroup.procs
> >> 3826771
> >> 3828513
> >> $ ps -ef|grep 3826771
> >> root 3826771 3826768 0 22:04 pts/1 00:00:00 -bash
> >> root 3828761 3826771 0 22:06 pts/1 00:00:00 ps -ef
> >> root 3828762 3826771 0 22:06 pts/1 00:00:00 grep --color=auto 3826771
> >> ```
> >>
> >> - Observe I/O performance using `dd` commands and `iostat`:
> >> ```
> >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 &
> >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 &
>
> You're testing buffer IO here, and I don't see that write back cgroup is
> enabled. Is this test intentional? Why not test direct IO?
Yes, I was testing buffered I/O and can confirm that CONFIG_CGROUP_WRITEBACK
was enabled.
$ cat /boot/config-6.10.0-rc5+ |grep CONFIG_CGROUP_WRITEBACK
CONFIG_CGROUP_WRITEBACK=y
We intend to configure both wbps (write bytes per second) and wiops
(write I/O operations
per second) for the containers. IIUC, this setup will effectively
restrict both their block device
I/Os and buffered I/Os.
> Why not test direct IO?
I was testing direct IO as well. However it did not work as expected with
`echo "8:0 wbps=10485760 wiops=100000" > io.max`.
$ time dd if=/dev/zero of=/data/file7 bs=512M count=1 oflag=direct
1+0 records in
1+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 51.5962 s, 10.4 MB/s
real 0m51.637s
user 0m0.000s
sys 0m0.313s
$ iostat -d 1 -h -y -p sda
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
12.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
12.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
11.00 0.0k 1.4M 0.0k 0.0k 1.4M
0.0k sda
11.00 0.0k 1.4M 0.0k 0.0k 1.4M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
11.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
11.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
55.00 0.0k 1.8M 0.0k 0.0k 1.8M
0.0k sda
55.00 0.0k 1.8M 0.0k 0.0k 1.8M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
11.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
11.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
11.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
11.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
14.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
14.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
12.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
12.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
14.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
14.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
12.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
12.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
13.00 0.0k 1.4M 0.0k 0.0k 1.4M
0.0k sda
13.00 0.0k 1.4M 0.0k 0.0k 1.4M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
12.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
12.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
13.00 0.0k 1.4M 0.0k 0.0k 1.4M
0.0k sda
13.00 0.0k 1.4M 0.0k 0.0k 1.4M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
12.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
12.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
18.00 0.0k 1.4M 0.0k 0.0k 1.4M
0.0k sda
18.00 0.0k 1.4M 0.0k 0.0k 1.4M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
11.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
11.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
12.00 0.0k 1.4M 0.0k 0.0k 1.4M
0.0k sda
12.00 0.0k 1.4M 0.0k 0.0k 1.4M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
1804.00 0.0k 445.8M 0.0k 0.0k 445.8M
0.0k sda
1804.00 0.0k 445.8M 0.0k 0.0k 445.8M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
4.00 0.0k 24.0k 0.0k 0.0k 24.0k
0.0k sda
4.00 0.0k 24.0k 0.0k 0.0k 24.0k
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
0.00 0.0k 0.0k 0.0k 0.0k 0.0k
0.0k sda
0.00 0.0k 0.0k 0.0k 0.0k 0.0k
0.0k sda1
There are two things that confuse me. First, initially, neither the
wbps nor the wiops
reached their limits. Second, in the last second, the wbps far
exceeded the limit.
But if I only set wbps, then everything works as expected with
`echo "8:0 wbps=10485760 wiops=max" > io.max`
> >> ```
> >> ```
> >> $ iostat -d 1 -h -y -p sda
> >>
> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> >> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda
> >> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1
> >>
> >>
> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda
> >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1
> >>
> >>
> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> >> 21.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda
> >> 21.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda1
> >>
> >>
> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda
> >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1
> >>
> >>
> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda
> >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1
> >>
> >>
> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> >> 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M 0.0k sda
> >> 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M 0.0k sda1
>
> Looks like all dirty buffer got flushed to disk at the last second while
> the file is closed, this is expected.
The dd command completed in less than a second, but flushing all the
dirty buffers to
disk took a much longer time. By the time the flushing was completed,
the file had
already been closed, IIUC.
$ time dd if=/dev/zero of=/data/file5 bs=512M count=1
1+0 records in
1+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 0.531944 s, 1.0 GB/s
real 0m0.578s
user 0m0.000s
sys 0m0.576s
$ iostat -d 1 -h -y -p sda
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
0.00 0.0k 0.0k 0.0k 0.0k 0.0k
0.0k sda
0.00 0.0k 0.0k 0.0k 0.0k 0.0k
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
74.00 0.0k 664.0k 0.0k 0.0k 664.0k
0.0k sda
74.00 0.0k 664.0k 0.0k 0.0k 664.0k
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
15.00 0.0k 1.1M 0.0k 0.0k 1.1M
0.0k sda
15.00 0.0k 1.1M 0.0k 0.0k 1.1M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
5.00 0.0k 1.2M 0.0k 0.0k 1.2M
0.0k sda
5.00 0.0k 1.2M 0.0k 0.0k 1.2M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
5.00 0.0k 1.2M 0.0k 0.0k 1.2M
0.0k sda
5.00 0.0k 1.2M 0.0k 0.0k 1.2M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
5.00 0.0k 1.2M 0.0k 0.0k 1.2M
0.0k sda
5.00 0.0k 1.2M 0.0k 0.0k 1.2M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
5.00 0.0k 1.2M 0.0k 0.0k 1.2M
0.0k sda
5.00 0.0k 1.2M 0.0k 0.0k 1.2M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
5.00 0.0k 1.2M 0.0k 0.0k 1.2M
0.0k sda
5.00 0.0k 1.2M 0.0k 0.0k 1.2M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
13.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
13.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
5.00 0.0k 1.2M 0.0k 0.0k 1.2M
0.0k sda
5.00 0.0k 1.2M 0.0k 0.0k 1.2M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
12.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
12.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
9.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
11.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
11.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
46.00 0.0k 1.7M 0.0k 0.0k 1.7M
0.0k sda
46.00 0.0k 1.7M 0.0k 0.0k 1.7M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
7.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
11.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
11.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
5.00 0.0k 1.2M 0.0k 0.0k 1.2M
0.0k sda
5.00 0.0k 1.2M 0.0k 0.0k 1.2M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
6.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
11.00 0.0k 1.4M 0.0k 0.0k 1.4M
0.0k sda
11.00 0.0k 1.4M 0.0k 0.0k 1.4M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
15.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
15.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
8.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
15.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
15.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda
10.00 0.0k 1.3M 0.0k 0.0k 1.3M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
49.00 0.0k 1.6M 0.0k 0.0k 1.6M
0.0k sda
49.00 0.0k 1.6M 0.0k 0.0k 1.6M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
53.00 0.0k 1.6M 0.0k 0.0k 1.6M
0.0k sda
53.00 0.0k 1.6M 0.0k 0.0k 1.6M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
1805.00 0.0k 448.4M 0.0k 0.0k 448.4M
0.0k sda
1805.00 0.0k 448.4M 0.0k 0.0k 448.4M
0.0k sda1
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
0.00 0.0k 0.0k 0.0k 0.0k 0.0k
0.0k sda
0.00 0.0k 0.0k 0.0k 0.0k 0.0k
0.0k sda1
> >> ```
> >> Initially, the write speed is slow (<2MB/s) then suddenly bursts to several
> >> hundreds of MB/s.
> >
> > What it would be on average?
> > IOW how long would the whole operation in throttled cgroup take?
> >
> >>
> >> - Testing with wiops set to max:
> >> ```
> >> echo "8:0 wbps=10485760 wiops=max" > io.max
> >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 &
> >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 &
> >> ```
> >> ```
> >> $ iostat -d 1 -h -y -p sda
> >>
> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> >> 48.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda
> >> 48.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1
> >>
> >>
> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> >> 40.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda
> >> 40.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1
> >>
> >>
> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> >> 41.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda
> >> 41.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1
> >>
> >>
> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> >> 46.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda
> >> 46.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1
> >>
> >>
> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
> >> 55.00 0.0k 10.2M 0.0k 0.0k 10.2M 0.0k sda
> >> 55.00 0.0k 10.2M 0.0k 0.0k 10.2M 0.0k sda1
>
> And I don't this wiops=max is the reason, what need to explain is that
> why dirty buffer got flushed to disk synchronously before the dd finish
> and close the file?
The dd command operates in the background, and it seems that the dirty
buffers begin
to flush after the command has completed.
>
> >> ```
> >> The iostat output shows the write operations as stabilizing at around 10 MB/s,
> >> which aligns with the defined limit of 10 MB/s. After setting wiops to max, the
> >> I/O limits appear to work as expected.
>
> Can you give the direct IO a test? And also enable write back cgroup for
> buffer IO.
>
> Thanks,
> Kuai
Thanks a lot again for your time!
Lance
>
> >>
> >>
> >> Thanks,
> >> Lance
> >
> > Thanks for the report Lance. Is this something you started seeing after
> > a kernel update or switch to cgroup v2? (Or you simply noticed with this
> > setup only?)
> >
> >
> > Michal
> >
>
Powered by blists - more mailing lists