| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <9ede36af-fca4-ed41-6b7e-cef157c640bb@huaweicloud.com> Date: Tue, 13 Aug 2024 09:37:03 +0800 From: Yu Kuai <yukuai1@...weicloud.com> To: Michal Koutný <mkoutny@...e.com>, Lance Yang <ioworker0@...il.com> Cc: linux-mm@...ck.org, linux-kernel@...r.kernel.org, linux-block@...r.kernel.org, cgroups@...r.kernel.org, josef@...icpanda.com, tj@...nel.org, fujita.tomonori@....ntt.co.jp, boqun.feng@...il.com, a.hindborg@...sung.com, paolo.valente@...more.it, axboe@...nel.dk, vbabka@...nel.org, david@...hat.com, 21cnbao@...il.com, baolin.wang@...ux.alibaba.com, libang.li@...group.com, "yukuai (C)" <yukuai3@...wei.com> Subject: Re: [BUG] cgroupv2/blk: inconsistent I/O behavior in Cgroup v2 with set device wbps and wiops Hi, 在 2024/08/12 23:43, Michal Koutný 写道: > +Cc Kuai > > On Mon, Aug 12, 2024 at 11:00:30PM GMT, Lance Yang <ioworker0@...il.com> wrote: >> Hi all, >> >> I've run into a problem with Cgroup v2 where it doesn't seem to correctly limit >> I/O operations when I set both wbps and wiops for a device. However, if I only >> set wbps, then everything works as expected. >> >> To reproduce the problem, we can follow these command-based steps: >> >> 1. **System Information:** >> - Kernel Version and OS Release: >> ``` >> $ uname -r >> 6.10.0-rc5+ >> >> $ cat /etc/os-release >> PRETTY_NAME="Ubuntu 24.04 LTS" >> NAME="Ubuntu" >> VERSION_ID="24.04" >> VERSION="24.04 LTS (Noble Numbat)" >> VERSION_CODENAME=noble >> ID=ubuntu >> ID_LIKE=debian >> HOME_URL="https://www.ubuntu.com/" >> SUPPORT_URL="https://help.ubuntu.com/" >> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" >> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" >> UBUNTU_CODENAME=noble >> LOGO=ubuntu-logo >> ``` >> >> 2. **Device Information and Settings:** >> - List Block Devices and Scheduler: >> ``` >> $ lsblk >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS >> sda 8:0 0 4.4T 0 disk >> └─sda1 8:1 0 4.4T 0 part /data >> ... >> >> $ cat /sys/block/sda/queue/scheduler >> none [mq-deadline] kyber bfq >> >> $ cat /sys/block/sda/queue/rotational >> 1 >> ``` >> >> 3. **Reproducing the problem:** >> - Navigate to the cgroup v2 filesystem and configure I/O settings: >> ``` >> $ cd /sys/fs/cgroup/ >> $ stat -fc %T /sys/fs/cgroup >> cgroup2fs >> $ mkdir test >> $ echo "8:0 wbps=10485760 wiops=100000" > io.max >> ``` >> In this setup: >> wbps=10485760 sets the write bytes per second limit to 10 MB/s. >> wiops=100000 sets the write I/O operations per second limit to 100,000. >> >> - Add process to the cgroup and verify: >> ``` >> $ echo $$ > cgroup.procs >> $ cat cgroup.procs >> 3826771 >> 3828513 >> $ ps -ef|grep 3826771 >> root 3826771 3826768 0 22:04 pts/1 00:00:00 -bash >> root 3828761 3826771 0 22:06 pts/1 00:00:00 ps -ef >> root 3828762 3826771 0 22:06 pts/1 00:00:00 grep --color=auto 3826771 >> ``` >> >> - Observe I/O performance using `dd` commands and `iostat`: >> ``` >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 & >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 & You're testing buffer IO here, and I don't see that write back cgroup is enabled. Is this test intentional? Why not test direct IO? >> ``` >> ``` >> $ iostat -d 1 -h -y -p sda >> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device >> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda >> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 >> >> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 >> >> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device >> 21.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda >> 21.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda1 >> >> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 >> >> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 >> >> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device >> 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M 0.0k sda >> 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M 0.0k sda1 Looks like all dirty buffer got flushed to disk at the last second while the file is closed, this is expected. >> ``` >> Initially, the write speed is slow (<2MB/s) then suddenly bursts to several >> hundreds of MB/s. > > What it would be on average? > IOW how long would the whole operation in throttled cgroup take? > >> >> - Testing with wiops set to max: >> ``` >> echo "8:0 wbps=10485760 wiops=max" > io.max >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 & >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 & >> ``` >> ``` >> $ iostat -d 1 -h -y -p sda >> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device >> 48.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda >> 48.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1 >> >> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device >> 40.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda >> 40.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1 >> >> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device >> 41.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda >> 41.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1 >> >> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device >> 46.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda >> 46.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1 >> >> >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device >> 55.00 0.0k 10.2M 0.0k 0.0k 10.2M 0.0k sda >> 55.00 0.0k 10.2M 0.0k 0.0k 10.2M 0.0k sda1 And I don't this wiops=max is the reason, what need to explain is that why dirty buffer got flushed to disk synchronously before the dd finish and close the file? >> ``` >> The iostat output shows the write operations as stabilizing at around 10 MB/s, >> which aligns with the defined limit of 10 MB/s. After setting wiops to max, the >> I/O limits appear to work as expected. Can you give the direct IO a test? And also enable write back cgroup for buffer IO. Thanks, Kuai >> >> >> Thanks, >> Lance > > Thanks for the report Lance. Is this something you started seeing after > a kernel update or switch to cgroup v2? (Or you simply noticed with this > setup only?) > > > Michal >
Powered by blists - more mailing lists