| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <73CA9E04-DD93-47E4-B0FE-0A12EC991DEF@linaro.org>
Date: Sat, 31 Aug 2019 09:10:26 +0200
From: Paolo Valente <paolo.valente@...aro.org>
To: Tejun Heo <tj@...nel.org>
Cc: Jens Axboe <axboe@...nel.dk>, newella@...com, clm@...com,
Josef Bacik <josef@...icpanda.com>, dennisz@...com,
Li Zefan <lizefan@...wei.com>,
Johannes Weiner <hannes@...xchg.org>,
linux-kernel <linux-kernel@...r.kernel.org>,
linux-block <linux-block@...r.kernel.org>, kernel-team@...com,
cgroups@...r.kernel.org, ast@...nel.org, daniel@...earbox.net,
kafai@...com, songliubraving@...com, yhs@...com,
bpf@...r.kernel.org
Subject: Re: [PATCHSET block/for-next] IO cost model based work-conserving
porportional controller
Hi Tejun,
thank you very much for this extra information, I'll try the
configuration you suggest. In this respect, is this still the branch
to use
https://kernel.googlesource.com/pub/scm/linux/kernel/git/tj/cgroup/+/refs/heads/review-iocost-v2
also after the issue spotted two days ago [1]?
Thanks,
Paolo
[1] https://lkml.org/lkml/2019/8/29/910
> Il giorno 31 ago 2019, alle ore 08:53, Tejun Heo <tj@...nel.org> ha scritto:
>
> Hello, Paolo.
>
> On Thu, Aug 22, 2019 at 10:58:22AM +0200, Paolo Valente wrote:
>> Ok, I tried with the parameters reported for a SATA SSD:
>>
>> rpct=95.00 rlat=10000 wpct=95.00 wlat=20000 min=50.00 max=400.00
>
> Sorry, I should have explained it with a lot more details.
>
> There are two things - the cost model and qos params. The default SSD
> cost model parameters are derived by averaging a number of mainstream
> SSD parameters. As a ballpark, this can be good enough because while
> the overall performance varied quite a bit from one ssd to another,
> the relative cost of different types of IOs wasn't drastically
> different.
>
> However, this means that the performance baseline can easily be way
> off from 100% depending on the specific device in use. In the above,
> you're specifying min/max which limits how far the controller is
> allowed to adjust the overall cost estimation. 50% and 400% are
> numbers which may make sense if the cost model parameter is expected
> to fall somewhere around 100% - ie. if the parameters are for that
> specific device.
>
> In your script, you're using default model params but limiting vrate
> range. It's likely that your device is significantly slower than what
> the default parameters are expecting. However, because min vrate is
> limited to 50%, it doesn't throttle below 50% of the estimated cost,
> so if the device is significantly slower than that, nothing gets
> controlled.
>
>> and with a simpler configuration [1]: one target doing random reads
>
> And without QoS latency targets, the controller is purely going by
> queue depth depletion which works fine for many usual workloads such
> as larger reads and writes but isn't likely to serve low-concurrency
> latency-sensitive IOs well.
>
>> and only four interferers doing sequential reads, with all the
>> processes (groups) having the same weight.
>>
>> But there seemed to be little or no control on I/O, because the target
>> got only 1.84 MB/s, against 1.15 MB/s without any control.
>>
>> So I tried with rlat=1000 and rlat=100.
>
> And this won't do anything as all rlat/wlat does is regulating how the
> overall vrate should be adjusted and it's being min'd at 50%.
>
>> Control did improve, with same results for both values of rlat. The
>> problem is that these results still seem rather bad, both in terms of
>> throughput guaranteed to the target and in terms of total throughput.
>> Here are results compared with BFQ (throughputs measured in MB/s):
>>
>> io.weight BFQ
>> target's throughput 3.415 6.224
>> total throughput 159.14 321.375
>
> So, what should have been configured is something like
>
> $ echo '8:0 enable=1 rpct=95 rlat=10000 wpct=95 wlat=20000' > /sys/fs/cgroup/io.cost.qos
>
> which just says "target 10ms p(95) read latency and 20ms p(95) write
> latency" without putting any restrictions on vrate range.
>
> With that, I got the following on Micron_1100_MTFDDAV256TBN which is a
> pretty old 256GB SATA drive.
>
> Aggregated throughput:
> min max avg std_dev conf99%
> 266.73 275.71 271.38 4.05144 45.7635
> Interfered total throughput:
> min max avg std_dev
> 9.608 13.008 10.941 0.664938
>
> During the run, iocost-monitor.py looked like the following.
>
> sda RUN per=40ms cur_per=2074.351:v1008.844 busy= +0 vrate= 59.85% params=ssd_dfl(CQ)
> active weight hweight% inflt% del_ms usages%
> InterfererGroup0 * 100/ 100 22.94/ 20.00 0.00 0*000 023:023:023
> InterfererGroup1 * 100/ 100 22.94/ 20.00 0.00 0*000 023:023:023
> InterfererGroup2 * 100/ 100 22.94/ 20.00 0.00 0*000 025:023:021
> InterfererGroup3 * 100/ 100 22.94/ 20.00 0.00 0*000 023:023:023
> interfered * 36/ 100 8.26/ 20.00 0.42 0*000 003:004:004
>
> Note that interfered is reported to only use 3-4% of the disk capacity
> while configured to consume 20%. This is because with single
> concurrency 4k randread job, its ability to consume IO capacity is
> limited by the completion latency.
>
> 10ms is pretty generous (ie. more work-conserving) target for SSDs.
> Let's say we're willing to tighten it to trade off total work for
> tighter latency.
>
> $ echo '8:0 enable=1 rpct=95 rlat=2500 wpct=95 wlat=5000' > /sys/fs/cgroup/io.cost.qos
>
> Aggregated throughput:
> min max avg std_dev conf99%
> 147.06 172.18 154.608 11.783 133.096
> Interfered total throughput:
> min max avg std_dev
> 17.992 19.32 18.698 0.313105
>
> and the monitoring output
>
> sda RUN per=10ms cur_per=2927.152:v1556.138 busy= -2 vrate= 34.74% params=ssd_dfl(CQ)
> active weight hweight% inflt% del_ms usages%
> InterfererGroup0 * 100/ 100 20.00/ 20.00 386.11 0*000 070:020:020
> InterfererGroup1 * 100/ 100 20.00/ 20.00 386.11 0*000 070:020:020
> InterfererGroup2 * 100/ 100 20.00/ 20.00 386.11 0*000 070:020:020
> InterfererGroup3 * 100/ 100 20.00/ 20.00 0.00 0*000 020:020:020
> interfered * 100/ 100 20.00/ 20.00 1.21 0*000 010:014:017
>
> The followings happened.
>
> * The vrate is now hovering way lower. The device is now doing less
> total work to acheive tighter completion latencies.
>
> * The overall throughput dropped but interfered's utilization is now
> significantly higher along with its bandwidth from lower completion
> latencies.
>
> For reference:
>
> [Disabled]
>
> Aggregated throughput:
> min max avg std_dev conf99%
> 493.98 511.37 502.808 9.52773 107.621
> Interfered total throughput:
> min max avg std_dev
> 0.056 0.304 0.107 0.0691052
>
> [Enabled, no QoS config]
>
> Aggregated throughput:
> min max avg std_dev conf99%
> 429.07 449.59 437.597 8.64952 97.7015
> Interfered total throughput:
> min max avg std_dev
> 0.456 3.12 1.08 0.774318
>
> Thanks.
>
> --
> tejun
Powered by blists - more mailing lists