linux-kernel - Re: RE: very low IOPS due to "block: reduce kblockd_mod_delayed_work

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20211228134926.GA31268@xsang-OptiPlex-9020>
Date:   Tue, 28 Dec 2021 21:49:27 +0800
From:   Oliver Sang <oliver.sang@...el.com>
To:     Yin Fengwei <fengwei.yin@...el.com>
Cc:     Dexuan Cui <decui@...rosoft.com>, Jens Axboe <axboe@...nel.dk>,
        "Alex Xu (Hello71)" <alex_y_xu@...oo.ca>,
        "linux-block@...r.kernel.org" <linux-block@...r.kernel.org>,
        "ming.lei@...hat.com" <ming.lei@...hat.com>,
        "hch@....de" <hch@....de>, Long Li <longli@...rosoft.com>,
        "Michael Kelley (LINUX)" <mikelley@...rosoft.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "Li, Philip" <philip.li@...el.com>
Subject: Re: RE: very low IOPS due to "block: reduce
 kblockd_mod_delayed_work_on() CPU consumption"

Hi Jens, Dexuan,

On Tue, Dec 28, 2021 at 10:30:43AM +0800, Yin Fengwei wrote:
> Hi Jens, Dexuan,
> 
> On 12/20/2021 1:15 PM, Dexuan Cui wrote:
> >> From: Jens Axboe <axboe@...nel.dk>
> >> Sent: Sunday, December 19, 2021 7:28 AM
> >>> ...
> >>> Dexuan, can you test this for your test case too? I'm going to queue
> >>> up a revert for -rc6 just in case.
> >>
> >> This one should be better...
> >> ...
> >> Jens Axboe
> > 
> > Hi Jens, sorry -- unluckily I lost the test environment.. :-(
> > I pinged the user to set up the test environment again, but he's
> > on vacation till the beginning of January.
> 
> We hit this issue in our testing env also and will help to verify
> your fixing patch. Thanks.

as we reported in [1], we found cb2ac2912a cause big regressions in fxmark
tests.
"[block]  cb2ac2912a: fxmark.ssd_f2fs_dbench_client_4_bufferedio.works/sec -66.0% regression"

By applying the patch supplied by Jens Axboe in previous thread directly
upon cb2ac2912a, we confirmed the regression could be fixed:

=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/bufferedio/1SSD/f2fs/x86_64-rhel-8.3/ssd/debian-10.4-x86_64-20200603.cgz/lkp-snr-a1/dbench_client/fxmark/0x9c02000e

commit:
  edaa26334c ("iocost: Fix divide-by-zero on donation from low hweight cgroup")
  cb2ac2912a ("block: reduce kblockd_mod_delayed_work_on() CPU consumption")
  80f0eab183 (fix for very low IOPS due to "block: reduce kblockd_mod_delayed_work_on() CPU consumption") <-- patch by Jens Axboe

edaa26334c117a58 cb2ac2912a9ca7d3d26291c5119 80f0eab18356919a1c0568c6348
---------------- --------------------------- ---------------------------
         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \
    573.42           -66.0%     194.68            -0.6%     570.11        fxmark.ssd_f2fs_dbench_client_4_bufferedio.works/sec

actually there are other subtests which either have regressions also fixed
by new patch, or we didn't observe obvious performance changes.

edaa26334c117a58 cb2ac2912a9ca7d3d26291c5119 80f0eab18356919a1c0568c6348
---------------- --------------------------- ---------------------------
         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \
    574.25           -18.5%     468.11            -0.4%     571.83        fxmark.ssd_f2fs_dbench_client_18_bufferedio.works/sec
    495.79            +0.2%     496.65            -0.6%     493.04        fxmark.ssd_f2fs_dbench_client_1_bufferedio.works/sec
    536.81           -72.8%     145.99 ± 19%      -0.6%     533.78        fxmark.ssd_f2fs_dbench_client_2_bufferedio.works/sec
    544.81            +1.8%     554.73            -0.1%     544.26        fxmark.ssd_f2fs_dbench_client_36_bufferedio.works/sec
    513.26 ±  2%      +2.5%     526.05 ±  3%      -1.8%     504.04        fxmark.ssd_f2fs_dbench_client_54_bufferedio.works/sec
    513.10 ±  4%      +3.9%     533.07 ±  3%      -0.7%     509.73 ±  5%  fxmark.ssd_f2fs_dbench_client_72_bufferedio.works/sec


also attached detail comparison table edaa26334c-cb2ac2912a-80f0eab183
which contains futher details such like perf data.


[1] https://lore.kernel.org/lkml/20211219141852.GH14057@xsang-OptiPlex-9020/


> 
> 
> Regards
> Yin, Fengwei

View attachment "edaa26334c-cb2ac2912a-80f0eab183" of type "text/plain" (524431 bytes)