linux-kernel - Re: [PATCH 00/14] introduce the BFQ-v0 I/O scheduler as an extra scheduler

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CACRpkda0fmPdM2WZ_gtW=T7xvMfP_4ErtNZseExN2AggzZbcDQ@mail.gmail.com>
Date:   Fri, 28 Oct 2016 11:32:21 +0200
From:   Linus Walleij <linus.walleij@...aro.org>
To:     Jens Axboe <axboe@...nel.dk>
Cc:     Ulf Hansson <ulf.hansson@...aro.org>,
        Paolo Valente <paolo.valente@...aro.org>,
        Christoph Hellwig <hch@...radead.org>,
        Arnd Bergmann <arnd@...db.de>,
        Bart Van Assche <bart.vanassche@...disk.com>,
        Jan Kara <jack@...e.cz>, Tejun Heo <tj@...nel.org>,
        linux-block@...r.kernel.org,
        Linux-Kernal <linux-kernel@...r.kernel.org>,
        Mark Brown <broonie@...nel.org>,
        Hannes Reinecke <hare@...e.de>,
        Grant Likely <grant.likely@...retlab.ca>,
        James Bottomley <James.Bottomley@...senpartnership.com>,
        Bartlomiej Zolnierkiewicz <b.zolnierkie@...sung.com>
Subject: Re: [PATCH 00/14] introduce the BFQ-v0 I/O scheduler as an extra scheduler

On Fri, Oct 28, 2016 at 12:27 AM, Linus Walleij
<linus.walleij@...aro.org> wrote:
> On Thu, Oct 27, 2016 at 11:08 PM, Jens Axboe <axboe@...nel.dk> wrote:
>
>> blk-mq has evolved to support a variety of devices, there's nothing
>> special about mmc that can't work well within that framework.
>
> There is. Read mmc_queue_thread() in drivers/mmc/card/queue.c

So I'm not just complaining by the way, I'm trying to fix this. Also
Bartlomiej from Samsung has done some stabs at switching MMC/SD
to blk-mq. I just rebased my latest stab at a naïve switch to blk-mq
to v4.9-rc2 with these results.

The patch to enable MQ looks like this:
https://git.kernel.org/cgit/linux/kernel/git/linusw/linux-stericsson.git/commit/?h=mmc-mq&id=8f79b527e2e854071d8da019451da68d4753f71d

I run these tests directly after boot with cold caches. The results
are consistent: I ran the same commands 10 times in a row.


BEFORE switching to BLK-MQ (clean v4.9-rc2):

time dd if=/dev/mmcblk0 of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.0GB) copied, 47.781464 seconds, 21.4MB/s
real    0m 47.79s
user    0m 0.02s
sys     0m 9.35s

mount /dev/mmcblk0p1 /mnt/
cd /mnt/
time find . > /dev/null
real    0m 3.60s
user    0m 0.25s
sys     0m 1.58s

mount /dev/mmcblk0p1 /mnt/
iozone -az -i0 -i1 -i2 -s 20m -I -f /mnt/foo.test
(kBytes/second)
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
 20480       4     2112     2157     6052     6060     6025       40
 20480       8     4820     5074     9163     9121     9125       81
 20480      16     5755     5242    12317    12320    12280      165
 20480      32     6176     6261    14981    14987    14962      336
 20480      64     6547     5875    16826    16828    16810      692
 20480     128     6762     6828    17899    17896    17896     1408
 20480     256     6802     6871    16960    17513    18373     3048
 20480     512     7220     7252    18675    18746    18741     7228
 20480    1024     7222     7304    18436    17858    18246     7322
 20480    2048     7316     7398    18744    18751    18526     7419
 20480    4096     7520     7636    20774    20995    20703     7609
 20480    8192     7519     7704    21850    21489    21467     7663
 20480   16384     7395     7782    22399    22210    22215     7781


AFTER switching to BLK-MQ:

time dd if=/dev/mmcblk0 of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.0GB) copied, 60.551117 seconds, 16.9MB/s
real    1m 0.56s
user    0m 0.02s
sys     0m 9.81s

mount /dev/mmcblk0p1 /mnt/
cd /mnt/
time find . > /dev/null
real    0m 4.42s
user    0m 0.24s
sys     0m 1.81s

mount /dev/mmcblk0p1 /mnt/
iozone -az -i0 -i1 -i2 -s 20m -I -f /mnt/foo.test
(kBytes/second)
                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
 20480       4     2086     2201     6024     6036     6006       40
 20480       8     4812     5036     8014     9121     9090       82
 20480      16     5432     5633    12267     9776    12212      168
 20480      32     6180     6233    14870    14891    14852      340
 20480      64     6382     5454    16744    16771    16746      702
 20480     128     6761     6776    17816    17846    17836     1394
 20480     256     6828     6842    17789    17895    17094     3084
 20480     512     7158     7222    17957    17681    17698     7232
 20480    1024     7215     7274    18642    17679    18031     7300
 20480    2048     7229     7269    17943    18642    17732     7358
 20480    4096     7212     7360    18272    18157    18889     7371
 20480    8192     7008     7271    18632    18707    18225     7282
 20480   16384     6889     7211    18243    18429    18018     7246


A simple dd readtest of 1 GB is always consistently 10+
seconds slower with MQ. find in the rootfs is a second slower.
iozone results are consistently lower throughput or the same.

This is without using Bartlomiej's clever hack to pretend we have
2 elements in the HW queue though. His early tests indicate that
it doesn't help much: the performance regression we see is due to
lack of block scheduling.

I try to find a way forward with this, and also massage the MMC/SD
code to be more MQ friendly to begin with (like only pick requests
when we get a request notification and stop pulling NULL requests
off the queue) but it's really a messy piece of code.

Yours,
Linus Walleij