Message-ID: <CAPLW+4=NcjOFRd14ecYd8sMsiJXH9c+ZXse7BVMCWe5ZbMmKMQ@mail.gmail.com>
Date: Mon, 26 Aug 2024 20:28:47 -0500
From: Sam Protsenko <semen.protsenko@...aro.org>
To: Arnd Bergmann <arnd@...db.de>
Cc: Jaehoon Chung <jh80.chung@...sung.com>, Ulf Hansson <ulf.hansson@...aro.org>, 
	Christoph Hellwig <hch@....de>, Chris Ball <cjb@...top.org>, Will Newton <will.newton@...il.com>, 
	Matt Fleming <matt@...sole-pimps.org>, Christian Brauner <brauner@...nel.org>, 
	Jens Axboe <axboe@...nel.dk>, Sumit Semwal <sumit.semwal@...aro.org>, 
	Dan Carpenter <dan.carpenter@...aro.org>, Anders Roxell <anders.roxell@...aro.org>, 
	Naresh Kamboju <naresh.kamboju@...aro.org>, 
	"linux-mmc @ vger . kernel . org" <linux-mmc@...r.kernel.org>, linux-block <linux-block@...r.kernel.org>, 
	linux-kernel@...r.kernel.org
Subject: Re: [PATCH] mmc: dw_mmc: Fix IDMAC operation with pages bigger than 4K

On Thu, Mar 7, 2024 at 1:52 AM Arnd Bergmann <arnd@...db.de> wrote:

[snip]

>
> > This change is not only fixing the boot with 16K/64K pages, but also
> > leads to a better MMC performance. The linear write performance was
> > tested on E850-96 board (eMMC only), before commit [1] (where it's
> > possible to boot with 16K/64K pages without this fix, to be able to do
> > a comparison). It was tested with this command:
> >
> >     # dd if=/dev/zero of=somefile bs=1M count=500 oflag=sync
> >
> > Test results are as follows:
> >
> >   - 4K pages,  .max_seg_size = 4 KiB:                   94.2 MB/s
> >   - 4K pages,  .max_seg_size = .max_req_size = 512 KiB: 96.9 MB/s
> >   - 16K pages, .max_seg_size = 4 KiB:                   126 MB/s
> >   - 16K pages, .max_seg_size = .max_req_size = 2 MiB:   128 MB/s
> >   - 64K pages, .max_seg_size = 4 KiB:                   138 MB/s
> >   - 64K pages, .max_seg_size = .max_req_size = 8 MiB:   138 MB/s
>
> Thanks for sharing these results. From what I can see here, the
> performance changes significantly with the page size, but barely
> with the max_seg_size, so this does not have the effect I was
> hoping for. On a more positive note this likely means that we
> don't have to urgently backport your fix.
>
> This could mean that either there is not much coalescing across
> pages after all, or that the bottleneck is somewhere else.
>
> > diff --git a/drivers/mmc/host/dw_mmc.c b/drivers/mmc/host/dw_mmc.c
> > index 8e2d676b9239..cccd5633ff40 100644
> > --- a/drivers/mmc/host/dw_mmc.c
> > +++ b/drivers/mmc/host/dw_mmc.c
> > @@ -2951,8 +2951,8 @@ static int dw_mci_init_slot(struct dw_mci *host)
> >       if (host->use_dma == TRANS_MODE_IDMAC) {
> >               mmc->max_segs = host->ring_size;
> >               mmc->max_blk_size = 65535;
> > -             mmc->max_seg_size = 0x1000;
> > -             mmc->max_req_size = mmc->max_seg_size * host->ring_size;
> > +             mmc->max_req_size = DW_MCI_DESC_DATA_LENGTH * host->ring_size;
> > +             mmc->max_seg_size = mmc->max_req_size;
>
> The change looks good to me.
>
> I see that the host->ring_size depends on PAGE_SIZE as well:
>
> #define DESC_RING_BUF_SZ        PAGE_SIZE
> host->ring_size = DESC_RING_BUF_SZ / sizeof(struct idmac_desc_64addr);
> host->sg_cpu = dmam_alloc_coherent(host->dev,
>                DESC_RING_BUF_SZ, &host->sg_dma, GFP_KERNEL);
>
> I don't see any reason for the ring buffer size to be tied to
> PAGE_SIZE at all, it was probably picked as a reasonable
> default in the initial driver but isn't necessarily ideal.
>
> From what I can see, the number of 4KB elements in the
> ring can be as small as 128 (4KB pages, 64-bit addresses)
> or as big as 4096 (64KB pages, 32-bit addresses), which is
> quite a difference. If you are still motivated to drill
> down into this, could you try changing DESC_RING_BUF_SZ
> to a fixed size of either 4KB or 64KB and test again
> with the opposite page size, to see if that changes the
> throughput?
>

Sorry for the huge delay. Just ran the tests:

- 4K pages, DESC_RING_BUF_SZ = 4K: 97 MB/s
- 4K pages, DESC_RING_BUF_SZ = 16K: 98 MB/s
- 4K pages, DESC_RING_BUF_SZ = 64K: 97 MB/s
- 16K pages, DESC_RING_BUF_SZ = 4K: 123 MB/s
- 16K pages, DESC_RING_BUF_SZ = 16K: 125 MB/s
- 16K pages, DESC_RING_BUF_SZ = 64K: 124 MB/s
- 64K pages, DESC_RING_BUF_SZ = 4K: 137 MB/s
- 64K pages, DESC_RING_BUF_SZ = 16K: 135 MB/s
- 64K pages, DESC_RING_BUF_SZE = 64K: 138 MB/s

As you can see, changing the DESC_RING_BUF_SZ value doesn't change MMC
throughput much; the fluctuations are just a measurement/statistical
error. Not that it matters much, but my measurement method was running
the same dd command I mentioned earlier:

    # dd if=/dev/zero of=somefile bs=1M count=500 oflag=sync

For each combination, I ran it 10 times, with different wait time
between runs (as it affects the throughput), then taking top 10%
percentile value -- it just felt right. The dispersion wasn't too big
either. I used the same method in my previous mail as well, so it's
safe to compare these values with those.

Not sure if my test procedure and results cover everything you wanted
to see, so please let me know if you want me to run more tests (e.g.
by changing dd params, or running multiple dd commands with shorter
wait in between, etc).

> If a larger ring buffer gives us significantly better
> throughput, we may want to always use a higher number
> independent of page size. On the other hand, if the
> 64KB number (the 138MB/s) does not change with a smaller
> ring, we may as well reduce that in order to limit the
> maximum latency that is caused by a single I/O operation.
>

>From what you said, it looks like it may make a sense to reduce
DESC_RING_BUF_SZ down to 4 KiB? If so, I'd suggest we do that in a
separate patch, as this one actually fixes kernel panic when 16k/64k
pages are enabled. Please let me know what you think.

Thanks!

Powered by blists - more mailing lists