Message-ID: <e05b416d-a298-cf71-0c04-a1fe6b993bf7@huawei.com>
Date:   Tue, 2 May 2023 14:15:49 +0200
From:   Petr Tesarik <petr.tesarik.ext@...wei.com>
To:     Mike Lothian <mike@...eburn.co.uk>,
        Petr Tesařík <petr@...arici.cz>
CC:     Jonathan Corbet <corbet@....net>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        "Rafael J. Wysocki" <rafael@...nel.org>,
        Maarten Lankhorst <maarten.lankhorst@...ux.intel.com>,
        Maxime Ripard <mripard@...nel.org>,
        Thomas Zimmermann <tzimmermann@...e.de>,
        David Airlie <airlied@...il.com>,
        Daniel Vetter <daniel@...ll.ch>,
        Christoph Hellwig <hch@....de>,
        Marek Szyprowski <m.szyprowski@...sung.com>,
        Robin Murphy <robin.murphy@....com>,
        Borislav Petkov <bp@...e.de>,
        "Paul E. McKenney" <paulmck@...nel.org>,
        Catalin Marinas <catalin.marinas@....com>,
        Will Deacon <will@...nel.org>,
        Zhen Lei <thunder.leizhen@...wei.com>,
        Randy Dunlap <rdunlap@...radead.org>,
        Damien Le Moal <damien.lemoal@...nsource.wdc.com>,
        Kim Phillips <kim.phillips@....com>,
        "Steven Rostedt (Google)" <rostedt@...dmis.org>,
        Muchun Song <muchun.song@...ux.dev>,
        Ondrej Zary <linux@...y.sk>,
        "Jason A. Donenfeld" <Jason@...c4.com>,
        Petr Tesarik <petr.tesarik.ext@...wei.com>,
        Hans de Goede <hdegoede@...hat.com>,
        Dan Williams <dan.j.williams@...el.com>,
        Andy Shevchenko <andriy.shevchenko@...ux.intel.com>,
        Kees Cook <keescook@...omium.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Won Chung <wonchung@...gle.com>,
        "open list:DOCUMENTATION" <linux-doc@...r.kernel.org>,
        open list <linux-kernel@...r.kernel.org>,
        "open list:DRM DRIVERS" <dri-devel@...ts.freedesktop.org>,
        "open list:DMA MAPPING HELPERS" <iommu@...ts.linux.dev>,
        Kefeng Wang <wangkefeng.wang@...wei.com>,
        Roberto Sassu <roberto.sassu@...wei.com>
Subject: Re: [PATCH v2 0/7] Allow dynamic allocation of software IO TLB bounce
 buffers

Hi Mike,

On 5/1/2023 12:29 PM, Mike Lothian wrote:
> Hi
> 
> I've not had any issues since using this, but I imagine most people
> won't know how to set swiotlb=dynamic if they start seeing this (when
> it lands)
> 
> Any clue as to why this broke last cycle?

IIRC the kernel messages indicate high swiotlb utilization. The most
likely cause is that either the wifi driver is using more DMA buffers at
once, or another device driver is now using swiotlb; it could also be a
driver for a device that was previously not supported.

If you're adventurous, you can use ftrace to capture the backtraces of
swiotlb_tbl_map_single. This would reveal the other driver. Debugging
increased DMA buffer usage by your wifi driver may be more difficult,
because that's a dynamic process, which may change dramatically if you
add some sort of tracing into the picture... As always, YMMV.

Petr T

> Mike
> 
> On Fri, 28 Apr 2023 at 10:07, Petr Tesařík <petr@...arici.cz> wrote:
>>
>> On Fri, 28 Apr 2023 09:53:38 +0100
>> Mike Lothian <mike@...eburn.co.uk> wrote:
>>
>>> On Wed, 19 Apr 2023 at 11:05, Petr Tesarik <petrtesarik@...weicloud.com> wrote:
>>>>
>>>> From: Petr Tesarik <petr.tesarik.ext@...wei.com>
>>>>
>>>> The goal of my work is to provide more flexibility in the sizing of
>>>> SWIOTLB.
>>>>
>>>> The software IO TLB was designed with these assumptions:
>>>>
>>>> 1. It would not be used much, especially on 64-bit systems.
>>>> 2. A small fixed memory area (64 MiB by default) is sufficient to
>>>>    handle the few cases which require a bounce buffer.
>>>> 3. 64 MiB is little enough that it has no impact on the rest of the
>>>>    system.
>>>>
>>>> First, if SEV is active, all DMA must be done through shared
>>>> unencrypted pages, and SWIOTLB is used to make this happen without
>>>> changing device drivers. The software IO TLB size is increased to
>>>> 6% of total memory in sev_setup_arch(), but that is more of an
>>>> approximation. The actual requirements may vary depending on the
>>>> amount of I/O and which drivers are used. These factors may not be
>>>> know at boot time, i.e. when SWIOTLB is allocated.
>>>>
>>>> Second, other colleagues have noticed that they can reliably get
>>>> rid of occasional OOM kills on an Arm embedded device by reducing
>>>> the SWIOTLB size. This can be achieved with a kernel parameter, but
>>>> determining the right value puts additional burden on pre-release
>>>> testing, which could be avoided if SWIOTLB is allocated small and
>>>> grows only when necessary.
>>>>
>>>> Changes from v1-devel-v7:
>>>> - Add comments to acquire/release barriers
>>>> - Fix whitespace issues reported by checkpatch.pl
>>>>
>>>> Changes from v1-devel-v6:
>>>> - Provide long description of functions
>>>> - Fix kernel-doc (Returns: to Return:)
>>>> - Rename __lookup_dyn_slot() to lookup_dyn_slot_locked()
>>>>
>>>> Changes from RFC:
>>>> - Track dynamic buffers per device instead of per swiotlb
>>>> - Use a linked list instead of a maple tree
>>>> - Move initialization of swiotlb fields of struct device to a
>>>>   helper function
>>>> - Rename __lookup_dyn_slot() to lookup_dyn_slot_locked()
>>>> - Introduce per-device flag if dynamic buffers are in use
>>>> - Add one more user of DMA_ATTR_MAY_SLEEP
>>>> - Add kernel-doc comments for new (and some old) code
>>>> - Properly escape '*' in dma-attributes.rst
>>>>
>>>> Petr Tesarik (7):
>>>>   swiotlb: Use a helper to initialize swiotlb fields in struct device
>>>>   swiotlb: Move code around in preparation for dynamic bounce buffers
>>>>   dma-mapping: introduce the DMA_ATTR_MAY_SLEEP attribute
>>>>   swiotlb: Dynamically allocated bounce buffers
>>>>   swiotlb: Add a boot option to enable dynamic bounce buffers
>>>>   drm: Use DMA_ATTR_MAY_SLEEP from process context
>>>>   swiotlb: per-device flag if there are dynamically allocated buffers
>>>>
>>>>  .../admin-guide/kernel-parameters.txt         |   6 +-
>>>>  Documentation/core-api/dma-attributes.rst     |  10 +
>>>>  drivers/base/core.c                           |   4 +-
>>>>  drivers/gpu/drm/drm_gem_shmem_helper.c        |   2 +-
>>>>  drivers/gpu/drm/drm_prime.c                   |   2 +-
>>>>  include/linux/device.h                        |  12 +
>>>>  include/linux/dma-mapping.h                   |   6 +
>>>>  include/linux/swiotlb.h                       |  54 ++-
>>>>  kernel/dma/swiotlb.c                          | 382 ++++++++++++++++--
>>>>  9 files changed, 443 insertions(+), 35 deletions(-)
>>>>
>>>> --
>>>> 2.25.1
>>>>
>>>
>>> Hi
>>>
>>> Is this a potential fix for
>>> https://bugzilla.kernel.org/show_bug.cgi?id=217310 where I'm manually
>>> setting bigger buffers to keep my wifi working?
>>
>> Yes. With these patches applied, your system should run just fine with
>> swiotlb=dynamic. However, keep in mind that this implementation adds a
>> bit of overhead. In short, it trades a bit of performance for not
>> having to figure out the optimal swiotlb size at boot time.
>>
>> Petr T

Powered by blists - more mailing lists