Date:   Tue, 31 May 2022 12:26:39 +0530
From:   Vinod Koul <vkoul@...nel.org>
To:     Arnd Bergmann <arnd@...db.de>
Cc:     Linus Walleij <linus.walleij@...aro.org>,
        Allen Pais <apais@...ux.microsoft.com>,
        Vincent Guittot <vincent.guittot@...aro.org>,
        olivier.dautricourt@...lia.com, Stefan Roese <sr@...x.de>,
        Kees Cook <keescook@...omium.org>,
        linux-hardening@...r.kernel.org,
        Ludovic Desroches <ludovic.desroches@...rochip.com>,
        Tudor Ambarus <tudor.ambarus@...rochip.com>,
        Florian Fainelli <f.fainelli@...il.com>,
        Ray Jui <rjui@...adcom.com>,
        Scott Branden <sbranden@...adcom.com>,
        bcm-kernel-feedback-list <bcm-kernel-feedback-list@...adcom.com>,
        Nicolas Saenz Julienne <nsaenz@...nel.org>,
        Paul Cercueil <paul@...pouillou.net>,
        Eugeniy.Paltsev@...opsys.com,
        Gustavo Pimentel <gustavo.pimentel@...opsys.com>,
        Viresh Kumar <vireshk@...nel.org>,
        Andy Shevchenko <andriy.shevchenko@...ux.intel.com>,
        Leo Li <leoyang.li@....com>, zw@...kernel.org,
        Zhou Wang <wangzhou1@...ilicon.com>,
        Shawn Guo <shawnguo@...nel.org>,
        Sascha Hauer <s.hauer@...gutronix.de>,
        Sean Wang <sean.wang@...iatek.com>,
        Matthias Brugger <matthias.bgg@...il.com>,
        Andreas Färber <afaerber@...e.de>,
        Manivannan Sadhasivam <mani@...nel.org>,
        Logan Gunthorpe <logang@...tatee.com>,
        Sanjay R Mehta <sanju.mehta@....com>,
        Daniel Mack <daniel@...que.org>,
        Haojian Zhuang <haojian.zhuang@...il.com>,
        Robert Jarzmik <robert.jarzmik@...e.fr>,
        Andy Gross <agross@...nel.org>,
        Bjorn Andersson <bjorn.andersson@...aro.org>,
        Krzysztof Kozlowski <krzysztof.kozlowski@...aro.org>,
        green.wan@...ive.com, Orson Zhai <orsonzhai@...il.com>,
        Baolin Wang <baolin.wang7@...il.com>,
        Lyra Zhang <zhang.lyra@...il.com>,
        Patrice CHOTARD <patrice.chotard@...s.st.com>,
        Chen-Yu Tsai <wens@...e.org>,
        Jernej Škrabec <jernej.skrabec@...il.com>,
        Samuel Holland <samuel@...lland.org>,
        dmaengine@...r.kernel.org,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: [RFC 1/1] drivers/dma/*: replace tasklets with workqueue

On 27-05-22, 12:59, Arnd Bergmann wrote:
> On Fri, May 27, 2022 at 10:06 AM Vinod Koul <vkoul@...nel.org> wrote:
> > On 25-05-22, 13:03, Arnd Bergmann wrote:
> > > What might work better in the case of the dmaengine API would
> > > be an approach like:
> > >
> > > 1. add helper functions to call the callback functions from a
> > >     tasklet locally defined in drivers/dma/dmaengine.c to allow
> > >     deferring it from hardirq context
> > >
> > > 2. Change all  tasklets that are not part of the callback
> > >     mechanism to work queue functions, I only see
> > >     xilinx_dpdma_chan_err_task in the patch, but there
> > >     may be more
> > >
> > > 3. change all drivers to move their custom tasklets back into
> > >     hardirq context and instead call the new helper for deferring
> > >     the callback.
> > >
> > > 4. Extend the dmaengine callback API to let slave drivers
> > >     pick hardirq, tasklet or task context for the callback.
> > >     task context can mean either a workqueue, or a threaded
> > >     IRQ here, with the default remaining the tasklet version.
> >
> > That does sound a good idea, but I dont know who will use the workqueue
> > or a threaded context here, it might be that most would default to
> > hardirq or tasklet context for obvious reasons...
> 
> If the idea is to remove tasklets from the kernel for good, then the
> choice is only between workqueue and hardirq at this point. The
> workqueue version is the one that would make sense for any driver
> that just defers execution from the callback down into task context.
> If that gets called in task context already, the driver can be simpler.
> 
> I took a brief look at the roughly 150 slave drivers, and it does
> seem like very few of them actually want task context:
> 
> * Over Half the drivers just do a complete(), which could
>   probably be pulled into the dmaengine layer and done from
>   hardirq, avoiding the callback entirely
> 
> * A lot of the remaining drivers have interrupts disabled for
>   the entire callback, which means they might as well use
>   hardirqs, regardless of what they want
> 
> * drivers/crypto/* and drivers/mmc/* tend to call another tasklet
>   to do the real work.
> 
> * drivers/ata/sata_dwc_460ex.c and drivers/ntb/ntb_transport.c
>    probably want task context
> 
> * Some drivers like sound/soc/sh/siu_pcm.c start a new DMA
>   from the callback. Is that allowed from hardirq?
> 
> If we do the first three steps above, and then add a 'struct
> completion' pointer to dma_async_tx_descriptor as an alternative
> to the callback, that would already reduce the number of drivers
> that end up in a tasklet significantly and should be completely
> safe.

That is a good idea, lot of drivers are waiting for completion which can
be signalled from hardirq, this would also reduce the hops we have and
help improve latency a bit. On the downside, some controllers provide
error information, which would need to be dealt with.

I will prototype this on Qcom boards I have...

> 
> Unfortunately we can't just move the rest into hardirq
> context because that breaks anything using spin_lock_bh
> to protect against concurrent execution of the tasklet.
> 
> A possible alternative might be to then replace the global
> dmaengine tasklet with a custom softirq. Obviously those
> are not so hot either,  but dmaengine could be considered
> special enough to fit in the same category as net_rx/tx
> and block with their global softirqs.

Yes that would be a very reasonable mechanism, thanks for the
suggestions.

-- 
~Vinod

Powered by blists - more mailing lists