[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <f9cfcffb-203b-4ed1-82ba-14fed2252c7e@intel.com>
Date: Fri, 1 Nov 2024 16:01:16 -0500
From: "Samudrala, Sridhar" <sridhar.samudrala@...el.com>
To: Joe Damato <jdamato@...tly.com>, <netdev@...r.kernel.org>,
<pabeni@...hat.com>, <namangulati@...gle.com>, <edumazet@...gle.com>,
<amritha.nambiar@...el.com>, <sdf@...ichev.me>, <peter@...eblog.net>,
<m2shafiei@...terloo.ca>, <bjorn@...osinc.com>, <hch@...radead.org>,
<willy@...radead.org>, <willemdebruijn.kernel@...il.com>,
<skhawaja@...gle.com>, <kuba@...nel.org>, Martin Karsten
<mkarsten@...terloo.ca>, Bagas Sanjaya <bagasdotme@...il.com>, "David S.
Miller" <davem@...emloft.net>, Simon Horman <horms@...nel.org>, "Jonathan
Corbet" <corbet@....net>, "open list:DOCUMENTATION"
<linux-doc@...r.kernel.org>, open list <linux-kernel@...r.kernel.org>, "open
list:BPF [MISC] :Keyword:(?:b|_)bpf(?:b|_)" <bpf@...r.kernel.org>
Subject: Re: [PATCH net-next v3 7/7] docs: networking: Describe irq suspension
On 10/31/2024 11:39 PM, Joe Damato wrote:
> On Thu, Oct 31, 2024 at 10:47:05PM -0500, Samudrala, Sridhar wrote:
>>
>>
>> On 10/31/2024 7:48 PM, Joe Damato wrote:
>>> Describe irq suspension, the epoll ioctls, and the tradeoffs of using
>>> different gro_flush_timeout values.
>>>
>>> Signed-off-by: Joe Damato <jdamato@...tly.com>
>>> Co-developed-by: Martin Karsten <mkarsten@...terloo.ca>
>>> Signed-off-by: Martin Karsten <mkarsten@...terloo.ca>
>>> Acked-by: Stanislav Fomichev <sdf@...ichev.me>
>>> Reviewed-by: Bagas Sanjaya <bagasdotme@...il.com>
>>> ---
>> <snip>
>>
>>
>>> +
>>> +IRQ suspension
>>> +--------------
>>> +
>>> +IRQ suspension is a mechanism wherein device IRQs are masked while epoll
>>> +triggers NAPI packet processing.
>>> +
>>> +While application calls to epoll_wait successfully retrieve events, the kernel will
>>> +defer the IRQ suspension timer. If the kernel does not retrieve any events
>>> +while busy polling (for example, because network traffic levels subsided), IRQ
>>> +suspension is disabled and the IRQ mitigation strategies described above are
>>> +engaged.
>>> +
>>> +This allows users to balance CPU consumption with network processing
>>> +efficiency.
>>> +
>>> +To use this mechanism:
>>> +
>>> + 1. The per-NAPI config parameter ``irq_suspend_timeout`` should be set to the
>>> + maximum time (in nanoseconds) the application can have its IRQs
>>> + suspended. This is done using netlink, as described above. This timeout
>>> + serves as a safety mechanism to restart IRQ driver interrupt processing if
>>> + the application has stalled. This value should be chosen so that it covers
>>> + the amount of time the user application needs to process data from its
>>> + call to epoll_wait, noting that applications can control how much data
>>> + they retrieve by setting ``max_events`` when calling epoll_wait.
>>> +
>>> + 2. The sysfs parameter or per-NAPI config parameters ``gro_flush_timeout``
>>> + and ``napi_defer_hard_irqs`` can be set to low values. They will be used
>>> + to defer IRQs after busy poll has found no data.
>>
>> Is it required to set gro_flush_timeout and napi_defer_hard_irqs when
>> irq_suspend_timeout is set? Doesn't it override any smaller
>> gro_flush_timeout value?
>
> It is not required to use gro_flush_timeout or napi_defer_hard_irqs,
> but if they are set they will take over when epoll finds no events.
> Their usage is recommended. See the Usage section of the cover
> letter for details.
>
> While gro_flush_timeout and napi_defer_hard_irqs are not strictly
> required, it is difficult for the polling-based packet delivery loop
> to gain control over packet delivery.
>
> Please see a previous email about this from the RFC for more
> details:
>
> https://lore.kernel.org/netdev/2bb121dd-3dcd-4142-ab87-02ccf4afd469@uwaterloo.ca/
OK. Thanks for the clarification.
>
> In the cover letter, you can note the difference in performance when
> gro_flush_timeout is set to different values. Note the explanation
> of suspendX; each suspend case is testing a different
> gro_flush_timeout.
May be you can also include a test scenario in your perf results where
gro_flush_timeout and napi_defer_hard_irqs are not set to show that a
non-zero value of gro_flush_timeout and napi_defer_hard_irqs is
recommended when using irq_suspend_timeout.
>
> Let us know if you have any other questions; both Martin and I are
> happy to help or further explain anything that is not clear.
>
Powered by blists - more mailing lists