Message-ID: <b8e7a0ba-af07-08a8-b987-f82b17f8c69d@intel.com>
Date:   Wed, 22 Mar 2023 13:05:45 -0700
From:   Jacob Keller <jacob.e.keller@...el.com>
To:     Jakub Kicinski <kuba@...nel.org>,
        Piotr Raczynski <piotr.raczynski@...el.com>
CC:     <intel-wired-lan@...ts.osuosl.org>, <netdev@...r.kernel.org>,
        <michal.swiatkowski@...el.com>, <shiraz.saleem@...el.com>,
        <sridhar.samudrala@...el.com>, <jesse.brandeburg@...el.com>,
        <aleksander.lobakin@...el.com>, <lukasz.czapnik@...el.com>
Subject: Re: [PATCH net-next v2 0/8] ice: support dynamic interrupt allocation



On 3/22/2023 12:37 PM, Jakub Kicinski wrote:
> On Wed, 22 Mar 2023 17:25:22 +0100 Piotr Raczynski wrote:
>> This patchset reimplements MSIX interrupt allocation logic to allow dynamic
>> interrupt allocation after MSIX has been initially enabled. This allows
>> current and future features to allocate and free interrupts as needed and
>> will help to drastically decrease number of initially preallocated
>> interrupts (even down to the API hard limit of 1). Although this patchset
>> does not change behavior in terms of actual number of allocated interrupts
>> during probe, it will be subject to change.
> 
> Have you seen the mlx5 patch set? I haven't read in detail but seems
> like you're working on the same stuff so could be worth cross-checking
> each other's work:
> 
> https://lore.kernel.org/all/20230320175144.153187-1-saeed@kernel.org/

Thanks for the pointer. I read through that series just now, and it
looks good.

We are doing similar work, though a big difference is that MLX has
converted *all* allocations to be dynamic. This, I think, works for them
because they already have a pool allocation scheme and basically treated
the available vectors as a resource across the function. We didn't
really do that before (we reserved total based on feature and tried to
ensure we don't starve some features), and so converting everything to
dynamic in one-go wasn't done here. Instead, we open the way to allow
dynamic, with a plan to look at refactoring and removing the initial
allocations at a later stage.

In addition the MLX code already has good data structure for tracking
vectors, where as we had the mess that was our res_tracker which was
poorly implemented.

Overall I think the end goal is going to be similar: use dynamic
allocation when possible.

For ice, we need to complete this work, then follow up with some work to
make handling of vector allocation failure better in the case of things
such as vectors for default PF queues. The current code basically
determines the number of queues we create based on the number of vectors
we got. That won't work with dynamic, and we need to instead pick number
of queues, and be able to handle vector exhaustion if we happen to have
limited number of vectors available. We also need to make sure other
uses of vectors can handle failure appropriately, and deal with the
iRDMA<->ice interfaces which currently assume static allocation as well.

My long term goal would be that the driver only allocates one vector at
load which is for the control vector used for miscellaneous interrupts,
and everything else is allocated and released dynamically based on
feature use. But we have a bit of a ways to get there due to the above
limitations in current design. We need to make sure every path has a
suitable path to report failure on vector exhaustion, and that every
consumer has available knobs or parameters to allow the system to be
reconfigured to prevent exhaustion.

Powered by blists - more mailing lists