netdev - Re: [net-next 10/14] net/mlx5: Let user configure io_eq

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <d29e330c2265bde648b3a7e2f5e24a275ac4f306.camel@nvidia.com>
Date:   Tue, 26 Oct 2021 18:01:39 +0000
From:   Saeed Mahameed <saeedm@...dia.com>
To:     "kuba@...nel.org" <kuba@...nel.org>
CC:     Moshe Shemesh <moshe@...dia.com>, Shay Drory <shayd@...dia.com>,
        Parav Pandit <parav@...dia.com>, Jiri Pirko <jiri@...dia.com>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: [net-next 10/14] net/mlx5: Let user configure io_eq_size param

On Tue, 2021-10-26 at 10:16 -0700, Jakub Kicinski wrote:
> On Tue, 26 Oct 2021 15:54:28 +0000 Saeed Mahameed wrote:
> > On Tue, 2021-10-26 at 08:05 -0700, Jakub Kicinski wrote:
> > > On Mon, 25 Oct 2021 13:54:27 -0700 Saeed Mahameed wrote:  
> > > > From: Shay Drory <shayd@...dia.com>
> > > > 
> > > > Currently, each I/O EQ is taking 128KB of memory. This size
> > > > is not needed in all use cases, and is critical with large
> > > > scale.
> > > > Hence, allow user to configure the size of I/O EQs.
> > > > 
> > > > For example, to reduce I/O EQ size to 64, execute:
> > > > $ devlink resource set pci/0000:00:0b.0 path /io_eq_size/ size
> > > > 64
> > > > $ devlink dev reload pci/0000:00:0b.0  
> > > 
> > > This sort of config is needed by more drivers,
> > > we need a standard way of configuring this.
> > 
> > We had a debate internally about the same thing, Jiri and I thought
> > that EQ might be a ConnectX only thing (maybe some other vendors
> > have
> > it) but it is not really popular
> 
> I thought it's a RDMA thing. At least according to grep there's 
> a handful of non-MLX drivers which have eqs. Are these not actual
> event queues? (huawei/hinic, ibm/ehea, microsoft/mana, qlogic/qed)
> 
> > we thought, until other vendors start contributing or asking for 
> > the same thing, maybe then we can standardize.
> 
> Yeah, like the standardization part ever happens :/ 
> 
> Look at the EQE/CQE interrupt generation thing. New vendor comes in
> and
> copies best known practice (which is some driver-specific garbage,
> ethtool priv-flags in that case). The maintainer (me) has to be the
> policeman remember all those knobs with prior art and push back. Most

Well, i can't even count the patches shot down internally because of
non standard APIs. The driver maintainer (me) has to also be a
policeman. As long as we are in sync I think this can scale, I am sure
other vendors maintainers are filtering as many patches as I do.

> of the time the vendor decides to just keep the knob out of tree at
> this point, kudos to Hauwei for not giving up. New vendor implements
> the API, none of the existing vendors provide reviews or feedback.
> Then none of the existing vendors implements the now-standard API.
> Someone working for a large customer (me, again) has to go and ask 
> for the API to be implemented. Which takes months even tho the
> patches
> should be trivial.
> 
> If the initial patches adding the cqe/eqe interrupt modes to priv-
> flags
> were nacked and the standard API created we'd all have saved much
> time.
> 

Sometimes it is hard to decide what is the best for the user that fits
all vendors, when you are the first to come up with a new concept,
EQE/CQE thing is a relatively new mechanism, took other vendors a while
to catch up, who would've known such mechanism would become popular ?

but I do agree with you  many apis can be standardize or at least
refined with better policing, sorry, but the only way to do this is to
have as many vendors as possible looking at each possible API patch.

Many times we are pioneering in latest features, especially for
smartnics scalability, that involves resource fine tuning and fine-
grained user controls. luckily for the EQ thing we can generalize.

> > > Sorry, I didn't have the time to look thru your patches
> > > yesterday, I'm sending a revert for all your new devlink
> > > params.  
> > 
> > Sure, we will submit a RFC to give other vendors a chance to
> > comment,
> > it will be basically the same patch (devlink resource) while making
> > the
> > parameters vendor generic.
> 
> IDK if resource is a right fit (as mentioned to Parav in the
> discussion
> on the revert).

will switch the discussion to that thread.