Message-ID: <CA+mtBx9inCEq7PS3h02tSThsMLBVv5OUrusDkMg7d-OOvmqdiQ@mail.gmail.com>
Date:	Fri, 20 Feb 2015 14:30:24 -0800
From:	Tom Herbert <therbert@...gle.com>
To:	Jonathon Reinhart <jonathon.reinhart@...il.com>
Cc:	Sunil Kovvuri <sunil.kovvuri@...il.com>,
	Linux Netdev List <netdev@...r.kernel.org>,
	"David S. Miller" <davem@...emloft.net>
Subject: Re: Setting RPS affinities from network driver

On Thu, Feb 19, 2015 at 6:07 PM, Jonathon Reinhart
<jonathon.reinhart@...il.com> wrote:
>
> I don't know if this is a good idea. It seems like allowing the driver to
> set this default configuration opens the door to a whole slew of driver-
> specific config customizations.
>
> As as user, I wouldn't expect one driver to have a different default
> value of rps_cpus than another driver. Furthermore, I could imagine a
> case where a user expects rps_cpus to default to zero for all of his
> NICs, and has other CPU affinity settings applied (think realtime).
>
> In my opinion, this should be left to userspace. It's not that hard to
> add it to your init scripts.
>
>
In the old days I might have agreed with that-- kernel implements
mechanism and policy is handled in user space. However, in this brave
new world of hardware offloads for more and more kernel networking,
I'm not so sure about this any more. It's starting to look like we may
want kernel to do more dynamic resource management under some
described policies.

In davem's keynote at netdev he mentioned that hardware offload should
be transparent to things like ip route, so one implication is that at
some point the kernel may need to decide which routes are the best to
offload per a policy. Also at netdev, Jesse Brandeburg mentioned that
Windows had a capabilities of spinning up RSS queues to handle
increased load, and lamented that we couldn't do this easily in Linux.
Willem deBruijn made RPS resilient under DOS with RPS flow limit as a
step in that direction. We have spent countless hours tuning RPS and
interrupt settings per platform, per NIC, and per some major
applications-- but the resultant init scripts that result are very
convoluted and static. It is an interesting idea if we could just tell
kernel to take it's best guess and be adaptive. I suppose something
like irqbalance is the alternative to do this sort of stuff in user
space, but I don't know if that is well deployed or sufficiently
reactive under DOS attack.

Tom

> On Wed, Feb 18, 2015 at 10:47 AM, Sunil Kovvuri <sunil.kovvuri@...il.com> wrote:
> > Thanks Tom,
> >
> > Will try to implement something like a library function which can be used
> > by any driver.
> >
> > Regards,
> > Sunil.
> >
> > On Wed, Feb 18, 2015 at 9:14 PM, Tom Herbert <therbert@...gle.com> wrote:
> >> On Tue, Feb 17, 2015 at 10:16 PM, Sunil Kovvuri <sunil.kovvuri@...il.com> wrote:
> >>> Hi,
> >>>
> >>> I am writing a network driver for a multicore SOC with a on-board
> >>> network interface and would like to set RPS affinities from driver by
> >>> default. So that the network performance is good with whatever driver
> >>> supplied. Ofcourse these can be further adjusted from userspace
> >>> anyways.
> >>>
> >>> Is there a way to set RPS settings from driver itself ?
> >>> The only issue i am seeing is while setting 'rps_needed' key.
> >>>
> >> There is nothing to prevent this, but it would be really cool to make
> >> this into a library function that drivers can call to initialize RPS
> >> in some sane way (you might want to look at cpu_rmap).
> >>
> >>> static_key_slow_inc(&rps_needed);
> >>>
> >>> Currently this is not exported, so having issues while compiling
> >>> driver as module.
> >>>
> >>> If i export this symbol, will it be acceptable ?
> >>>
> >> Seems okay to me.
> >>
> >> Thanks,
> >> Tom
> >>
> >>>
> >>> Thanks,
> >>> Sunil.
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe netdev" in
> >>> the body of a message to majordomo@...r.kernel.org
> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@...r.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Computers are incredibly fast, accurate and stupid. Human beings are
> incredibly slow, inaccurate and brilliant. Together they are powerful
> beyond imagination.
>                                     A. Einstein
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists