netdev - Re: [RFC] net;sched: Try to find idle cpu for RPS to handle packets

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANn89iJ1UnwLFv5+AwXgeb1BUhYg8UJYTJLtiipavJee+2SWxQ@mail.gmail.com>
Date:   Wed, 19 Sep 2018 07:55:11 -0700
From:   Eric Dumazet <edumazet@...gle.com>
To:     Kirill Tkhai <ktkhai@...tuozzo.com>
Cc:     Peter Zijlstra <peterz@...radead.org>,
        David Miller <davem@...emloft.net>,
        Daniel Borkmann <daniel@...earbox.net>, tom@...ntonium.net,
        netdev <netdev@...r.kernel.org>,
        LKML <linux-kernel@...r.kernel.org>
Subject: Re: [RFC] net;sched: Try to find idle cpu for RPS to handle packets

On Wed, Sep 19, 2018 at 5:29 AM Kirill Tkhai <ktkhai@...tuozzo.com> wrote:
>
> Many workloads have polling mode of work. The application
> checks for incomming packets from time to time, but it also
> has a work to do, when there is no packets. This RFC
> tries to develop an idea to queue RPS packets on idle
> CPU in the the L3 domain of the consumer, so backlog
> processing of the packets and the application can execute
> in parallel.
>
> We require this in case of network cards does not
> have enough RX queues to cover all online CPUs (this seems
> to be the most cards), and  get_rps_cpu() actually chooses
> remote cpu, and SMP interrupt is sent. Here we may try
> our best, and to find idle CPU nearly the consumer's CPU.
> Note, that in case of consumer works in poll mode and it
> does not waits for incomming packets, its CPU will be not
> idle, while CPU of a sleeping consumer may be idle. So,
> not polling consumers will still be able to have skb
> handled on its CPU.
>
> In case of network card has many queues, the device
> interrupts will come on consumer's CPU, and this patch
> won't try to find idle cpu for them.
>
> I've tried simple netperf test for this:
> netserver -p 1234
> netperf -L 127.0.0.1 -p 1234 -l 100
>
> Before:
>  87380  16384  16384    100.00   60323.56
>  87380  16384  16384    100.00   60388.46
>  87380  16384  16384    100.00   60217.68
>  87380  16384  16384    100.00   57995.41
>  87380  16384  16384    100.00   60659.00
>
> After:
>  87380  16384  16384    100.00   64569.09
>  87380  16384  16384    100.00   64569.25
>  87380  16384  16384    100.00   64691.63
>  87380  16384  16384    100.00   64930.14
>  87380  16384  16384    100.00   62670.15
>
> The difference between best runs is +7%,
> the worst runs differ +8%.
>
> What do you think about following somehow in this way?

Hi Kirill

In my experience, scheduler has a poor view of softirq processing
happening on various cpus.
A cpu spending 90% of its cycles processing IRQ might be considered 'idle'

So please run a real workload (it is _very_ uncommon anyone set up RPS
on lo interface !)

Like 400 or more concurrent netperf -t TCP_RR on a 10Gbit NIC.

Thanks.

PS: Idea of playing with L3 domains is interesting, I have personally
tried various strategies in the past but none of them
demonstrated a clear win.