netdev - ADQ - comparison to aRFS, clarifications on NAPI ID, binding with busy-polling

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <e13faf29-5db3-91a2-4a95-c2cd8c2d15fe@mellanox.com>
Date:   Wed, 17 Jun 2020 16:15:23 +0300
From:   Maxim Mikityanskiy <maximmi@...lanox.com>
To:     Amritha Nambiar <amritha.nambiar@...el.com>,
        Kiran Patil <kiran.patil@...el.com>,
        Sridhar Samudrala <sridhar.samudrala@...el.com>
Cc:     Alexander Duyck <alexander.h.duyck@...el.com>,
        Eric Dumazet <edumazet@...gle.com>,
        Tom Herbert <tom@...bertland.com>, netdev@...r.kernel.org
Subject: ADQ - comparison to aRFS, clarifications on NAPI ID, binding with
 busy-polling

Hi,

I discovered Intel ADQ feature [1] that allows to boost performance by 
picking dedicated queues for application traffic. We did some research, 
and I got some level of understanding how it works, but I have some 
questions, and I hope you could answer them.

1. SO_INCOMING_NAPI_ID usage. In my understanding, every connection has 
a key (sk_napi_id) that is unique to the NAPI where this connection is 
handled, and the application uses that key to choose a handler thread 
from the thread pool. If we have a one-to-one relationship between 
application threads and NAPI IDs of connections, each application thread 
will handle only traffic from a single NAPI. Is my understanding correct?

1.1. I wonder how the application thread gets scheduled on the same core 
that NAPI runs at. It currently only works with busy_poll, so when the 
application initiates busy polling (calls epoll), does the Linux 
scheduler move the thread to the right CPU? Do we have to have a strict 
one-to-one relationship between threads and NAPIs, or can one thread 
handle multiple NAPIs? When the data arrives, does the scheduler run the 
application thread on the same CPU that NAPI ran on?

1.2. I see that SO_INCOMING_NAPI_ID is tightly coupled with busy_poll. 
It is enabled only if CONFIG_NET_RX_BUSY_POLL is set. Is there a real 
reason why it can't be used without busy_poll? In other words, if we 
modify the kernel to drop this requirement, will the kernel still 
schedule the application thread on the same CPU as NAPI when busy_poll 
is not used?

2. Can you compare ADQ to aRFS+XPS? aRFS provides a way to steer traffic 
to the application's CPU in an automatic fashion, and xps_rxqs can be 
used to transmit from the corresponding queues. This setup doesn't need 
manual configuration of TCs and is not limited to 4 applications. The 
difference of ADQ is that (in my understanding) it moves the application 
to the RX CPU, while aRFS steers the traffic to the RX queue handled my 
the application's CPU. Is there any advantage of ADQ over aRFS, that I 
failed to find?

3. At [1], you mention that ADQ can be used to create separate RSS sets. 
  Could you elaborate about the API used? Does the tc mqprio 
configuration also affect RSS? Can it be turned on/off?

4. How is tc flower used in context of ADQ? Does the user need to 
reflect the configuration in both mqprio qdisc (for TX) and tc flower 
(for RX)? It looks like tc flower maps incoming traffic to TCs, but what 
is the mechanism of mapping TCs to RX queues?

I really hope you will be able to shed more light on this feature to 
increase my awareness on how to use it and to compare it with aRFS.

Thanks,
Max

[1]: 
https://netdevconf.info/0x14/session.html?talk-ADQ-for-system-level-network-io-performance-improvements