| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <20230123221727.30423-1-nnac123@linux.ibm.com>
Date: Mon, 23 Jan 2023 16:17:27 -0600
From: Nick Child <nnac123@...ux.ibm.com>
To: netdev@...r.kernel.org
Cc: bjking1@...ux.ibm.com, haren@...ux.ibm.com, ricklind@...ibm.com,
Nick Child <nnac123@...ux.ibm.com>
Subject: [PATCH net-next] ibmvnic: Toggle between queue types in affinity mapping
Previously, ibmvnic IRQs were assigned to CPU numbers by assigning all
the IRQs for transmit queues then assigning all the IRQs for receive
queues. With multi-threaded processors, in a heavy RX or TX environment,
physical cores would either be overloaded or underutilized (due to the
IRQ assignment algorithm). This approach is sub-optimal because IRQs for
the same subprocess (RX or TX) would be bound to adjacent CPU numbers,
meaning they were more likely to be contending for the same core.
For example, in a system with 64 CPU's and 32 queues, the IRQs would
be bound to CPU in the following pattern:
IRQ type | CPU number
-----------------------
TX0 | 0-1
TX1 | 2-3
<etc>
RX0 | 32-33
RX1 | 34-35
<etc>
Observe that in SMT-8, the first 4 tx queues would be sharing the
same core.
A more optimal algorithm would balance the number RX and TX IRQ's across
the physical cores. Therefore, to increase performance, distribute RX and
TX IRQs across cores by alternating between assigning IRQs for RX and TX
queues to CPUs.
With a system with 64 CPUs and 32 queues, this results in the following
pattern (binding is done in reverse order for readable code):
IRQ type | CPU number
-----------------------
TX15 | 0-1
RX15 | 2-3
TX14 | 4-5
RX14 | 6-7
<etc>
Observe that in SMT-8, there is equal distribution of RX and TX IRQs
per core. In the above case, each core handles 2 TX and 2 RX IRQ's.
Signed-off-by: Nick Child <nnac123@...ux.ibm.com>
Reviewed-by: Haren Myneni <haren@...ux.ibm.com>
---
drivers/net/ethernet/ibm/ibmvnic.c | 23 ++++++++++++-----------
1 file changed, 12 insertions(+), 11 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index e19a6bb3f444..314a72cef592 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -254,6 +254,7 @@ static void ibmvnic_set_affinity(struct ibmvnic_adapter *adapter)
int num_txqs = adapter->num_active_tx_scrqs;
int total_queues, stride, stragglers, i;
unsigned int num_cpu, cpu;
+ bool is_rx_queue;
int rc = 0;
netdev_dbg(adapter->netdev, "%s: Setting irq affinity hints", __func__);
@@ -273,14 +274,22 @@ static void ibmvnic_set_affinity(struct ibmvnic_adapter *adapter)
/* next available cpu to assign irq to */
cpu = cpumask_next(-1, cpu_online_mask);
- for (i = 0; i < num_txqs; i++) {
- queue = txqs[i];
+ for (i = 0; i < total_queues; i++) {
+ is_rx_queue = false;
+ /* balance core load by alternating rx and tx assignments */
+ if ((i % 2 == 1 && num_rxqs > 0) || num_txqs == 0) {
+ queue = rxqs[--num_rxqs];
+ is_rx_queue = true;
+ } else {
+ queue = txqs[--num_txqs];
+ }
+
rc = ibmvnic_set_queue_affinity(queue, &cpu, &stragglers,
stride);
if (rc)
goto out;
- if (!queue)
+ if (!queue || is_rx_queue)
continue;
rc = __netif_set_xps_queue(adapter->netdev,
@@ -291,14 +300,6 @@ static void ibmvnic_set_affinity(struct ibmvnic_adapter *adapter)
__func__, i, rc);
}
- for (i = 0; i < num_rxqs; i++) {
- queue = rxqs[i];
- rc = ibmvnic_set_queue_affinity(queue, &cpu, &stragglers,
- stride);
- if (rc)
- goto out;
- }
-
out:
if (rc) {
netdev_warn(adapter->netdev,
--
2.31.1
Powered by blists - more mailing lists