netdev - RE: [RFC] Setting processor affinity for network queues

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <41885A88D9043444B138491FB10E15420476F1ED89@azsmsx506.amr.corp.intel.com>
Date:	Mon, 1 Mar 2010 10:43:15 -0700
From:	"Tadepalli, Hari K" <hari.k.tadepalli@...el.com>
To:	Ben Hutchings <bhutchings@...arflare.com>,
	netdev <netdev@...r.kernel.org>
Subject: RE: [RFC] Setting processor affinity for network queues

>> From: netdev-owner@...r.kernel.org [mailto:netdev-owner@...r.kernel.org] On Behalf Of Ben Hutchings
>> Sent: Monday, March 01, 2010 10:22 AM
>> To: netdev
>> Cc: Waskiewicz Jr, Peter P; Peter Zijlstra; Thomas Gleixner; Tom Herbert; Stephen Hemminger; sf-linux-drivers
>> Subject: [RFC] Setting processor affinity for network queues
>>
>> Currently we have a userspace interface for setting affinity of IRQs and
>> a convention for naming each channel's IRQ handler, but no such
>> interface for memory allocation.  For RX buffers this should not be a
>> problem since they are normally allocated as older buffers are
>> completed, in the NAPI context.  However, the DMA descriptor rings and
>> driver structures for a channel should also be allocated on the NUMA
>> node where NAPI processing is done.  Currently this allocation takes
>> place when a net device is created or when it is opened, before an
>> administrator has any opportunity to configure affinity.  Reallocation
>> will normally require a complete stop to network traffic (at least on
>> the affected queues) so it should not be done automatically when the
>> driver detects a change in IRQ affinity
y.  There needs to be an explicit
>> mechanism for changing it.

I have sought clarifications on this issue 2 weeks ago in the context of IxGbE: 
http://marc.info/?l=linux-netdev&m=126638089307398&w=2

>> (i) tx_ring, rx_ring control structures of tx/rx rings:
>>	 tx_ring->tx_buffer_info = vmalloc_node(size, tx_ring->numa_node);
>> 
>> (ii) descriptor rings in the DMA region: tx_ring->dma, rx_ring->dma
>>	  tx_ring->desc = pci_alloc_consistent(pdev, tx_ring->size, &tx_ring->dma);
>> 
>> (iii) packet buffers:
>>	   struct sk_buff *skb = netdev_alloc_skb(adapter->netdev,bufsz);

The above memory allocation calls (sparing vmalloc_node()) conventionally used in a network driver allow for node awareness only at the physical device (pdev) level, where queues in a device do not have independent node-status. This is a serious setback to the benefits expected of recent developments for high throughput networking: RSS, 10/40GbE, NUMA & many cores on an SBC (single board computer or server). Cross node traffic, even in small proportions, is a retard on performance in the million packets/sec range. 

I have hacked into some of the above, by using alloc_pages_node() & setting node_id explicitly when allocating for specific queues. But, look forward to a clean, 'hack-free' solution. 

Thanks, 

- Hari
__________________________________________
Embeedded Communications Group
Intel/ IAG, Chandler, AZ
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html