netdev - IxGbE: Node Aware Descriptor (Ring) Allocation

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <41885A88D9043444B138491FB10E15420476DD180E@azsmsx506.amr.corp.intel.com>
Date:	Tue, 16 Feb 2010 21:28:03 -0700
From:	"Tadepalli, Hari K" <hari.k.tadepalli@...el.com>
To:	"netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: IxGbE: Node Aware Descriptor (Ring) Allocation

Greetings! This is my first post to this list. So please bear with any protocol violations. 

I am looking for hints on NUMA node-aware packet descriptor ring allocation in the IxGbE driver for Intel 82599 (10Gb) NICs, with multi queue feature enabled. 

There are three places in the IxGbE source where we need node aware allocation (shown below for TX only; analogous code for RX assumed): 

(i) tx_ring, rx_ring control structures of tx/rx rings:
	tx_ring->tx_buffer_info = vmalloc_node(size, tx_ring->numa_node);

(ii) descriptor rings in the DMA region: tx_ring->dma, rx_ring->dma
	tx_ring->desc = pci_alloc_consistent(pdev, tx_ring->size, &tx_ring->dma);

(iii) packet buffers:
	struct sk_buff *skb = netdev_alloc_skb(adapter->netdev,bufsz);

(i) and (iii) are able take a node parameter and accordingly return with node-aware allocation (on x86_64/NUMA enabled kernel with SLUB allocator). But, the same is not true of (ii) for descriptor ring allocation. Adding a node_id parameter to pdev structure with set_dev_node() call has no effect - presumably because the underlying dma_alloc_coherent() call has no node awareness. 

I have, hacked pci_alloc_consistent to ingest a node parameter & allocate with dma_generic_alloc_coherent(), which seems to allocate with alloc_pages_node() after stripping node parameter from the pdev->dev object. But, this did not work - I can dump TXD/RXD base addresses with ethregs tool & observe that all allocations are coming from only one of the two nodes present in Nehalem (Xeon 55xx) platforms. 

What is the status of node-aware DMA region allocation for IO devices on x86_64 platforms? Any known fixes?

Thanks, 

Hari
------------------------------------
Intel/Embedded Comms/Chandler/AZ
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html