linux-kernel - Re: QUESTION: can netdev_alloc_skb() errors be reduced by tuning?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20090616091932.GB14241@csn.ul.ie>
Date:	Tue, 16 Jun 2009 10:19:32 +0100
From:	Mel Gorman <mel@....ul.ie>
To:	starlight@...nacle.cx
Cc:	linux-kernel@...r.kernel.org, linux-mm@...ck.org,
	hugh.dickins@...cali.co.uk, Lee.Schermerhorn@...com,
	kosaki.motohiro@...fujitsu.com, ebmunson@...ibm.com,
	agl@...ibm.com, apw@...onical.com, wli@...ementarian.org
Subject: Re: QUESTION: can netdev_alloc_skb() errors be reduced by tuning?

On Mon, Jun 15, 2009 at 08:19:33PM -0400, starlight@...nacle.cx wrote:
> Hello,
> 
> I submitted testcase for a hugepages bug that has been 
> successfully resolved.  Have an apparently obscure question 
> related to MM, and so I am asking anyone who might have some idea 
> on this.  Nothing much turned up via Google and digging into
> the KMEM code looks daunting.
> 
> Running Intel 82598/ixgbe 10 gig Ethernet under heavy stress. 
> Generally is working well after tuning IRQ affinities, but a 
> fair number of buffer allocation failures are occurring in the 
> 'ixgbe' device driver and are reported via 'ethtool' statistics. 
>  This may be causing data loss.
> 

Can you give an example of an allocation failure? Specifically, I want to
see what sort of allocation it was and what order.

For reliable protocols, an allocation failure should recover and the
data get through but obviously there is a drop in network performance
when this happens.

> The kernel primitive returning the error is netdev_alloc_skb().
> 
> Are any tuneable parameters available that can reduce or 
> eliminate these allocation failures?  Have about eleven 
> gigabytes of free memory, though most of that is consumed 
> by non-dirty file cache data.  Total system memory is 16GB with 
> 4GB allocated to hugepages.  Zero swap usage and activity though
> swap is enabled.  Most application memory is hugepage or is
> 'mlock()'ed.
> 

If the allocations are high-order and atomic, increasing min_free_kbytes
can help, particularly in situations where there is a burst of network
traffic. I won't know if they are atomic until I see an error message
though.

> Thank you.
> 
> 
> 
> 
> 
> System rebooted before test run.
> 
> Dual Xeon E5430, 16GB FB-DIMM RAM.
> 
> 
> $ cat /proc/meminfo
> MemTotal:     16443828 kB
> MemFree:        281176 kB
> Buffers:         53896 kB
> Cached:       11331924 kB
> SwapCached:          0 kB
> Active:         200740 kB
> Inactive:     11284312 kB
> HighTotal:           0 kB
> HighFree:            0 kB
> LowTotal:     16443828 kB
> LowFree:        281176 kB
> SwapTotal:     2031608 kB
> SwapFree:      2031400 kB
> Dirty:               4 kB
> Writeback:           0 kB
> AnonPages:      104464 kB
> Mapped:          14644 kB
> Slab:           440452 kB
> PageTables:       4032 kB
> NFS_Unstable:        0 kB
> Bounce:              0 kB
> CommitLimit:   8156368 kB
> Committed_AS:   122452 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed:    266872 kB
> VmallocChunk: 34359471043 kB
> HugePages_Total:  2048
> HugePages_Free:    735
> HugePages_Rsvd:      0
> Hugepagesize:     2048 kB
> 
> 
> # ethtool -S eth2 | egrep -v ': 0$'
> NIC statistics:
>      rx_packets: 724246449
>      tx_packets: 229847
>      rx_bytes: 152691992335
>      tx_bytes: 10573426
>      multicast: 725997241
>      broadcast: 6
>      rx_csum_offload_good: 723051776
>      alloc_rx_buff_failed: 7119
>      tx_queue_0_packets: 229847
>      tx_queue_0_bytes: 10573426
>      rx_queue_0_packets: 340698332
>      rx_queue_0_bytes: 70844299683
>      rx_queue_1_packets: 385298923
>      rx_queue_1_bytes: 82276167594
> 
> 
> ixgbe driver fragment
> =====================
>     struct sk_buff *skb = netdev_alloc_skb(adapter->netdev, bufsz);
> 
>     if (!skb) {
>         adapter->alloc_rx_buff_failed++;
>         goto no_buffers;
>     }
> 

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/