linux-kernel - Re: QUESTION: can netdev_alloc_skb() errors be reduced by tuning?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4A3702CF.9070303@gmail.com>
Date:	Tue, 16 Jun 2009 04:26:23 +0200
From:	Eric Dumazet <eric.dumazet@...il.com>
To:	starlight@...nacle.cx
CC:	linux-kernel@...r.kernel.org, Mel Gorman <mel@....ul.ie>,
	linux-mm@...ck.org, hugh.dickins@...cali.co.uk,
	Lee.Schermerhorn@...com, kosaki.motohiro@...fujitsu.com,
	ebmunson@...ibm.com, agl@...ibm.com, apw@...onical.com,
	wli@...ementarian.org
Subject: Re: QUESTION: can netdev_alloc_skb() errors be reduced by  tuning?

starlight@...nacle.cx a écrit :
> Hello,
> 
> I submitted testcase for a hugepages bug that has been 
> successfully resolved.  Have an apparently obscure question 
> related to MM, and so I am asking anyone who might have some idea 
> on this.  Nothing much turned up via Google and digging into
> the KMEM code looks daunting.
> 
> Running Intel 82598/ixgbe 10 gig Ethernet under heavy stress. 
> Generally is working well after tuning IRQ affinities, but a 
> fair number of buffer allocation failures are occurring in the 
> 'ixgbe' device driver and are reported via 'ethtool' statistics. 
>  This may be causing data loss.
> 
> The kernel primitive returning the error is netdev_alloc_skb().
> 
> Are any tuneable parameters available that can reduce or 
> eliminate these allocation failures?  Have about eleven 
> gigabytes of free memory, though most of that is consumed 
> by non-dirty file cache data.  Total system memory is 16GB with 
> 4GB allocated to hugepages.  Zero swap usage and activity though
> swap is enabled.  Most application memory is hugepage or is
> 'mlock()'ed.
> 
> Thank you.
> 
> 
> 
> 
> 
> System rebooted before test run.
> 
> Dual Xeon E5430, 16GB FB-DIMM RAM.
> 
> 
> $ cat /proc/meminfo
> MemTotal:     16443828 kB
> MemFree:        281176 kB
> Buffers:         53896 kB
> Cached:       11331924 kB
> SwapCached:          0 kB
> Active:         200740 kB
> Inactive:     11284312 kB
> HighTotal:           0 kB
> HighFree:            0 kB
> LowTotal:     16443828 kB
> LowFree:        281176 kB
> SwapTotal:     2031608 kB
> SwapFree:      2031400 kB
> Dirty:               4 kB
> Writeback:           0 kB
> AnonPages:      104464 kB
> Mapped:          14644 kB
> Slab:           440452 kB
> PageTables:       4032 kB
> NFS_Unstable:        0 kB
> Bounce:              0 kB
> CommitLimit:   8156368 kB
> Committed_AS:   122452 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed:    266872 kB
> VmallocChunk: 34359471043 kB
> HugePages_Total:  2048
> HugePages_Free:    735
> HugePages_Rsvd:      0
> Hugepagesize:     2048 kB
> 
> 
> # ethtool -S eth2 | egrep -v ': 0$'
> NIC statistics:
>      rx_packets: 724246449
>      tx_packets: 229847
>      rx_bytes: 152691992335
>      tx_bytes: 10573426
>      multicast: 725997241
>      broadcast: 6
>      rx_csum_offload_good: 723051776
>      alloc_rx_buff_failed: 7119
>      tx_queue_0_packets: 229847
>      tx_queue_0_bytes: 10573426
>      rx_queue_0_packets: 340698332
>      rx_queue_0_bytes: 70844299683
>      rx_queue_1_packets: 385298923
>      rx_queue_1_bytes: 82276167594
> 
> 
> ixgbe driver fragment
> =====================
>     struct sk_buff *skb = netdev_alloc_skb(adapter->netdev, bufsz);
> 
>     if (!skb) {
>         adapter->alloc_rx_buff_failed++;
>         goto no_buffers;
>     }
> 

152691992335/724246449 = 210 bytes per rx packet in average

It could make sense to add copybreak feature in this driver to reduce memory needs,
but that also would consume more cpu cycles, and slow down forwarding setups.

Maybe this packet trimming could be done generically in UDP stack input path,
before queueing packet into a receive queue, if amount of available memory
is under a given threshold.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/