Message-ID: <1ade15b1-f533-4cc6-8522-2d725532e251@shopee.com>
Date: Wed, 8 Jan 2025 11:36:29 +0800
From: Haifeng Xu <haifeng.xu@...pee.com>
To: Tony Nguyen <anthony.l.nguyen@...el.com>,
 Edward Cree <ecree.xilinx@...il.com>, Eric Dumazet <edumazet@...gle.com>,
 Aleksandr Loktionov <aleksandr.loktionov@...el.com>,
 "Kwapulinski, Piotr" <piotr.kwapulinski@...el.com>
Cc: Przemek Kitszel <przemyslaw.kitszel@...el.com>,
 "David S. Miller" <davem@...emloft.net>, Jakub Kicinski <kuba@...nel.org>,
 Paolo Abeni <pabeni@...hat.com>, linux-kernel@...r.kernel.org,
 netdev@...r.kernel.org, intel-wired-lan@...ts.osuosl.org
Subject: Re: [Question] ixgbe：Mechanism of RSS



On 2025/1/8 01:16, Tony Nguyen wrote:
> 
> 
> On 1/2/2025 7:05 PM, Haifeng Xu wrote:
>>
>>
>> On 2025/1/3 00:01, Edward Cree wrote:
>>> On 02/01/2025 11:23, Haifeng Xu wrote:
>>>> We want to make full use of cpu resources to receive packets. So
>>>> we enable 63 rx queues. But we found the rate of interrupt growth
>>>> on cpu 0~15 is faster than other cpus(almost twice).
>>> ...
>>>> I am confused that why ixgbe NIC can dispatch the packets
>>>> to the rx queues that not specified in RSS configuration.
>>>
>>> Hypothesis: it isn't doing so, RX is only happening on cpus (and
>>>   queues) 0-15, but the other CPUs are still sending traffic and
>>>   thus getting TX completion interrupts from their TX queues.
>>> `ethtool -S` output has per-queue traffic stats which should
>>>   confirm this.
>>>
>>
>> I use ethtool -S to check the rx_queus stats and here is the result.
>>
>> According to the below stats, all cpus have new packets received.
> 
> + Alex and Piotr
> 
> What's your ntuple filter setting? If it's off, I suspect it may be the Flow Director ATR (Application Targeting Routing) feature which will utilize all queues. I believe if you turn on ntuple filters this will turn that feature off.

Yes, our ntuple filter setting is off. After turning on the ntuple filters, I compare the delta of recieved packets,
only 0~15 rx rings are non-zero, other rx rings are zero.

If we want to spread the packets across 0~62, how can we tune the NIC setting?
we have enabled 63 rx queues, irq_affinity and rx-flow-hash, but the 0~15 cpu
received more packets than others.

Thanks!



> 
> Thanks,
> Tony
> 
>>
>> cpu     t1(bytes)       t2(bytes)       delta(bytes)
>>
>> 0    154155550267550    154156433828875    883561325
>> 1    148748566285840    148749509346247    943060407
>> 2    148874911191685    148875798038140    886846455
>> 3    152483460327704    152484251468998    791141294
>> 4    147790981836915    147791775847804    794010889
>> 5    146047892285722    146048778285682    885999960
>> 6    142880516825921    142881213804363    696978442
>> 7    152016735168735    152017707542774    972374039
>> 8    146019936404393    146020739070311    802665918
>> 9    147448522715540    147449258018186    735302646
>> 10    145865736299432    145866601503106    865203674
>> 11    149548527982122    149549289026453    761044331
>> 12    146848384328236    146849303547769    919219533
>> 13    152942139118542    152942769029253    629910711
>> 14    150884661854828    150885556866976    895012148
>> 15    149222733506734    149223510491115    776984381
>> 16    34150226069524    34150375855113    149785589
>> 17    34115700500819    34115914271025    213770206
>> 18    33906215129998    33906448044501    232914503
>> 19    33983812095357    33983986258546    174163189
>> 20    34156349675011    34156565159083    215484072
>> 21    33574293379024    33574490695725    197316701
>> 22    33438129453422    33438297911151    168457729
>> 23    32967454521585    32967612494711    157973126
>> 24    33507443427266    33507604828468    161401202
>> 25    33413275870121    33413433901940    158031819
>> 26    33852322542796    33852527061150    204518354
>> 27    33131162685385    33131330621474    167936089
>> 28    33407661780251    33407823112381    161332130
>> 29    34256799173845    34256944837757    145663912
>> 30    33814458585183    33814623673528    165088345
>> 31    33848638714862    33848775218038    136503176
>> 32    18683932398308    18684069540891    137142583
>> 33    19454524281229    19454647908293    123627064
>> 34    19717744365436    19717900618222    156252786
>> 35    20295086765202    20295245869666    159104464
>> 36    20501853066588    20502000738936    147672348
>> 37    20954631043374    20954797204375    166161001
>> 38    21102911073326    21103062510369    151437043
>> 39    21376404644179    21376515307288    110663109
>> 40    20935812784743    20935983891491    171106748
>> 41    20721278456831    20721435955715    157498884
>> 42    21268291801465    21268425244578    133443113
>> 43    21661413672829    21661629019091    215346262
>> 44    21696437732484    21696568800049    131067565
>> 45    21027869000890    21028020401214    151400324
>> 46    21707137252644    21707293761990    156509346
>> 47    20655623913790    20655740452889    116539099
>> 48    32692002128477    32692138244468    136115991
>> 49    33548445851486    33548569927672    124076186
>> 50    33197264968787    33197448645817    183677030
>> 51    33379544010500    33379746565576    202555076
>> 52    33503579011721    33503722596159    143584438
>> 53    33145734550468    33145892305819    157755351
>> 54    33422692741858    33422844156764    151414906
>> 55    32750945531107    32751131302251    185771144
>> 56    33404955373530    33405157766253    202392723
>> 57    33701185654471    33701313174725    127520254
>> 58    33014531699810    33014700058409    168358599
>> 59    32948906758429    32949151147605    244389176
>> 60    33470813725985    33470993164755    179438770
>> 61    33803771479735    33803971758441    200278706
>> 62    33509751180818    33509926649969    175469151
>>
>> Thanks!
>>
>>> (But Eric is right that if you _want_ RX to use every CPU you
>>>   should just change the indirection table.)
>>
> 

Powered by blists - more mailing lists