netdev - Re: [Question] ixgbe：Mechanism of RSS

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <b9c88c0f-7909-43a3-8229-2b0ce7c68c10@shopee.com>
Date: Thu, 2 Jan 2025 19:23:27 +0800
From: Haifeng Xu <haifeng.xu@...pee.com>
To: Eric Dumazet <edumazet@...gle.com>
Cc: Tony Nguyen <anthony.l.nguyen@...el.com>,
 Przemek Kitszel <przemyslaw.kitszel@...el.com>,
 "David S. Miller" <davem@...emloft.net>, Jakub Kicinski <kuba@...nel.org>,
 Paolo Abeni <pabeni@...hat.com>, linux-kernel@...r.kernel.org,
 netdev@...r.kernel.org, intel-wired-lan@...ts.osuosl.org
Subject: Re: [Question] ixgbe：Mechanism of RSS



On 2025/1/2 18:34, Eric Dumazet wrote:
> On Thu, Jan 2, 2025 at 9:43 AM Haifeng Xu <haifeng.xu@...pee.com> wrote:
>>
>>
>>
>> On 2025/1/2 16:13, Eric Dumazet wrote:
>>> On Thu, Jan 2, 2025 at 4:53 AM Haifeng Xu <haifeng.xu@...pee.com> wrote:
>>>>
>>>> Hi masters,
>>>>
>>>>         We use the Intel Corporation 82599ES NIC in our production environment. And it has 63 rx queues, every rx queue interrupt is processed by a single cpu.
>>>>         The RSS configuration can be seen as follow:
>>>>
>>>>         RX flow hash indirection table for eno5 with 63 RX ring(s):
>>>>         0:      0     1     2     3     4     5     6     7
>>>>         8:      8     9    10    11    12    13    14    15
>>>>         16:      0     1     2     3     4     5     6     7
>>>>         24:      8     9    10    11    12    13    14    15
>>>>         32:      0     1     2     3     4     5     6     7
>>>>         40:      8     9    10    11    12    13    14    15
>>>>         48:      0     1     2     3     4     5     6     7
>>>>         56:      8     9    10    11    12    13    14    15
>>>>         64:      0     1     2     3     4     5     6     7
>>>>         72:      8     9    10    11    12    13    14    15
>>>>         80:      0     1     2     3     4     5     6     7
>>>>         88:      8     9    10    11    12    13    14    15
>>>>         96:      0     1     2     3     4     5     6     7
>>>>         104:      8     9    10    11    12    13    14    15
>>>>         112:      0     1     2     3     4     5     6     7
>>>>         120:      8     9    10    11    12    13    14    15
>>>>
>>>>         The maximum number of RSS queues is 16. So I have some questions about this. Will other cpus except 0~15 receive the rx interrupts?
>>>>
>>>>         In our production environment, cpu 16~62 also receive the rx interrupts. Was our RSS misconfigured?
>>>
>>> It really depends on which cpus are assigned to each IRQ.
>>>
>>
>> Hi Eric,
>>
>> Each irq was assigned to a single cpu, for exapmle:
>>
>> irq     cpu
>>
>> 117      0
>> 118      1
>>
>> ......
>>
>> 179      62
>>
>> All cpus trigger interrupts not only cpus 0~15.
>> It seems that the result is inconsistent with the RSS hash value.
>>
>>
> 
> I misread your report, I thought you had 16 receive queues.
> 
> Why don't you change "ethtool -L eno5 rx 16", instead of trying to
> configure RSS manually ?

Hi Eric,

We want to make full use of cpu resources to receive packets. So
we enable 63 rx queues. But we found the rate of interrupt growth
on cpu 0~15 is faster than other cpus(almost twice). I don't know 
whether it is related to RSS configuration. We didn't make any changes
on the RSS configration after the server is up.



FYI, on another server, we use Mellanox Technologies MT27800 NIC.
The rate of interrupt growth on cpu 0~63 seems have little gap.

It's RSS configration can be seen as follow:

RX flow hash indirection table for ens2f0np0 with 63 RX ring(s):
    0:      0     1     2     3     4     5     6     7
    8:      8     9    10    11    12    13    14    15
   16:     16    17    18    19    20    21    22    23
   24:     24    25    26    27    28    29    30    31
   32:     32    33    34    35    36    37    38    39
   40:     40    41    42    43    44    45    46    47
   48:     48    49    50    51    52    53    54    55
   56:     56    57    58    59    60    61    62     0
   64:      1     2     3     4     5     6     7     8
   72:      9    10    11    12    13    14    15    16
   80:     17    18    19    20    21    22    23    24
   88:     25    26    27    28    29    30    31    32
   96:     33    34    35    36    37    38    39    40
  104:     41    42    43    44    45    46    47    48
  112:     49    50    51    52    53    54    55    56
  120:     57    58    59    60    61    62     0     1
  128:      2     3     4     5     6     7     8     9
  136:     10    11    12    13    14    15    16    17
  144:     18    19    20    21    22    23    24    25
  152:     26    27    28    29    30    31    32    33
  160:     34    35    36    37    38    39    40    41
  168:     42    43    44    45    46    47    48    49
  176:     50    51    52    53    54    55    56    57
  184:     58    59    60    61    62     0     1     2
  192:      3     4     5     6     7     8     9    10
  200:     11    12    13    14    15    16    17    18
  208:     19    20    21    22    23    24    25    26
  216:     27    28    29    30    31    32    33    34
  224:     35    36    37    38    39    40    41    42
  232:     43    44    45    46    47    48    49    50
  240:     51    52    53    54    55    56    57    58
  248:     59    60    61    62     0     1     2     3


I am confused that why ixgbe NIC can dispatch the packets
to the rx queues that not specified in RSS configuration.

Thanks!