| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <03b02880-4aae-4a4a-9533-7a756cc84001@shopee.com>
Date: Fri, 3 Jan 2025 10:36:50 +0800
From: Haifeng Xu <haifeng.xu@...pee.com>
To: Eric Dumazet <edumazet@...gle.com>
Cc: Tony Nguyen <anthony.l.nguyen@...el.com>,
Przemek Kitszel <przemyslaw.kitszel@...el.com>,
"David S. Miller" <davem@...emloft.net>, Jakub Kicinski <kuba@...nel.org>,
Paolo Abeni <pabeni@...hat.com>, linux-kernel@...r.kernel.org,
netdev@...r.kernel.org, intel-wired-lan@...ts.osuosl.org
Subject: Re: [Question] ixgbe：Mechanism of RSS
On 2025/1/2 19:46, Eric Dumazet wrote:
> On Thu, Jan 2, 2025 at 12:23 PM Haifeng Xu <haifeng.xu@...pee.com> wrote:
>>
>>
>>
>> On 2025/1/2 18:34, Eric Dumazet wrote:
>>> On Thu, Jan 2, 2025 at 9:43 AM Haifeng Xu <haifeng.xu@...pee.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2025/1/2 16:13, Eric Dumazet wrote:
>>>>> On Thu, Jan 2, 2025 at 4:53 AM Haifeng Xu <haifeng.xu@...pee.com> wrote:
>>>>>>
>>>>>> Hi masters,
>>>>>>
>>>>>> We use the Intel Corporation 82599ES NIC in our production environment. And it has 63 rx queues, every rx queue interrupt is processed by a single cpu.
>>>>>> The RSS configuration can be seen as follow:
>>>>>>
>>>>>> RX flow hash indirection table for eno5 with 63 RX ring(s):
>>>>>> 0: 0 1 2 3 4 5 6 7
>>>>>> 8: 8 9 10 11 12 13 14 15
>>>>>> 16: 0 1 2 3 4 5 6 7
>>>>>> 24: 8 9 10 11 12 13 14 15
>>>>>> 32: 0 1 2 3 4 5 6 7
>>>>>> 40: 8 9 10 11 12 13 14 15
>>>>>> 48: 0 1 2 3 4 5 6 7
>>>>>> 56: 8 9 10 11 12 13 14 15
>>>>>> 64: 0 1 2 3 4 5 6 7
>>>>>> 72: 8 9 10 11 12 13 14 15
>>>>>> 80: 0 1 2 3 4 5 6 7
>>>>>> 88: 8 9 10 11 12 13 14 15
>>>>>> 96: 0 1 2 3 4 5 6 7
>>>>>> 104: 8 9 10 11 12 13 14 15
>>>>>> 112: 0 1 2 3 4 5 6 7
>>>>>> 120: 8 9 10 11 12 13 14 15
>>>>>>
>>>>>> The maximum number of RSS queues is 16. So I have some questions about this. Will other cpus except 0~15 receive the rx interrupts?
>>>>>>
>>>>>> In our production environment, cpu 16~62 also receive the rx interrupts. Was our RSS misconfigured?
>>>>>
>>>>> It really depends on which cpus are assigned to each IRQ.
>>>>>
>>>>
>>>> Hi Eric,
>>>>
>>>> Each irq was assigned to a single cpu, for exapmle:
>>>>
>>>> irq cpu
>>>>
>>>> 117 0
>>>> 118 1
>>>>
>>>> ......
>>>>
>>>> 179 62
>>>>
>>>> All cpus trigger interrupts not only cpus 0~15.
>>>> It seems that the result is inconsistent with the RSS hash value.
>>>>
>>>>
>>>
>>> I misread your report, I thought you had 16 receive queues.
>>>
>>> Why don't you change "ethtool -L eno5 rx 16", instead of trying to
>>> configure RSS manually ?
>>
>> Hi Eric,
>>
>> We want to make full use of cpu resources to receive packets. So
>> we enable 63 rx queues. But we found the rate of interrupt growth
>> on cpu 0~15 is faster than other cpus(almost twice). I don't know
>> whether it is related to RSS configuration. We didn't make any changes
>> on the RSS configration after the server is up.
>>
>>
>>
>> FYI, on another server, we use Mellanox Technologies MT27800 NIC.
>> The rate of interrupt growth on cpu 0~63 seems have little gap.
>>
>> It's RSS configration can be seen as follow:
>>
>> RX flow hash indirection table for ens2f0np0 with 63 RX ring(s):
>> 0: 0 1 2 3 4 5 6 7
>> 8: 8 9 10 11 12 13 14 15
>> 16: 16 17 18 19 20 21 22 23
>> 24: 24 25 26 27 28 29 30 31
>> 32: 32 33 34 35 36 37 38 39
>> 40: 40 41 42 43 44 45 46 47
>> 48: 48 49 50 51 52 53 54 55
>> 56: 56 57 58 59 60 61 62 0
>> 64: 1 2 3 4 5 6 7 8
>> 72: 9 10 11 12 13 14 15 16
>> 80: 17 18 19 20 21 22 23 24
>> 88: 25 26 27 28 29 30 31 32
>> 96: 33 34 35 36 37 38 39 40
>> 104: 41 42 43 44 45 46 47 48
>> 112: 49 50 51 52 53 54 55 56
>> 120: 57 58 59 60 61 62 0 1
>> 128: 2 3 4 5 6 7 8 9
>> 136: 10 11 12 13 14 15 16 17
>> 144: 18 19 20 21 22 23 24 25
>> 152: 26 27 28 29 30 31 32 33
>> 160: 34 35 36 37 38 39 40 41
>> 168: 42 43 44 45 46 47 48 49
>> 176: 50 51 52 53 54 55 56 57
>> 184: 58 59 60 61 62 0 1 2
>> 192: 3 4 5 6 7 8 9 10
>> 200: 11 12 13 14 15 16 17 18
>> 208: 19 20 21 22 23 24 25 26
>> 216: 27 28 29 30 31 32 33 34
>> 224: 35 36 37 38 39 40 41 42
>> 232: 43 44 45 46 47 48 49 50
>> 240: 51 52 53 54 55 56 57 58
>> 248: 59 60 61 62 0 1 2 3
>>
>>
>> I am confused that why ixgbe NIC can dispatch the packets
>> to the rx queues that not specified in RSS configuration.
>
> Perhaps make sure to change RX flow hash indirection table on the
> Intel NIC then...
>
> Maybe you changed the default configuration.
>
> ethtool -X eno5 equal 64
The maximum number of RSS queues supported by Intel Corporation 82599ES NIC
is 16. When I specify the number which is larger than 16, it shows the below message.
"Cannot set RX flow hash configuration: Invalid argument."
> Or
> ethtool -X eno5 default
This command can run sucessfully and as I saied above, it only has 16 queues.
RX flow hash indirection table for eno5 with 63 RX ring(s):
0: 0 1 2 3 4 5 6 7
8: 8 9 10 11 12 13 14 15
16: 0 1 2 3 4 5 6 7
24: 8 9 10 11 12 13 14 15
32: 0 1 2 3 4 5 6 7
40: 8 9 10 11 12 13 14 15
48: 0 1 2 3 4 5 6 7
56: 8 9 10 11 12 13 14 15
64: 0 1 2 3 4 5 6 7
72: 8 9 10 11 12 13 14 15
80: 0 1 2 3 4 5 6 7
88: 8 9 10 11 12 13 14 15
96: 0 1 2 3 4 5 6 7
104: 8 9 10 11 12 13 14 15
112: 0 1 2 3 4 5 6 7
120: 8 9 10 11 12 13 14 15
Thanks!
Powered by blists - more mailing lists