netdev - Re: Performance issue with igb with lots of different src-ip addrs.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <fb63a694-e854-c94a-2e03-8737295775b1@candelatech.com>
Date:   Fri, 24 Mar 2017 13:54:08 -0700
From:   Ben Greear <greearb@...delatech.com>
To:     Alexander Duyck <alexander.duyck@...il.com>
Cc:     e1000-devel list <e1000-devel@...ts.sourceforge.net>,
        netdev <netdev@...r.kernel.org>
Subject: Re: Performance issue with igb with lots of different src-ip addrs.

On 03/16/2017 08:51 PM, Ben Greear wrote:
> I think we can, might take us a day or two to get time to do it.
>
> Thanks,
> Ben
>
> On 03/16/2017 08:05 PM, Alexander Duyck wrote:
>> I'm not really interested in installing a custom version of pktgen.
>> Any chance you can recreate the issue with standard pktgen?
>>
>> You might try running perf to get a snapshot of what is using CPU time
>> on the system.  It will probably give you a pretty good idea where the
>> code is that is eating up all your CPU time.

So, I had time to dig into this today.

Turns out that our tool was reporting drops because of sequence number
gaps, which in turn were caused by out-of-order frames...not actually
dropping frames.

If I force the rss_queues to one, the problem goes away.

Sorry for mis-reporting a bug.

Thanks,
Ben


>>
>> - Alex
>>
>> On Thu, Mar 16, 2017 at 7:46 PM, Ben Greear <greearb@...delatech.com> wrote:
>>> I'm actually using a hacked up version of pktgen nicely driven by our
>>> GUI tool, but the crux is that you need to set min and max src IP to some
>>> large
>>> range.
>>>
>>> We are driving pktgen from a separate machine.  Stock pktgen isn't good at
>>> reporting
>>> received pkts last I checked, so it may be more difficult to easily view the
>>> problem.
>>>
>>> I'll be happy to set up my tool on your Fedora 24 or similar VM or machine
>>> if you
>>> want.
>>>
>>> Thanks,
>>> Ben
>>>
>>>
>>> On 03/16/2017 07:35 PM, Alexander Duyck wrote:
>>>>
>>>> Can you include the pktgen script you are running?
>>>>
>>>> Also when you say you are driving traffic through the bridge are you
>>>> sending from something external on the system or are you actually
>>>> directing the traffic from pktgen into the bridge directly?
>>>>
>>>> - Alex
>>>>
>>>> On Thu, Mar 16, 2017 at 3:49 PM, Ben Greear <greearb@...delatech.com>
>>>> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> We notice that when using two igb ports as a bridge, if we use pktgen to
>>>>> drive traffic through the bridge and randomize (or use a very large
>>>>> range)
>>>>> for the source IP addr in pktgen, then performance of igb is very poor
>>>>> (like
>>>>> 150Mbps
>>>>> throughput instead of 1Gbps).  It runs right at line speed if we use same
>>>>> src/dest
>>>>> IP addr in pktgen.  So, seems it is related to lots of src/dest IP
>>>>> addresses.
>>>>>
>>>>> We see same problem when using pktgen to send to itself, and we see this
>>>>> in
>>>>> several different kernels.  We specifically tested bridge mode in this
>>>>> stock
>>>>> Fedora kernel:
>>>>>
>>>>>  Linux lfo350-59cc 4.9.13-101.fc24.x86_64 #1 SMP Tue Mar 7 23:48:32 UTC
>>>>> 2017
>>>>> x86_64 x86_64 x86_64 GNU/Linux
>>>>>
>>>>> e1000e does not show this problem in our testing.
>>>>>
>>>>> Any ideas what the issue might be and how to fix it?
>>>>>
>>>>> Thanks,
>>>>> Ben
>>>>>
>>>>> --
>>>>> Ben Greear <greearb@...delatech.com>
>>>>> Candela Technologies Inc  http://www.candelatech.com
>>>>>
>>>>
>>>
>>> --
>>> Ben Greear <greearb@...delatech.com>
>>> Candela Technologies Inc  http://www.candelatech.com
>>
>


-- 
Ben Greear <greearb@...delatech.com>
Candela Technologies Inc  http://www.candelatech.com