netdev - RE: Low performance Intel 10GE NIC (3.2.10) on 2.6.38 Kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <D12839161ADD3A4B8DA63D1A134D084026E48BA58D@ESGSCCMS0001.eapac.ericsson.se>
Date:	Fri, 8 Apr 2011 16:59:49 +0800
From:	Wei Gu <wei.gu@...csson.com>
To:	Alexander Duyck <alexander.h.duyck@...el.com>,
	Eric Dumazet <eric.dumazet@...il.com>
CC:	netdev <netdev@...r.kernel.org>,
	"Kirsher, Jeffrey T" <jeffrey.t.kirsher@...el.com>
Subject: RE: Low performance Intel 10GE NIC (3.2.10) on 2.6.38 Kernel

Hi,
On my configuration of Linux 2.6.38 we do enabled the CONFIG_DMAR
CONFIG_DMAR=y
CONFIG_DMAR_DEFAULT_ON=y
CONFIG_DMAR_FLOPPY_WA=y
CONFIG_INTR_REMAP=y

But I enabled this DMAR also by default of the RHEL6 2.6.32 kernel, which we don't see this problem on this kernel. The only different I can see on the config is "CONFIG_DMAR_DEFAULT_ON"
CONFIG_DMAR=y
# CONFIG_DMAR_DEFAULT_ON is not set
CONFIG_DMAR_FLOPPY_WA=y
CONFIG_INTR_REMAP=y

I could try the same configuration as 2.6.32 for the 2.6.38 kernel by unset the CONFIG_DMAR_DEFAULT_ON. And run the test again, hopefully we can get the test result next Monday:)

Thanks
WeiGu

-----Original Message-----
From: Alexander Duyck [mailto:alexander.h.duyck@...el.com]
Sent: Friday, April 08, 2011 12:21 AM
To: Eric Dumazet
Cc: Wei Gu; netdev; Kirsher, Jeffrey T
Subject: Re: Low performance Intel 10GE NIC (3.2.10) on 2.6.38 Kernel

On 4/7/2011 9:03 AM, Eric Dumazet wrote:
> Le jeudi 07 avril 2011 à 08:58 -0700, Alexander Duyck a écrit :
>> On 4/7/2011 4:46 AM, Eric Dumazet wrote:
>>> Le jeudi 07 avril 2011 à 19:15 +0800, Wei Gu a écrit :
>>>> Hi,
>>>> I compile the ixgbe driver into the kernel and run the test again
>>>> and also change the copy to clone in the fw hook This is the perf
>>>> report while I was forwarding 150Kpps with The attached file include the basic info about my test system. Please let me know if I did some thing wrong.
>>>>
>>>> +     71.91%          swapper  [kernel.kallsyms]            [k] poll_idle
>>>> +     10.43%          swapper  [kernel.kallsyms]            [k] intel_idle
>>>> -      8.00%     ksoftirqd/24  [kernel.kallsyms]            [k] _raw_spin_unlock_irqrestore
>>>> \u2592   - _raw_spin_unlock_irqrestore
>>>> \u2592      - 42.25% alloc_iova
>>>> \u2592           intel_alloc_iova
>>>> \u2592           __intel_map_single
>>>> \u2592           intel_map_page
>>
>> I'm almost certain this is the issue here.  I am pretty sure the
>> intel_map_page call indicates that you are running with the Intel
>> IOMMU enabled.  As Eric suggested you can either rebuild your kernel
>> with "CONFIG_DMAR=N", or pass the kernel the parameter
>> "intel_iommu=off" in order to disable it so that it will instead just use SWIOTLB.
>
> What's the purpose of intel_iommu ?
>
> Could this be speedup if ixgbe uses a perqueue iommu context instead
> of a per device, so that we dont hit a single spinlock ?

The intel_iommu is meant to be a security feature.  Primarily it is used in virtualzation where it allows KVM or Xen to direct assign a device without having to worry about the guest getting access to the hosts physical memory by submitting invalid DMA requests.

If virtualzation isn't in use I would recommend turning it off as it can have a negative impact especially on small packet performance due to the extra locking overhead that is required for DMA map and unmap calls.

Thanks,

Alex

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html