netdev - Re: [lxc-devel] Poor bridging performance on 10 GbE

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <b30d1c3b0903180856r6dce554ap35b8d6a3ee7829e0@mail.gmail.com>
Date:	Thu, 19 Mar 2009 00:56:53 +0900
From:	Ryousei Takano <ryousei@...il.com>
To:	Daniel Lezcano <dlezcano@...ibm.com>
Cc:	lxc-devel@...ts.sourceforge.net,
	Linux Containers <containers@...ts.osdl.org>,
	Linux Netdev List <netdev@...r.kernel.org>
Subject: Re: [lxc-devel] Poor bridging performance on 10 GbE

Hi Daniel,

On Wed, Mar 18, 2009 at 7:10 PM, Daniel Lezcano <dlezcano@...ibm.com> wrote:
> Ryousei Takano wrote:
>>
>> Hi all,
>>
>> I am evaluating the networking performance of lxc on 10 Gigabit Ethernet
>> by
>> using netperf benchmark.
>
> Thanks for doing benchmarking.
> I did two years ago similar tests and there is an analysis of the
> performances at:
> http://lxc.sourceforge.net/network/benchs.php
>
> It is not up to date, but that will give you some clues of what is happening
> with this overhead.
>
I am using VServer because other virtualization mechanisms, including OpenVZ,
Xen, and KVM cannot fully utilize the network bandwidth of 10 GbE.

Here are the results of netperf bencmark:
	vanilla (2.6.27-9)		9525.94
	Vserver (2.6.27.10)	9521.79
	OpenVZ (2.6.27.10)	2049.89
	Xen (2.6.26.1)		1011.47
	KVM (2.6.27-9)		1022.42

Now I am interesting to use LXC instead of VServer.

>> Using a macvlan device, the throughput was 9.6 Gbps. But, using a veth
>> device,
>> the throughput was only 2.7 Gbps.
>
> Yeah, definitively the macvlan interfaces is the best in terms of
> performances but with the restriction of not being able to communicate
> between containers on the same hosts.
>
This restriction is not so big issue for my purpose.

> There are some discussions around that:
>
> http://marc.info/?l=linux-netdev&m=123643508124711&w=2
>
> The veth is a virtual device hence it has not offloading. When the packet
> are sent out, the network stack looks at the nic offloading capability which
> is not present. So the kernel will compute the checksums instead of letting
> the nic to do that either if the packet is transmitted through the physical
> nic. This is a well known issue related to network virtualization and xen
> has developed a specific network driver:
> http://www.cse.psu.edu/~bhuvan/teaching/spring06/papers/xen-net-opt.pdf
>
>> I think this is because the overhead of bridging devices is high.
>
> Yes, bridging adds some overhead and AFAIR bridging + netfilter does some
> skb copy.
>
Thank you for pointers.

>> I also checked the host OS's performance when I used a veth device.
>> I observed a strange phenomenon.
>>
>> Before issuing lxc-start command, the throughput was 9.6 Gbps.
>> Here is the output of brctl show:
>>        $ brctl show
>>        bridge name     bridge id               STP enabled     interfaces
>>        br0             8000.0060dd470d49       no              eth1
>>
>> After issuing lxc-start command, the throughput decreased to 3.2 Gbps.
>> Here is the output of brctl show:
>>        $ sudo brctl show
>>        bridge name     bridge id               STP enabled     interfaces
>>        br0             8000.0060dd470d49       no              eth1
>>                                                                veth0_7573
>>
>> I wonder why the performance is greatly influenced by adding a veth device
>> to a bridge device.
>
> Hmm, good question :)
>
>> Here is my experimental setting:
>>        OS: Ubuntu server 8.10 amd64
>>        Kernel: 2.6.27-rc8 (checkout from the lxc git repository)
>
> I would recommend to use the 2.6.29-rc8 vanilla because this kernel does no
> longer need patches, a lot of fixes were done in the network namespace and
> maybe the bridge has been improved in the meantime :)
>
I checked out the 2.6.29-rc8 vanilla kernel.
The performance after issuing lxc-start improved to 8.7 Gbps!
It's a big improvement, while some performance loss remains.
Can not we avoid this loss?

>>        Userland tool: 0.6.0
>>        NIC: Myricom Myri-10G
>>
>> Any comments and suggestions will be appreciate.
>> If this list is not proper place to talk about this problem, can anyone
>> tell me
>> the proper one?
>
> The performances question is more related to the network virtualization
> implementation and should be sent to netdev@ and containers@ (added in the
> Cc' of this email), of course people at lxc-devel@ will be interested by
> these aspects, so lxc-devel@ is the right mailing list too.
>
> Thanks for your testings
>  -- Daniel
>

Best regards,
Ryousei Takano
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html