Message-ID: <347.1256232960@death.nxdomain.ibm.com>
Date:	Thu, 22 Oct 2009 10:36:00 -0700
From:	Jay Vosburgh <fubar@...ibm.com>
To:	Eric Dumazet <eric.dumazet@...il.com>
cc:	Jasper Spaans <spaans@...-it.com>, netdev@...r.kernel.org
Subject: Re: bridging + load balancing bonding

Eric Dumazet <eric.dumazet@...il.com> wrote:

>Jasper Spaans a écrit :
>> Hi,
>> 
>> We're using the following setup for bonding and bridging, to be able to put
>> large amounts of data through multiple IDS analyzers:
>> 
>>                              +---[br0]----+     +--- eth1 ---(IDS machine 1)
>> (Span port from switch) -- eth0          bond0--+
>>                                                 +--- eth2 ---(IDS machine 2)
>> 
>> eth0 receives network traffic, which should be passed to machines which are
>> connected to eth1 and eth2. These machines run an IDS package, and there are
>> two of those for performance reasons.
>> 
>> bond0 is configured to load balance the packets using "balance-xor", in this
>> case combined with xmit_hash_policy layer2.
>> 
>> However, we're seeing problems: packets from one flow do not end up at the
>> same IDS machine.  This is because this selection is not based on the source
>> _and_ destination mac addresses of the original packet, but on the mac
>> address of the bonding device and the destination mac address of the
>> package.

	By "packets from one flow" do you really mean that packets from
a given "flow" (TCP connection, UDP "stream", etc) are not always
delivered to the same bonding port?  I.e., that two packets from the
same "flow" will be delivered to different ports?  I'm not sure how
that's possible unless the source MAC in the packets changes during the
course of the flow.

	Or is your problem really that the balance algorithm on the
bonding send side doesn't match the algorithm used on the other side of
the IDS machines coming the other direction (and, thus, packets for a
given flow going in one direction end up at a different IDS than the
packets going the other direction)?

>> This is also clear in the code:
>> For example, in bond_main.c, in bond_xmit_hash_policy_l2:
>> 	return (data->h_dest[5] ^ bond_dev->dev_addr[5]) % count;
>> 
>> Changing this to
>> 	return (data->h_dest[5] ^ data->h_source[5]) % count;
>> fixes our problems, but is this harmful for packets originating locally (or
>> being routed?)
>> 
>> If not, can this be applied? Or does anyone have other ideas?
>> 
>
>Hi Jasper
>
>Very nice setup, and nice finding.
>
>Dont locally generated (or outed) packets have h_source set to
>bond_dev->dev_addr anyway ?

	Locally generated packets do, but he's got a bridge in there, so
the traffic they're balancing is presumably not locally generated (i.e.,
is being forwarded by the bridge, in which case they'll still bear the
source MAC of the originating node on the subnet).  If the packets were
being routed instead of bridged, then, yah, they'd have the bond's
source MAC.

>So your solution might be the right fix...

	Yes, I think he's found a legitimate bug, one that only will
manifest when balancing bridged traffic.  I had to think for a minute if
this change would break anything, and I'm coming up empty.  Locally
generated or routed traffic won't see a change, and bridged traffic will
be correctly balanced according to the "source MAC XOR destination MAC"
forumla described in the documentation.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@...ibm.com
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists