Date:	Sat, 1 Nov 2008 05:20:19 +0100 (CET)
From:	Krzysztof Oledzki <ole@....pl>
To:	"Brandeburg, Jesse" <jesse.brandeburg@...el.com>
cc:	"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	"kaber@...sh.net" <kaber@...sh.net>,
	"e1000-devel@...ts.sourceforge.net" 
	<e1000-devel@...ts.sourceforge.net>,
	"Graham, David" <david.graham@...el.com>
Subject: RE: e1000 kills IPMI on 82541GI (Dell PE1425SC) + small 2.6.27
 regression



On Fri, 31 Oct 2008, Brandeburg, Jesse wrote:

> <modified the mail list a bit>
>
> Krzysztof Oledzki wrote:
>> I have started implementing IPMI on my servers and there is some
>> inconvenience with e1000+82541GI (Dell PE1425SC).
>
> ugh, sorry to hear about that.
>
>> With older kernels IPMI works if there are no vlans attached to
>> primary interface. Adding vlans breaks it as e1000 driver enables
>> VLAN filter and it somehow interferences with IPMI.
>>
>> (no vlans):
>> # ethtool -d eth0|grep -i vlan
>>        VLAN mode:                         disabled
>>        VLAN filter:                       disabled
>>
>> (vlans):
>> # ethtool -d eth0|grep -i vlan
>>        VLAN mode:                         enabled
>>        VLAN filter:                       enabled
>
> vlan enable does a couple of things, it mostly just allows the device to
>
> strip the vlan tag (that matches our vlan table in hardware) from
> incoming
> packets and put it in the rx descriptor. This modifies the original
> packet as the vlan is stripped.

So, the same vlan table is used for both host OS and IPMI, right?

>> However, I found that using a workaround by adding the IPMI vlan to
>> the OS restores the functionality. Very ugly but works.
>
> interesting, this is whith the older <= 2.6.26 kernel right?

Yep, exactly. So far I have tested 2.6.22, 2.6.24, 2.6.26 and 2.6.27 
kernels.

If there are no vlans configured on a host I have:

<=2.6.26:
# ethtool -d eth0|grep -i vlan
       VLAN mode:                         disabled
       VLAN filter:                       disabled

>=2.6.27:
# ethtool -d eth0|grep -i vlan
       VLAN mode:                         disabled
       VLAN filter:                       enabled


>> So, it is OK with no-vlan-enabled hosts and with vlan-enabled with the
>> above workaround, but still - it is a ugly workaround: IPMI is
>> non-functional until OS *properly* initializes network configuration,
>> which may never happen.
>
> yeah that's pretty ugly, and not acceptable if you're trying to use
> IPMI.

Indeed.

>> Everything becomes much more complicated with more recent kernels
>> (2.6.27 for example), where vlan filter is now enabled
>> unconditionally and I have to add the dummy IPMI vlan also on
>> no-vlan-enabled hosts to make it working. That makes it really
>> confusing.
>
> yeah, that's a not so good regression due to unconditionally turning on
> VLAN hardware offload.
>
>> So, is it possible to fix this? For example to change the driver to
>> read IPMI vlan id and add it to vlan filter regardless of current OS
>> configuration? Or maybe it is possible to prevent IPMI from
>> interferencing with the current driver's state?
>
> It might be possible to fix this.  The hardware doesn't *have* to
> support VLAN offload in hardware.  This would essentially leave all
> packets
> unmodified as they arrive.  We will definitely look
>
> So, do you intend that IPMI traffic should be on its own vlan?

Exactly. This configuration is very handy - most servers are connected to 
one, untagged vlan and a host OS does not need to know about vlans at all. 
IPMI is provided by one, dedicated, configured as an additional, tagged 
vlan on hosts' ports.

>  Would this kind of configuration work?

Why not? Dell's BMC setup even allow to set it, on IBM servers for example 
I need to use ipmitool to do this, however there are different NICs 
installed (bnx2). Having a dedicated vlan for IPMI service is very handy 
as you may limit the traffic to one network and only allow to access it 
from a very limited number of hosts. Plus, you don't need to worry about 
IPMI configuration even when you move a server from one vlan to another.

> What if the IPMI traffic was 
> *not* on a vlan but you had vlans enabled in your network?

Good question, I'll check this. However, as I stated above, I really would 
like to keep IPMI on one, dedicated vlan. Having to spread IPMI over 40+ 
vlans connected to a number of different routers and keeping such a 
configuration running looks like a very painful job. Please also note that 
you remember about IPMI only when you have to use it, mostly in emergency 
cases. Keeping it simple may even allow to make it accidentally usable at 
that time. ;)

> Let me summarize, please correct me if I'm wrong:
> 0) you want to run IPMI traffic on its own VLAN
Yep.

> 1) 2.6.26 kernel with vlan enabled IPMI breaks doesn't work until driver
> is loaded?

I have a non-modular kernel with the driver build statically into it. IPMI 
obviously works before I booted a Linux kernel, not sure if making th 
driver modular changes anything (I hope not). With the statically linked 
driver I have:

<=2.6.26:
  - non-vlan host -> works
  - vlan host -> works with the dummy vlan

>=2.6.27:
  - works only with the dummy vlan

I'll retest with a modular configuration and let you know.

> 2) 2.6.27 kernel (which enables vlan stripping) breaks IPMI vlan traffic
> until "dummy"
>   vlan is added with driver (on non-vlan hosts)

Exactly.

> I'm feeling a bit confused at this point, can you please describe what
> you want to do
> with both the host and the ipmi config?

I would like to use IPMI so I'll be able to power on/off or reset my 
systems in case of problem with a host OS. There are number of things you 
may do with a IP-enabled KVM but usually dealing with hardware buttons is 
not one of them. ;)

> I'll have our lab see if they can reproduce this, I'm not even sure how
> many others have
> tried IPMI over VLAN.

Thanks, I very appreciate it.

Best regards,

 			Krzysztof Olędzki

Powered by blists - more mailing lists