| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Thu, 17 May 2012 19:54:28 +0300
From: Denys Fedoryshchenko <denys@...p.net.lb>
To: <netdev@...r.kernel.org>, <e1000-devel@...ts.sourceforge.net>,
<jeffrey.t.kirsher@...el.com>, <jesse.brandeburg@...el.com>,
<therbert@...gle.com>, <eric.dumazet@...il.com>,
<davem@...emloft.net>
Subject: Re: Strange latency spikes/TX network stalls on Sun Fire X4150(x86)
and e1000e
Also i notice, limit constantly changing over time (even i am not
touching it).
centaur ~ # grep "" /sys/class/net/eth0/queues/tx-0/byte_queue_limits/*
/sys/class/net/eth0/queues/tx-0/byte_queue_limits/hold_time:1000
/sys/class/net/eth0/queues/tx-0/byte_queue_limits/inflight:0
/sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit:13018
/sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit_max:1879048192
/sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit_min:0
centaur ~ # grep "" /sys/class/net/eth0/queues/tx-0/byte_queue_limits/*
/sys/class/net/eth0/queues/tx-0/byte_queue_limits/hold_time:1000
/sys/class/net/eth0/queues/tx-0/byte_queue_limits/inflight:4542
/sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit:13018
/sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit_max:1879048192
/sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit_min:0
Is it supposed to be like this?
On 2012-05-17 16:42, Denys Fedoryshchenko wrote:
> Found commit that cause problem:
>
> author Tom Herbert <therbert@...gle.com>
> Mon, 28 Nov 2011 16:33:16 +0000 (16:33 +0000)
> committer David S. Miller <davem@...emloft.net>
> Tue, 29 Nov 2011 17:46:19 +0000 (12:46 -0500)
> commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c
> tree d6670a4f94b2b9dedacc38edb6f0e1306b889f6b tree | snapshot
> parent 114cf5802165ee93e3ab461c9c505cd94a08b800 commit | diff
> e1000e: Support for byte queue limits
>
> Changes to e1000e to use byte queue limits.
>
> Signed-off-by: Tom Herbert <therbert@...gle.com>
> Acked-by: Eric Dumazet <eric.dumazet@...il.com>
> Signed-off-by: David S. Miller <davem@...emloft.net>
>
> If i reverse it, problem disappearing.
>
> How i reproduce it:
> In two consoles do "fast" ping to nearby host
> ping 194.146.XXX.XXX -s1472 -i0.0001
> ping 194.146.XXX.XXX -s1472 -i0.1
>
> For third open ssh to host with "problem", open mcedit, and just
> scroll down large text file.
> After few seconds some "stalls" will occur, and in ping history i can
> see:
> 1480 bytes from 194.146.153.7: icmp_req=1797 ttl=64 time=0.161 ms
> 1480 bytes from 194.146.153.7: icmp_req=1798 ttl=64 time=0.198 ms
> 1480 bytes from 194.146.153.7: icmp_req=1799 ttl=64 time=0.340 ms
> 1480 bytes from 194.146.153.7: icmp_req=1800 ttl=64 time=0.381 ms
> 1480 bytes from 194.146.153.7: icmp_req=1801 ttl=64 time=914 ms
> 1480 bytes from 194.146.153.7: icmp_req=1802 ttl=64 time=804 ms
> 1480 bytes from 194.146.153.7: icmp_req=1803 ttl=64 time=704 ms
> 1480 bytes from 194.146.153.7: icmp_req=1804 ttl=64 time=594 ms
> 1480 bytes from 194.146.153.7: icmp_req=1805 ttl=64 time=0.287 ms
> 1480 bytes from 194.146.153.7: icmp_req=1806 ttl=64 time=0.226 ms
>
>
> If i apply small patch - problem will disappear. Sure it is not a
> solution, but
> let me know how i can help to debug problem more.
>
> --- netdev.c 2012-05-12 20:08:37.000000000 +0300
> +++ netdev.c.patched 2012-05-17 16:32:28.895760472 +0300
> @@ -1135,7 +1135,7 @@
>
> tx_ring->next_to_clean = i;
>
> - netdev_completed_queue(netdev, pkts_compl, bytes_compl);
> +// netdev_completed_queue(netdev, pkts_compl, bytes_compl);
>
> #define TX_WAKE_THRESHOLD 32
> if (count && netif_carrier_ok(netdev) &&
> @@ -2263,7 +2263,7 @@
> e1000_put_txbuf(adapter, buffer_info);
> }
>
> - netdev_reset_queue(adapter->netdev);
> +// netdev_reset_queue(adapter->netdev);
> size = sizeof(struct e1000_buffer) * tx_ring->count;
> memset(tx_ring->buffer_info, 0, size);
>
> @@ -5056,7 +5056,7 @@
> /* if count is 0 then mapping error has occurred */
> count = e1000_tx_map(adapter, skb, first, max_per_txd,
> nr_frags, mss);
> if (count) {
> - netdev_sent_queue(netdev, skb->len);
> +// netdev_sent_queue(netdev, skb->len);
> e1000_tx_queue(adapter, tx_flags, count);
> /* Make sure there is space in the ring for the next
> send. */
> e1000_maybe_stop_tx(netdev, MAX_SKB_FRAGS + 2);
>
>
>
> On 2012-05-15 17:15, Denys Fedoryshchenko wrote:
>> Hi
>>
>> I have two identical servers, Sun Fire X4150, both has different
>> flavors of Linux, x86_64 and i386.
>> 04:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit
>> Ethernet Controller (Copper) (rev 01)
>> 04:00.1 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit
>> Ethernet Controller (Copper) (rev 01)
>> 0b:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit
>> Ethernet Controller (rev 06)
>> 0b:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit
>> Ethernet Controller (rev 06)
>> I am using now interface:
>> #ethtool -i eth0
>> driver: e1000e
>> version: 1.9.5-k
>> firmware-version: 2.1-11
>> bus-info: 0000:04:00.0
>> There is 2 CPU , Intel(R) Xeon(R) CPU E5440 @ 2.83GHz .
>>
>> i386 was acting as NAT and shaper, and as soon as i removed shaper
>> from it, i started to experience strange lockups, e.g. traffic is
>> normal for 5-30 seconds, then short lockup for 500-3000ms (usually
>> around 1000ms) with dropped packets counter increasing. I was
>> suspecting it is due load, but it seems was wrong.
>> Recently, on another server, x86_64 i am using as development, i
>> upgrade kernel (it was old, from 2.6 series) and on completely idle
>> machine started to experience same latency spikes, while i am just
>> running mc and for example typing in text editor - i notice
>> "stalls".
>> After i investigate it a little more, i notice also small amount of
>> drops on interface. No tcpdump running. Also this machine is idle,
>> and
>> the only traffic there - some small broadcasts from network, my ssh,
>> and ping.
>>
>> Dropped packets in ifconfig
>> RX packets:3752868 errors:0 dropped:5350 overruns:0
>> frame:0
>> Counter is increasing sometimes, when this stall happening.
>>
>> ethtool -S is clean, there is no dropped packets.
>>
>> I did tried to check load (mpstat and perf), there is nothing
>> suspicious, latencytop also doesn't show anything suspicious.
>> dropwatch report a lot of drops, but mostly because there is some
>> broadcasts and etc. tcpdump at the moment of such drops doesn't show
>> anything suspicious.
>> Changed qdisc from default fifo_fast to bfifo, without any result.
>> Tried: ethtool -K eth0 tso off gso off gro off sg off , no result
>> Problem occured at 3.3.6 - 3.4.0-rc7, most probably 3.3.0 also, but
>> i
>> don't remember for sure. I thik on some kernels like 3.1 probably it
>> doesn't occur, i will check it soon, because it is not always
>> reliable
>> to reproduce it. All tests i did on 3.4.0-rc7.
>>
>> I did run also in background tcpdump, additionally iptables with
>> timestamps, and at time when stall occured, seems i am still
>> receiving
>> packets properly, also on iperf udp (from some host to this
>> SunFire)
>> at this moments no packets missing. But i am sure RX interface
>> errors
>> are increasing.
>> If i do iperf from SunFire to test host - there is packetloss at
>> moments when stall occured.
>>
>> I suspect that by some reason network card stop to transmit, but
>> unable to pinpoint issue. All other hosts in this network are fine
>> and
>> don't have such problems.
>> Can you help me with that please? Maybe i can provide more debug
>> information, compile with patches and etc. Also i will try to
>> fallback
>> to 3.1 and 3.0 kernels.
>>
>> Here it is how it occurs and i am reproducing it:
>> I'm just opening file, and start to scroll it in mc, then in another
>> console i run ping
>> [1337089061.844167] 1480 bytes from 194.146.153.20: icmp_req=162
>> ttl=64 time=0.485 ms
>> [1337089061.944138] 1480 bytes from 194.146.153.20: icmp_req=163
>> ttl=64 time=0.470 ms
>> [1337089062.467759] 1480 bytes from 194.146.153.20: icmp_req=164
>> ttl=64 time=424 ms
>> [1337089062.467899] 1480 bytes from 194.146.153.20: icmp_req=165
>> ttl=64 time=324 ms
>> [1337089062.468058] 1480 bytes from 194.146.153.20: icmp_req=166
>> ttl=64 time=214 ms
>> [1337089062.468161] 1480 bytes from 194.146.153.20: icmp_req=167
>> ttl=64 time=104 ms
>> [1337089062.468958] 1480 bytes from 194.146.153.20: icmp_req=168
>> ttl=64 time=1.15 ms
>> [1337089062.568604] 1480 bytes from 194.146.153.20: icmp_req=169
>> ttl=64 time=0.477 ms
>> [1337089062.668909] 1480 bytes from 194.146.153.20: icmp_req=170
>> ttl=64 time=0.667 ms
>>
>> Remote host tcpdump:
>> 1337089061.934737 IP 194.146.153.20 > 194.146.153.22: ICMP echo
>> reply, id 3486, seq 163, length 1480
>> 1337089062.458360 IP 194.146.153.22 > 194.146.153.20: ICMP echo
>> request, id 3486, seq 164, length 1480
>> 1337089062.458380 IP 194.146.153.20 > 194.146.153.22: ICMP echo
>> reply, id 3486, seq 164, length 1480
>> 1337089062.458481 IP 194.146.153.22 > 194.146.153.20: ICMP echo
>> request, id 3486, seq 165, length 1480
>> 1337089062.458502 IP 194.146.153.20 > 194.146.153.22: ICMP echo
>> reply, id 3486, seq 165, length 1480
>> 1337089062.458606 IP 194.146.153.22 > 194.146.153.20: ICMP echo
>> request, id 3486, seq 166, length 1480
>> 1337089062.458623 IP 194.146.153.20 > 194.146.153.22: ICMP echo
>> reply, id 3486, seq 166, length 1480
>> 1337089062.458729 IP 194.146.153.22 > 194.146.153.20: ICMP echo
>> request, id 3486, seq 167, length 1480
>> 1337089062.458745 IP 194.146.153.20 > 194.146.153.22: ICMP echo
>> reply, id 3486, seq 167, length 1480
>> 1337089062.459537 IP 194.146.153.22 > 194.146.153.20: ICMP echo
>> request, id 3486, seq 168, length 1480
>> 1337089062.459545 IP 194.146.153.20 > 194.146.153.22: ICMP echo
>> reply, id 3486, seq 168, length 1480
>>
>> Local host(SunFire) tcpdump:
>> 1337089061.844140 IP 194.146.153.20 > 194.146.153.22: ICMP echo
>> reply, id 3486, seq 162, length 1480
>> 1337089061.943661 IP 194.146.153.22 > 194.146.153.20: ICMP echo
>> request, id 3486, seq 163, length 1480
>> 1337089061.944124 IP 194.146.153.20 > 194.146.153.22: ICMP echo
>> reply, id 3486, seq 163, length 1480
>> 1337089062.465622 IP 194.146.153.22 > 194.146.153.20: ICMP echo
>> request, id 3486, seq 164, length 1480
>> 1337089062.465630 IP 194.146.153.22 > 194.146.153.20: ICMP echo
>> request, id 3486, seq 165, length 1480
>> 1337089062.465632 IP 194.146.153.22 > 194.146.153.20: ICMP echo
>> request, id 3486, seq 166, length 1480
>> 1337089062.465634 IP 194.146.153.22 > 194.146.153.20: ICMP echo
>> request, id 3486, seq 167, length 1480
>> 1337089062.467730 IP 194.146.153.20 > 194.146.153.22: ICMP echo
>> reply, id 3486, seq 164, length 1480
>> 1337089062.467785 IP 194.146.153.22 > 194.146.153.20: ICMP echo
>> request, id 3486, seq 168, length 1480
>> 1337089062.467884 IP 194.146.153.20 > 194.146.153.22: ICMP echo
>> reply, id 3486, seq 165, length 1480
>> 1337089062.468035 IP 194.146.153.20 > 194.146.153.22: ICMP echo
>> reply, id 3486, seq 166, length 1480
>> 1337089062.468129 IP 194.146.153.20 > 194.146.153.22: ICMP echo
>> reply, id 3486, seq 167, length 1480
>> 1337089062.468928 IP 194.146.153.20 > 194.146.153.22: ICMP echo
>> reply, id 3486, seq 168, length 1480
>> 1337089062.568112 IP 194.146.153.22 > 194.146.153.20: ICMP echo
>> request, id 3486, seq 169, length 1480
>> 1337089062.568578 IP 194.146.153.20 > 194.146.153.22: ICMP echo
>> reply, id 3486, seq 169, length 1480
>>
>> lspci -t
>> centaur src # lspci -t
>> -[0000:00]-+-00.0
>> +-02.0-[01-05]--+-00.0-[02-04]--+-00.0-[03]--
>> | | \-02.0-[04]--+-00.0
>> | | \-00.1
>> | \-00.3-[05]--
>> +-03.0-[06]--
>> +-04.0-[07]----00.0
>> +-05.0-[08]--
>> +-06.0-[09]--
>> +-07.0-[0a]--
>> +-08.0
>> +-10.0
>> +-10.1
>> +-10.2
>> +-11.0
>> +-13.0
>> +-15.0
>> +-16.0
>> +-1c.0-[0b]--+-00.0
>> | \-00.1
>> +-1d.0
>> +-1d.1
>> +-1d.2
>> +-1d.3
>> +-1d.7
>> +-1e.0-[0c]----05.0
>> +-1f.0
>> +-1f.1
>> +-1f.2
>> \-1f.3
>> lspci
>> 00:00.0 Host bridge: Intel Corporation 5000P Chipset Memory
>> Controller Hub (rev b1)
>> 00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI
>> Express
>> x4 Port 2 (rev b1)
>> 00:03.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI
>> Express
>> x4 Port 3 (rev b1)
>> 00:04.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI
>> Express
>> x8 Port 4-5 (rev b1)
>> 00:05.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI
>> Express
>> x4 Port 5 (rev b1)
>> 00:06.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI
>> Express
>> x8 Port 6-7 (rev b1)
>> 00:07.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI
>> Express
>> x4 Port 7 (rev b1)
>> 00:08.0 System peripheral: Intel Corporation 5000 Series Chipset DMA
>> Engine (rev b1)
>> 00:10.0 Host bridge: Intel Corporation 5000 Series Chipset FSB
>> Registers (rev b1)
>> 00:10.1 Host bridge: Intel Corporation 5000 Series Chipset FSB
>> Registers (rev b1)
>> 00:10.2 Host bridge: Intel Corporation 5000 Series Chipset FSB
>> Registers (rev b1)
>> 00:11.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved
>> Registers (rev b1)
>> 00:13.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved
>> Registers (rev b1)
>> 00:15.0 Host bridge: Intel Corporation 5000 Series Chipset FBD
>> Registers (rev b1)
>> 00:16.0 Host bridge: Intel Corporation 5000 Series Chipset FBD
>> Registers (rev b1)
>> 00:1c.0 PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset
>> PCI Express Root Port 1 (rev 09)
>> 00:1d.0 USB controller: Intel Corporation 631xESB/632xESB/3100
>> Chipset UHCI USB Controller #1 (rev 09)
>> 00:1d.1 USB controller: Intel Corporation 631xESB/632xESB/3100
>> Chipset UHCI USB Controller #2 (rev 09)
>> 00:1d.2 USB controller: Intel Corporation 631xESB/632xESB/3100
>> Chipset UHCI USB Controller #3 (rev 09)
>> 00:1d.3 USB controller: Intel Corporation 631xESB/632xESB/3100
>> Chipset UHCI USB Controller #4 (rev 09)
>> 00:1d.7 USB controller: Intel Corporation 631xESB/632xESB/3100
>> Chipset EHCI USB2 Controller (rev 09)
>> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9)
>> 00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset
>> LPC Interface Controller (rev 09)
>> 00:1f.1 IDE interface: Intel Corporation 631xESB/632xESB IDE
>> Controller (rev 09)
>> 00:1f.2 SATA controller: Intel Corporation 631xESB/632xESB SATA AHCI
>> Controller (rev 09)
>> 00:1f.3 SMBus: Intel Corporation 631xESB/632xESB/3100 Chipset SMBus
>> Controller (rev 09)
>> 01:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
>> Upstream Port (rev 01)
>> 01:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to
>> PCI-X Bridge (rev 01)
>> 02:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
>> Downstream Port E1 (rev 01)
>> 02:02.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
>> Downstream Port E3 (rev 01)
>> 04:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit
>> Ethernet Controller (Copper) (rev 01)
>> 04:00.1 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit
>> Ethernet Controller (Copper) (rev 01)
>> 07:00.0 RAID bus controller: Adaptec AAC-RAID (rev 09)
>> 0b:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit
>> Ethernet Controller (rev 06)
>> 0b:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit
>> Ethernet Controller (rev 06)
>> 0c:05.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED
>> Graphics Family
>>
>>
>> dmesg:
>> [ 4.936885] e1000: Intel(R) PRO/1000 Network Driver - version
>> 7.3.21-k8-NAPI
>> [ 4.936887] e1000: Copyright (c) 1999-2006 Intel Corporation.
>> [ 4.936966] e1000e: Intel(R) PRO/1000 Network Driver - 1.9.5-k
>> [ 4.936967] e1000e: Copyright(c) 1999 - 2012 Intel Corporation.
>> [ 4.938529] e1000e 0000:04:00.0: (unregistered net_device):
>> Interrupt Throttling Rate (ints/sec) set to dynamic conservative
>> mode
>> [ 4.939598] e1000e 0000:04:00.0: irq 65 for MSI/MSI-X
>> [ 4.992246] e1000e 0000:04:00.0: eth0: (PCI Express:2.5GT/s:Width
>> x4) 00:1e:68:04:99:f8
>> [ 4.992657] e1000e 0000:04:00.0: eth0: Intel(R) PRO/1000 Network
>> Connection
>> [ 4.992964] e1000e 0000:04:00.0: eth0: MAC: 5, PHY: 5, PBA No:
>> FFFFFF-0FF
>> [ 4.994745] e1000e 0000:04:00.1: (unregistered net_device):
>> Interrupt Throttling Rate (ints/sec) set to dynamic conservative
>> mode
>> [ 4.996233] e1000e 0000:04:00.1: irq 66 for MSI/MSI-X
>> [ 5.050901] e1000e 0000:04:00.1: eth1: (PCI Express:2.5GT/s:Width
>> x4) 00:1e:68:04:99:f9
>> [ 5.051317] e1000e 0000:04:00.1: eth1: Intel(R) PRO/1000 Network
>> Connection
>> [ 5.051623] e1000e 0000:04:00.1: eth1: MAC: 5, PHY: 5, PBA No:
>> FFFFFF-0FF
>> [ 5.051857] e1000e 0000:0b:00.0: Disabling ASPM L1
>> [ 5.052168] e1000e 0000:0b:00.0: (unregistered net_device):
>> Interrupt Throttling Rate (ints/sec) set to dynamic conservative
>> mode
>> [ 5.052611] e1000e 0000:0b:00.0: irq 67 for MSI/MSI-X
>> [ 5.223454] e1000e 0000:0b:00.0: eth2: (PCI Express:2.5GT/s:Width
>> x4) 00:1e:68:04:99:fa
>> [ 5.223864] e1000e 0000:0b:00.0: eth2: Intel(R) PRO/1000 Network
>> Connection
>> [ 5.224178] e1000e 0000:0b:00.0: eth2: MAC: 0, PHY: 4, PBA No:
>> C83246-002
>> [ 5.224412] e1000e 0000:0b:00.1: Disabling ASPM L1
>> [ 5.224709] e1000e 0000:0b:00.1: (unregistered net_device):
>> Interrupt Throttling Rate (ints/sec) set to dynamic conservative
>> mode
>> [ 5.225168] e1000e 0000:0b:00.1: irq 68 for MSI/MSI-X
>> [ 5.397603] e1000e 0000:0b:00.1: eth3: (PCI Express:2.5GT/s:Width
>> x4) 00:1e:68:04:99:fb
>> [ 5.398021] e1000e 0000:0b:00.1: eth3: Intel(R) PRO/1000 Network
>> Connection
>> [ 5.398336] e1000e 0000:0b:00.1: eth3: MAC: 0, PHY: 4, PBA No:
>> C83246-002
>> [ 13.859817] e1000e 0000:04:00.0: irq 65 for MSI/MSI-X
>> [ 13.962309] e1000e 0000:04:00.0: irq 65 for MSI/MSI-X
>> [ 17.150392] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex,
>> Flow Control: None
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@...r.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> ---
> Network engineer
> Denys Fedoryshchenko
>
> Dora Highway - Center Cebaco - 2nd Floor
> Beirut, Lebanon
> Tel: +961 1 247373
> E-Mail: denys@...p.net.lb
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
---
Network engineer
Denys Fedoryshchenko
Dora Highway - Center Cebaco - 2nd Floor
Beirut, Lebanon
Tel: +961 1 247373
E-Mail: denys@...p.net.lb
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists