Date:	Wed, 18 Nov 2015 09:58:00 -0800
From:	Cong Wang <cwang@...pensource.com>
To:	arekm@...en.pl
Cc:	netdev <netdev@...r.kernel.org>
Subject: Re: lockdep problem? netlink_dump & rtnl_lock

On Wed, Nov 18, 2015 at 2:11 AM, Arkadiusz Miskiewicz
<a.miskiewicz@...il.com> wrote:
>
> Hi.
>
> I'm running 4.3.0 kernel (+ few backported mm patches). Machine is doing many rsnapshots
> in parallel (so heavily loaded).
>
> I'm seeing this from time to time (below). What would be the reason for so long hangs?


It is not a lockdep problem, it is that someone was holding that mutex
for a long time or that mutex was not released correctly.

Can you do a sysrq-t when you reproduce this problem?

Thanks!

>
> [ 2278.385668] INFO: task snmpd:1775 blocked for more than 120 seconds.
> [ 2278.392089]       Not tainted 4.3.0-0.1 #1
> [ 2278.396226] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 2278.404111] snmpd           D ffff88081fc158c0     0  1775      1 0x00000000
> [ 2278.411229]  ffff8807e6a8bcf8 0000000000000082 ffffffff81c13500 ffff8807ebb88d80
> [ 2278.418753]  ffff8807e6a8c000 ffff8807ebb88d80 ffffffff81cc7c84 00000000ffffffff
> [ 2278.426286]  ffffffff81cc7c88 ffff8807e6a8bd10 ffffffff8161c203 ffffffff81cc7c80
> [ 2278.433854] Call Trace:
> [ 2278.436341]  [<ffffffff8161c203>] schedule+0x33/0x80
> [ 2278.441349]  [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10
> [ 2278.447733]  [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120
> [ 2278.453956]  [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30
> [ 2278.459133]  [<ffffffff81523f15>] rtnl_lock+0x15/0x20
> [ 2278.464212]  [<ffffffff8158b902>] devinet_ioctl+0xd2/0x6e0
> [ 2278.469757]  [<ffffffff81208c14>] ? mntput+0x24/0x40
> [ 2278.474769]  [<ffffffff811ebe62>] ? __fput+0x142/0x200
> [ 2278.479955]  [<ffffffff8158d5a0>] inet_ioctl+0x50/0x70
> [ 2278.485122]  [<ffffffff814f6e05>] sock_do_ioctl+0x25/0x50
> [ 2278.490578]  [<ffffffff814f7322>] sock_ioctl+0x1d2/0x290
> [ 2278.495920]  [<ffffffff811fc3f5>] do_vfs_ioctl+0x295/0x470
> [ 2278.501444]  [<ffffffff814f6cd6>] ? sock_alloc_file+0xa6/0x130
> [ 2278.507338]  [<ffffffff81090eb4>] ? task_work_add+0x44/0x60
> [ 2278.512942]  [<ffffffff811ebff7>] ? fput+0x47/0x90
> [ 2278.517771]  [<ffffffff811fc649>] SyS_ioctl+0x79/0x90
> [ 2278.522845]  [<ffffffff8161f876>] entry_SYSCALL_64_fastpath+0x16/0x75
> [ 2278.529341] INFO: task sudo:4534 blocked for more than 120 seconds.
> [ 2278.535635]       Not tainted 4.3.0-0.1 #1
> [ 2278.539759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 2278.547680] sudo            D 0000000000000000     0  4534      1 0x00000004
> [ 2278.554825]  ffff880080677bb8 0000000000000086 ffff8807f3874380 ffff8807f3eb8000
> [ 2278.562389]  ffff880080678000 ffff8807f3eb8000 ffffffff81cc7c84 00000000ffffffff
> [ 2278.569914]  ffffffff81cc7c88 ffff880080677bd0 ffffffff8161c203 ffffffff81cc7c80
> [ 2278.577436] Call Trace:
> [ 2278.579911]  [<ffffffff8161c203>] schedule+0x33/0x80
> [ 2278.584918]  [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10
> [ 2278.591329]  [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120
> [ 2278.597571]  [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30
> [ 2278.602743]  [<ffffffff8154746e>] __netlink_dump_start+0x5e/0x190
> [ 2278.608890]  [<ffffffff81526100>] ? rtnl_fill_ifinfo+0x13f0/0x13f0
> [ 2278.615105]  [<ffffffff81526eac>] rtnetlink_rcv_msg+0x1bc/0x230
> [ 2278.621054]  [<ffffffff81526100>] ? rtnl_fill_ifinfo+0x13f0/0x13f0
> [ 2278.627262]  [<ffffffff81526cf0>] ? rtnetlink_rcv+0x30/0x30
> [ 2278.632861]  [<ffffffff815498d4>] netlink_rcv_skb+0xa4/0xc0
> [ 2278.638463]  [<ffffffff81526ce8>] rtnetlink_rcv+0x28/0x30
> [ 2278.643908]  [<ffffffff81548fc7>] netlink_unicast+0x127/0x1a0
> [ 2278.649692]  [<ffffffff81549604>] netlink_sendmsg+0x4f4/0x5f0
> [ 2278.655480]  [<ffffffff814f8bd8>] sock_sendmsg+0x38/0x50
> [ 2278.660830]  [<ffffffff814fa2a1>] SyS_sendto+0x101/0x150
> [ 2278.666189]  [<ffffffff814f6cd6>] ? sock_alloc_file+0xa6/0x130
> [ 2278.672064]  [<ffffffff81206715>] ? fd_install+0x25/0x30
> [ 2278.677407]  [<ffffffff814f985f>] ? SyS_socket+0x9f/0xf0
> [ 2278.682749]  [<ffffffff8161f876>] entry_SYSCALL_64_fastpath+0x16/0x75
>
>
>
>
>
>
>
>
>  [ 7433.492510] INFO: task bird:1503 blocked for more than 120 seconds.
> [ 7433.498819]       Not tainted 4.3.0-0.1 #1
> [ 7433.502954] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 7433.510840] bird            D ffff88081fd158c0     0  1503      1 0x00000000
> [ 7433.517966]  ffff8807eb923bb8 0000000000000086 ffff8807fb711b00 ffff8807f38a4380
> [ 7433.525493]  ffff8807eb924000 ffff8807f38a4380 ffffffff81cc7c84 00000000ffffffff
> [ 7433.533017]  ffffffff81cc7c88 ffff8807eb923bd0 ffffffff8161c203 ffffffff81cc7c80
> [ 7433.540561] Call Trace:
> [ 7433.543050]  [<ffffffff8161c203>] schedule+0x33/0x80
> [ 7433.548041]  [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10
> [ 7433.554415]  [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120
> [ 7433.560624]  [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30
> [ 7433.565810]  [<ffffffff81546be1>] netlink_dump+0x21/0x2c0
> [ 7433.571236]  [<ffffffff815470c9>] netlink_recvmsg+0x249/0x3d0
> [ 7433.577005]  [<ffffffff814f8eeb>] sock_recvmsg+0x3b/0x50
> [ 7433.582347]  [<ffffffff814f96ab>] ___sys_recvmsg+0xdb/0x1f0
> [ 7433.587952]  [<ffffffff810146ec>] ? __switch_to+0x25c/0x4a0
> [ 7433.593579]  [<ffffffff810a6f17>] ? update_curr+0xb7/0x130
> [ 7433.599115]  [<ffffffff810a354c>] ? __enqueue_entity+0x6c/0x70
> [ 7433.604970]  [<ffffffff810a44f1>] ? set_next_entity+0x71/0x840
> [ 7433.610832]  [<ffffffff810146ec>] ? __switch_to+0x25c/0x4a0
> [ 7433.616431]  [<ffffffff814fa9b1>] __sys_recvmsg+0x51/0x90
> [ 7433.621873]  [<ffffffff814faa02>] SyS_recvmsg+0x12/0x20
> [ 7433.627125]  [<ffffffff8161f876>] entry_SYSCALL_64_fastpath+0x16/0x75
> [ 7433.633642] INFO: task kworker/3:4:12895 blocked for more than 120 seconds.
> [ 7433.640651]       Not tainted 4.3.0-0.1 #1
> [ 7433.644797] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 7433.652682] kworker/3:4     D 0000000000000000     0 12895      2 0x00000000
> [ 7433.659794] Workqueue: ipv6_addrconf addrconf_verify_work
> [ 7433.665240]  ffff8803b0073d58 0000000000000046 ffff880781adb600 ffff880736aa3600
> [ 7433.672763]  ffff8803b0074000 ffff880736aa3600 ffffffff81cc7c84 00000000ffffffff
> [ 7433.680287]  ffffffff81cc7c88 ffff8803b0073d70 ffffffff8161c203 ffffffff81cc7c80
> [ 7433.687811] Call Trace:
> [ 7433.690288]  [<ffffffff8161c203>] schedule+0x33/0x80
> [ 7433.695297]  [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10
> [ 7433.701688]  [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120
> [ 7433.707923]  [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30
> [ 7433.713093]  [<ffffffff81523f15>] rtnl_lock+0x15/0x20
> [ 7433.718168]  [<ffffffff815c903e>] addrconf_verify_work+0xe/0x20
> [ 7433.724114]  [<ffffffff8108cc6f>] process_one_work+0x14f/0x430
> [ 7433.730006]  [<ffffffff8108cf9b>] worker_thread+0x4b/0x4b0
> [ 7433.735517]  [<ffffffff8161bb71>] ? __schedule+0x2a1/0x900
> [ 7433.741031]  [<ffffffff8108cf50>] ? process_one_work+0x430/0x430
> [ 7433.747083]  [<ffffffff8108cf50>] ? process_one_work+0x430/0x430
> [ 7433.753131]  [<ffffffff81092be8>] kthread+0xd8/0xf0
> [ 7433.758063]  [<ffffffff81092b10>] ? kthread_worker_fn+0x170/0x170
> [ 7433.764204]  [<ffffffff8161fc0f>] ret_from_fork+0x3f/0x70
> [ 7433.769657]  [<ffffffff81092b10>] ? kthread_worker_fn+0x170/0x170
>
> config:
> http://sprunge.us/BBLN
>
> # ethtool -i eth0
> driver: igb
> version: 5.3.0-k
> firmware-version: 3.16, 0x800004d7
> bus-info: 0000:05:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
> # lspci -v
> 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
>         Subsystem: Super Micro Computer Inc Device 0804
>         Flags: bus master, fast devsel, latency 0
>         Capabilities: [e0] Vendor Specific Information: Len=0c <?>
>         Kernel driver in use: ie31200_edac
>         Kernel modules: ie31200_edac
>
> 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>         Capabilities: [88] Subsystem: Super Micro Computer Inc Device 0804
>         Capabilities: [80] Power Management version 3
>         Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
>         Capabilities: [a0] Express Root Port (Slot+), MSI 00
>         Capabilities: [100] Virtual Channel
>         Capabilities: [140] Root Complex Link
>         Capabilities: [d94] #19
>         Kernel driver in use: pcieport
>         Kernel modules: shpchp
>
> 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
>         I/O behind bridge: 0000e000-0000efff
>         Memory behind bridge: f7100000-f72fffff
>         Capabilities: [88] Subsystem: Super Micro Computer Inc Device 0804
>         Capabilities: [80] Power Management version 3
>         Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
>         Capabilities: [a0] Express Root Port (Slot+), MSI 00
>         Capabilities: [100] Virtual Channel
>         Capabilities: [140] Root Complex Link
>         Capabilities: [d94] #19
>         Kernel driver in use: pcieport
>         Kernel modules: shpchp
>
> 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) (prog-if 30 [XHCI])
>         Subsystem: Super Micro Computer Inc Device 0804
>         Flags: bus master, medium devsel, latency 0, IRQ 34
>         Memory at f7500000 (64-bit, non-prefetchable) [size=64K]
>         Capabilities: [70] Power Management version 2
>         Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
>         Kernel driver in use: xhci_hcd
>         Kernel modules: xhci_pci
>
> 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) (prog-if 20 [EHCI])
>         Subsystem: Super Micro Computer Inc Device 0804
>         Flags: bus master, medium devsel, latency 0, IRQ 16
>         Memory at f7513000 (32-bit, non-prefetchable) [size=1K]
>         Capabilities: [50] Power Management version 2
>         Capabilities: [58] Debug port: BAR=1 offset=00a0
>         Capabilities: [98] PCI Advanced Features
>         Kernel driver in use: ehci-pci
>         Kernel modules: ehci_pci
>
> 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=00, secondary=03, subordinate=04, sec-latency=0
>         I/O behind bridge: 0000d000-0000dfff
>         Memory behind bridge: f6000000-f70fffff
>         Capabilities: [40] Express Root Port (Slot+), MSI 00
>         Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
>         Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0804
>         Capabilities: [a0] Power Management version 3
>         Kernel driver in use: pcieport
>         Kernel modules: shpchp
>
> 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
>         I/O behind bridge: 0000c000-0000cfff
>         Memory behind bridge: f7400000-f74fffff
>         Capabilities: [40] Express Root Port (Slot+), MSI 00
>         Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
>         Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0804
>         Capabilities: [a0] Power Management version 3
>         Kernel driver in use: pcieport
>         Kernel modules: shpchp
>
> 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
>         I/O behind bridge: 0000b000-0000bfff
>         Memory behind bridge: f7300000-f73fffff
>         Capabilities: [40] Express Root Port (Slot+), MSI 00
>         Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
>         Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0804
>         Capabilities: [a0] Power Management version 3
>         Kernel driver in use: pcieport
>         Kernel modules: shpchp
>
> 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) (prog-if 20 [EHCI])
>         Subsystem: Super Micro Computer Inc Device 0804
>         Flags: bus master, medium devsel, latency 0, IRQ 23
>         Memory at f7512000 (32-bit, non-prefetchable) [size=1K]
>         Capabilities: [50] Power Management version 2
>         Capabilities: [58] Debug port: BAR=1 offset=00a0
>         Capabilities: [98] PCI Advanced Features
>         Kernel driver in use: ehci-pci
>         Kernel modules: ehci_pci
>
> 00:1f.0 ISA bridge: Intel Corporation C222 Series Chipset Family Server Essential SKU LPC Controller (rev 05)
>         Subsystem: Super Micro Computer Inc Device 0804
>         Flags: bus master, medium devsel, latency 0
>         Capabilities: [e0] Vendor Specific Information: Len=0c <?>
>         Kernel driver in use: lpc_ich
>         Kernel modules: lpc_ich
>
> 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) (prog-if 01 [AHCI 1.0])
>         Subsystem: Super Micro Computer Inc Device 0804
>         Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 29
>         I/O ports at f050 [size=8]
>         I/O ports at f040 [size=4]
>         I/O ports at f030 [size=8]
>         I/O ports at f020 [size=4]
>         I/O ports at f000 [size=32]
>         Memory at f7511000 (32-bit, non-prefetchable) [size=2K]
>         Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
>         Capabilities: [70] Power Management version 3
>         Capabilities: [a8] SATA HBA v1.0
>         Kernel driver in use: ahci
>         Kernel modules: ahci
>
> 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
>         Subsystem: Super Micro Computer Inc Device 0804
>         Flags: medium devsel, IRQ 18
>         Memory at f7510000 (64-bit, non-prefetchable) [size=256]
>         I/O ports at 0580 [size=32]
>         Kernel driver in use: i801_smbus
>         Kernel modules: i2c_i801
>
> 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
>         Subsystem: Super Micro Computer Inc Device 0804
>         Flags: bus master, fast devsel, latency 0, IRQ 10
>         Memory at dfff00000 (64-bit, non-prefetchable) [size=4K]
>         Capabilities: [50] Power Management version 3
>         Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
>
> 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
>         Subsystem: Super Micro Computer Inc Device 0691
>         Flags: bus master, fast devsel, latency 0
>         I/O ports at e000 [size=256]
>         Memory at f7240000 (64-bit, non-prefetchable) [size=64K]
>         Memory at f7200000 (64-bit, non-prefetchable) [size=256K]
>         Expansion ROM at f7100000 [disabled] [size=1M]
>         Capabilities: [50] Power Management version 3
>         Capabilities: [68] Express Endpoint, MSI 00
>         Capabilities: [d0] Vital Product Data
>         Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
>         Capabilities: [c0] MSI-X: Enable+ Count=16 Masked-
>         Capabilities: [100] Advanced Error Reporting
>         Capabilities: [1e0] #19
>         Capabilities: [1c0] Power Budgeting <?>
>         Capabilities: [190] #16
>         Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
>         Kernel driver in use: mpt2sas
>         Kernel modules: mpt2sas
>
> 03:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03) (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=03, secondary=04, subordinate=04, sec-latency=0
>         I/O behind bridge: 0000d000-0000dfff
>         Memory behind bridge: f6000000-f70fffff
>         Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
>         Capabilities: [78] Power Management version 3
>         Capabilities: [80] Express PCI-Express to PCI/PCI-X Bridge, MSI 00
>         Capabilities: [a4] Subsystem: Super Micro Computer Inc Device 0804
>         Capabilities: [100] Virtual Channel
>         Kernel modules: shpchp
>
> 04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30) (prog-if 00 [VGA controller])
>         Subsystem: Super Micro Computer Inc Device 0804
>         Flags: bus master, medium devsel, latency 0, IRQ 11
>         Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
>         Memory at f7000000 (32-bit, non-prefetchable) [size=128K]
>         I/O ports at d000 [size=128]
>         Expansion ROM at <unassigned> [disabled]
>         Capabilities: [40] Power Management version 3
>         Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+
>
> 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
>         Subsystem: Super Micro Computer Inc Device 1533
>         Flags: bus master, fast devsel, latency 0
>         Memory at f7400000 (32-bit, non-prefetchable) [size=512K]
>         I/O ports at c000 [size=32]
>         Memory at f7480000 (32-bit, non-prefetchable) [size=16K]
>         Capabilities: [40] Power Management version 3
>         Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
>         Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
>         Capabilities: [a0] Express Endpoint, MSI 00
>         Capabilities: [100] Advanced Error Reporting
>         Capabilities: [140] Device Serial Number 00-25-90-ff-ff-87-5d-0e
>         Capabilities: [1a0] Transaction Processing Hints
>         Kernel driver in use: igb
>         Kernel modules: igb
>
> 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
>         Subsystem: Super Micro Computer Inc Device 1533
>         Flags: bus master, fast devsel, latency 0
>         Memory at f7300000 (32-bit, non-prefetchable) [size=512K]
>         I/O ports at b000 [size=32]
>         Memory at f7380000 (32-bit, non-prefetchable) [size=16K]
>         Capabilities: [40] Power Management version 3
>         Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
>         Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
>         Capabilities: [a0] Express Endpoint, MSI 00
>         Capabilities: [100] Advanced Error Reporting
>         Capabilities: [140] Device Serial Number 00-25-90-ff-ff-87-5d-0f
>         Capabilities: [1a0] Transaction Processing Hints
>         Kernel driver in use: igb
>         Kernel modules: igb
>
>
> --
> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists