| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Tue, 11 Nov 2014 17:12:51 +0800 From: Wei Yang <weiyang@...ux.vnet.ibm.com> To: Amir Vadai <amirv@...lanox.com> Cc: Wei Yang <weiyang@...ux.vnet.ibm.com>, Eric Dumazet <eric.dumazet@...il.com>, Eric Dumazet <edumazet@...gle.com>, David Miller <davem@...emloft.net>, netdev <netdev@...r.kernel.org>, gerlitz.or@...il.com Subject: Re: Face some error after applying commit 7dfa4b414d4(net/mlx4_en: Code cleanups in tx path) On Tue, Nov 11, 2014 at 10:40:24AM +0200, Amir Vadai wrote: >On Tue, Nov 11, 2014 at 9:42 AM, Wei Yang <weiyang@...ux.vnet.ibm.com> wrote: >>>>>Hi, >>>>> >>>>>Lets see that we're on the same page here: >>>>>1. There was a compilation problem that you fixed (Yes, it was my fault >>>>>- I just discovered it a minute after the code was applied). >>>>>2. When you're using SR-IOV, during initialization, you get a CQE error >>>>>with syndrome 0x2 on one of the probed VF's. >>>> >>>> From the log, seems yes. >>>> >>>>>3. Regarding the BlueFlame - I don't see how it is related to the issue >>>>>that you see. But it is a very easy experiment. Issue: "ethtool >>>>>--set-priv-flags eth1 blueflame off" >>>> >>>> I tried to use this after mlx4_en is loaded, still see the CQE error. >>>> >>>>> >>>>>Please send me the module parameters you used when loading mlx4_core, a >>>>>full dmesg with both mlx4_core and mlx4_en loading. >>>> >>>> The command line I use is: >>>> modprobe mlx4_core num_vfs=1 probe_vf=1 port_type_array=2,2 >>>> >>>> The log I sent in the first mail is the full log, including the CQE error, one >>>> warning in watchdog, and then print the CQE error periodicly. What else >>>> message you would like me to capture? >>> >>>The log in the first mail has only mlx4_en logs. I would like to see >>>the full log, that has mlx4_core messages too. And as Or suggested, >>>debug_level=1 could be useful here too. >>> >> >> Ah, you need the log from mlx4_core too. Ok, I will do it again. >> >> BTW, how to add the debug_level=1 in the command line? Like this? >> >> modprobe mlx4_core num_vfs=1 probe_vf=1 port_type_array=2,2 debug_level=1 >yes > I past the full log here, but I don't see the log level affact. [ 5723.612882] mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014) [ 5723.612979] mlx4_core: Initializing 0003:05:00.0 [ 5723.613023] pci 0003:05: 0.1: [PE# 222] VF 0003:05:00.1 associated with PE#222 [ 5723.613299] pci 0003:05: 0.1: [PE# 222] Setting up 32-bit TCE table at 0..80000000 [ 5723.684140] pci 0003:05: 0.1: [PE# 222] Enabling 64-bit DMA bypass [ 5723.684215] pci 0003:05: 0.2: [PE# 223] VF 0003:05:00.2 associated with PE#223 [ 5723.684409] pci 0003:05: 0.2: [PE# 223] Setting up 32-bit TCE table at 0..80000000 [ 5723.754871] pci 0003:05: 0.2: [PE# 223] Enabling 64-bit DMA bypass [ 5723.754944] pci 0003:05: 0.3: [PE# 224] VF 0003:05:00.3 associated with PE#224 [ 5723.755150] pci 0003:05: 0.3: [PE# 224] Setting up 32-bit TCE table at 0..80000000 [ 5723.825646] pci 0003:05: 0.3: [PE# 224] Enabling 64-bit DMA bypass [ 5723.825718] pci 0003:05: 0.4: [PE# 225] VF 0003:05:00.4 associated with PE#225 [ 5723.825911] pci 0003:05: 0.4: [PE# 225] Setting up 32-bit TCE table at 0..80000000 [ 5723.896342] pci 0003:05: 0.4: [PE# 225] Enabling 64-bit DMA bypass [ 5723.896416] pci 0003:05: 0.5: [PE# 226] VF 0003:05:00.5 associated with PE#226 [ 5723.896610] pci 0003:05: 0.5: [PE# 226] Setting up 32-bit TCE table at 0..80000000 [ 5723.967052] pci 0003:05: 0.5: [PE# 226] Enabling 64-bit DMA bypass [ 5723.967135] pci 0003:05: 0.6: [PE# 227] VF 0003:05:00.6 associated with PE#227 [ 5723.967328] pci 0003:05: 0.6: [PE# 227] Setting up 32-bit TCE table at 0..80000000 [ 5724.037755] pci 0003:05: 0.6: [PE# 227] Enabling 64-bit DMA bypass [ 5724.037829] pci 0003:05: 0.7: [PE# 228] VF 0003:05:00.7 associated with PE#228 [ 5724.038022] pci 0003:05: 0.7: [PE# 228] Setting up 32-bit TCE table at 0..80000000 [ 5724.108452] pci 0003:05: 0.7: [PE# 228] Enabling 64-bit DMA bypass [ 5724.108525] pci 0003:05: 1.0: [PE# 229] VF 0003:05:01.0 associated with PE#229 [ 5724.108718] pci 0003:05: 1.0: [PE# 229] Setting up 32-bit TCE table at 0..80000000 [ 5724.179209] pci 0003:05: 1.0: [PE# 229] Enabling 64-bit DMA bypass [ 5724.179293] pci 0003:05: 1.1: [PE# 230] VF 0003:05:01.1 associated with PE#230 [ 5724.179486] pci 0003:05: 1.1: [PE# 230] Setting up 32-bit TCE table at 0..80000000 [ 5724.249946] pci 0003:05: 1.1: [PE# 230] Enabling 64-bit DMA bypass [ 5724.250020] pci 0003:05: 1.2: [PE# 231] VF 0003:05:01.2 associated with PE#231 [ 5724.250214] pci 0003:05: 1.2: [PE# 231] Setting up 32-bit TCE table at 0..80000000 [ 5724.320703] pci 0003:05: 1.2: [PE# 231] Enabling 64-bit DMA bypass [ 5724.320787] pci 0003:05: 1.3: [PE# 232] VF 0003:05:01.3 associated with PE#232 [ 5724.320982] pci 0003:05: 1.3: [PE# 232] Setting up 32-bit TCE table at 0..80000000 [ 5724.391424] pci 0003:05: 1.3: [PE# 232] Enabling 64-bit DMA bypass [ 5724.391497] pci 0003:05: 1.4: [PE# 233] VF 0003:05:01.4 associated with PE#233 [ 5724.391690] pci 0003:05: 1.4: [PE# 233] Setting up 32-bit TCE table at 0..80000000 [ 5724.462129] pci 0003:05: 1.4: [PE# 233] Enabling 64-bit DMA bypass [ 5724.462215] pci 0003:05: 1.5: [PE# 234] VF 0003:05:01.5 associated with PE#234 [ 5724.462408] pci 0003:05: 1.5: [PE# 234] Setting up 32-bit TCE table at 0..80000000 [ 5724.532834] pci 0003:05: 1.5: [PE# 234] Enabling 64-bit DMA bypass [ 5724.532907] pci 0003:05: 1.6: [PE# 235] VF 0003:05:01.6 associated with PE#235 [ 5724.533100] pci 0003:05: 1.6: [PE# 235] Setting up 32-bit TCE table at 0..80000000 [ 5724.603556] pci 0003:05: 1.6: [PE# 235] Enabling 64-bit DMA bypass [ 5724.603628] pci 0003:05: 1.7: [PE# 236] VF 0003:05:01.7 associated with PE#236 [ 5724.603822] pci 0003:05: 1.7: [PE# 236] Setting up 32-bit TCE table at 0..80000000 [ 5724.674261] pci 0003:05: 1.7: [PE# 236] Enabling 64-bit DMA bypass [ 5724.674334] pci 0003:05: 2.0: [PE# 237] VF 0003:05:02.0 associated with PE#237 [ 5724.674528] pci 0003:05: 2.0: [PE# 237] Setting up 32-bit TCE table at 0..80000000 [ 5724.745012] pci 0003:05: 2.0: [PE# 237] Enabling 64-bit DMA bypass [ 5724.745088] pci 0003:05: 2.1: [PE# 238] VF 0003:05:02.1 associated with PE#238 [ 5724.745281] pci 0003:05: 2.1: [PE# 238] Setting up 32-bit TCE table at 0..80000000 [ 5724.815714] pci 0003:05: 2.1: [PE# 238] Enabling 64-bit DMA bypass [ 5724.815786] pci 0003:05: 2.2: [PE# 239] VF 0003:05:02.2 associated with PE#239 [ 5724.815979] pci 0003:05: 2.2: [PE# 239] Setting up 32-bit TCE table at 0..80000000 [ 5724.886413] pci 0003:05: 2.2: [PE# 239] Enabling 64-bit DMA bypass [ 5724.886487] pci 0003:05: 2.3: [PE# 240] VF 0003:05:02.3 associated with PE#240 [ 5724.886680] pci 0003:05: 2.3: [PE# 240] Setting up 32-bit TCE table at 0..80000000 [ 5724.957126] pci 0003:05: 2.3: [PE# 240] Enabling 64-bit DMA bypass [ 5724.957210] pci 0003:05: 2.4: [PE# 241] VF 0003:05:02.4 associated with PE#241 [ 5724.957403] pci 0003:05: 2.4: [PE# 241] Setting up 32-bit TCE table at 0..80000000 [ 5725.027842] pci 0003:05: 2.4: [PE# 241] Enabling 64-bit DMA bypass [ 5725.027917] pci 0003:05: 2.5: [PE# 242] VF 0003:05:02.5 associated with PE#242 [ 5725.028109] pci 0003:05: 2.5: [PE# 242] Setting up 32-bit TCE table at 0..80000000 [ 5725.098551] pci 0003:05: 2.5: [PE# 242] Enabling 64-bit DMA bypass [ 5725.098624] pci 0003:05: 2.6: [PE# 243] VF 0003:05:02.6 associated with PE#243 [ 5725.098818] pci 0003:05: 2.6: [PE# 243] Setting up 32-bit TCE table at 0..80000000 [ 5725.169255] pci 0003:05: 2.6: [PE# 243] Enabling 64-bit DMA bypass [ 5725.169338] pci 0003:05: 2.7: [PE# 244] VF 0003:05:02.7 associated with PE#244 [ 5725.169532] pci 0003:05: 2.7: [PE# 244] Setting up 32-bit TCE table at 0..80000000 [ 5725.239962] pci 0003:05: 2.7: [PE# 244] Enabling 64-bit DMA bypass [ 5725.240036] pci 0003:05: 3.0: [PE# 245] VF 0003:05:03.0 associated with PE#245 [ 5725.240229] pci 0003:05: 3.0: [PE# 245] Setting up 32-bit TCE table at 0..80000000 [ 5725.310678] pci 0003:05: 3.0: [PE# 245] Enabling 64-bit DMA bypass [ 5725.310751] pci 0003:05: 3.1: [PE# 246] VF 0003:05:03.1 associated with PE#246 [ 5725.310944] pci 0003:05: 3.1: [PE# 246] Setting up 32-bit TCE table at 0..80000000 [ 5725.381375] pci 0003:05: 3.1: [PE# 246] Enabling 64-bit DMA bypass [ 5725.381459] pci 0003:05: 3.2: [PE# 247] VF 0003:05:03.2 associated with PE#247 [ 5725.381653] pci 0003:05: 3.2: [PE# 247] Setting up 32-bit TCE table at 0..80000000 [ 5725.452087] pci 0003:05: 3.2: [PE# 247] Enabling 64-bit DMA bypass [ 5725.452171] pci 0003:05: 3.3: [PE# 248] VF 0003:05:03.3 associated with PE#248 [ 5725.452365] pci 0003:05: 3.3: [PE# 248] Setting up 32-bit TCE table at 0..80000000 [ 5725.522815] pci 0003:05: 3.3: [PE# 248] Enabling 64-bit DMA bypass [ 5725.522889] pci 0003:05: 3.4: [PE# 249] VF 0003:05:03.4 associated with PE#249 [ 5725.523082] pci 0003:05: 3.4: [PE# 249] Setting up 32-bit TCE table at 0..80000000 [ 5725.593516] pci 0003:05: 3.4: [PE# 249] Enabling 64-bit DMA bypass [ 5725.593590] pci 0003:05: 3.5: [PE# 250] VF 0003:05:03.5 associated with PE#250 [ 5725.593783] pci 0003:05: 3.5: [PE# 250] Setting up 32-bit TCE table at 0..80000000 [ 5725.664330] pci 0003:05: 3.5: [PE# 250] Enabling 64-bit DMA bypass [ 5725.664403] pci 0003:05: 3.6: [PE# 251] VF 0003:05:03.6 associated with PE#251 [ 5725.664597] pci 0003:05: 3.6: [PE# 251] Setting up 32-bit TCE table at 0..80000000 [ 5725.735081] pci 0003:05: 3.6: [PE# 251] Enabling 64-bit DMA bypass [ 5725.735154] pci 0003:05: 3.7: [PE# 252] VF 0003:05:03.7 associated with PE#252 [ 5725.735348] pci 0003:05: 3.7: [PE# 252] Setting up 32-bit TCE table at 0..80000000 [ 5725.805792] pci 0003:05: 3.7: [PE# 252] Enabling 64-bit DMA bypass [ 5725.805865] pci 0003:05: 4.0: [PE# 253] VF 0003:05:04.0 associated with PE#253 [ 5725.806058] pci 0003:05: 4.0: [PE# 253] Setting up 32-bit TCE table at 0..80000000 [ 5725.876496] pci 0003:05: 4.0: [PE# 253] Enabling 64-bit DMA bypass [ 5725.876570] pci 0003:05: 4.1: [PE# 254] VF 0003:05:04.1 associated with PE#254 [ 5725.876763] pci 0003:05: 4.1: [PE# 254] Setting up 32-bit TCE table at 0..80000000 [ 5725.945133] pci 0003:05: 4.1: [PE# 254] Enabling 64-bit DMA bypass [ 5725.945212] pci 0003:05: 4.2: [PE# 255] VF 0003:05:04.2 associated with PE#255 [ 5725.945316] pci 0003:05: 4.2: [PE# 255] Setting up 32-bit TCE table at 0..80000000 [ 5726.006975] pci 0003:05: 4.2: [PE# 255] Enabling 64-bit DMA bypass [ 5726.007163] mlx4_core 0003:05:00.0: Using 64-bit DMA iommu bypass [ 5726.007227] mlx4_core 0003:05:00.0: Enabling SR-IOV with 1 VFs [ 5726.116742] mlx4_core: Initializing 0003:05:00.1 [ 5726.116825] mlx4_core 0003:05:00.1: enabling device (0000 -> 0002) [ 5726.116914] mlx4_core 0003:05:00.1: Using 64-bit DMA iommu bypass [ 5726.116976] mlx4_core 0003:05:00.1: Detected virtual function - running in slave mode [ 5726.117191] mlx4_core 0003:05:00.1: PF is not ready - Deferring probe [ 5726.117276] pci 0003:05:00.1: Driver mlx4_core requests probe deferral [ 5726.117363] mlx4_core 0003:05:00.0: Running in master mode [ 5731.123148] mlx4_core 0003:05:00.0: PCIe link speed is 8.0GT/s, device supports 8.0GT/s [ 5731.123227] mlx4_core 0003:05:00.0: PCIe link width is x8, device supports x8 [ 5731.151307] mlx4_core: Initializing 0003:05:00.1 [ 5731.151389] mlx4_core 0003:05:00.1: enabling device (0000 -> 0002) [ 5731.151501] mlx4_core 0003:05:00.1: Using 64-bit DMA iommu bypass [ 5731.151559] mlx4_core 0003:05:00.1: Detected virtual function - running in slave mode [ 5731.151636] mlx4_core 0003:05:00.1: Sending reset [ 5731.151770] mlx4_core 0003:05:00.0: Received reset from slave:1 [ 5731.152189] mlx4_core 0003:05:00.1: Sending vhcr0 [ 5731.152991] mlx4_core 0003:05:00.1: HCA minimum page size:512 [ 5731.153444] mlx4_core 0003:05:00.1: Timestamping is not supported in slave mode [root@...n-lp1 ywywyang]# [ 5731.229958] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014) [ 5731.230572] mlx4_en 0003:05:00.0: registered PHC clock [ 5731.230817] mlx4_en 0003:05:00.0: Activating port:1 [ 5731.262377] mlx4_en: eth12: Using 256 TX rings [ 5731.262494] mlx4_en: eth12: Using 4 RX rings [ 5731.262546] mlx4_en: eth12: frag:0 - size:1518 prefix:0 align:0 stride:1536 [ 5731.262761] mlx4_en: eth12: Initializing port [ 5731.263357] mlx4_en 0003:05:00.1: Activating port:1 [ 5731.263467] mlx4_en: 0003:05:00.1: Port 1: Assigned random MAC address d6:0f:0b:57:36:24 [ 5731.442926] mlx4_en: eth13: Using 256 TX rings [ 5731.443010] mlx4_en: eth13: Using 4 RX rings [ 5731.443018] mlx4_en: eth13: frag:0 - size:1518 prefix:0 align:0 stride:1536 [ 5731.443304] mlx4_en: eth13: Initializing port [ 5732.096548] mlx4_en: eth13: frag:0 - size:1518 prefix:0 align:0 stride:1536 [ 5732.452283] IPv6: ADDRCONF(NETDEV_UP): eth13: link is not ready [ 5732.951420] mlx4_en: eth12: Link Up [ 5732.951477] mlx4_en: eth13: Link Up [ 5732.951627] IPv6: ADDRCONF(NETDEV_CHANGE): eth13: link becomes ready [ 5733.171020] mlx4_en: eth13: CQE error - vendor syndrome: 0x6f syndrome: 0x2 [ 5733.315756] mlx4_en: eth13: CQE error - vendor syndrome: 0x6f syndrome: 0x2 [ 5734.165387] mlx4_en: eth13: CQE error - vendor syndrome: 0x6f syndrome: 0x2 [ 5748.045084] NETDEV WATCHDOG: eth13 (mlx4_core): transmit queue 1 timed out [ 5748.045245] ------------[ cut here ]------------ [ 5748.045301] WARNING: at net/sched/sch_generic.c:303 [ 5748.045355] Modules linked in: mlx4_en(O) mlx4_core nf_conntrack_ipv6 nf_defrag_ipv6 bnep nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack bluetooth rfkill ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_mangle iptable_security iptable_raw be2net tg3 ptp pps_core nfsd nfs_acl ses enclosure kvm_hv kvm_pr kvm lockd grace binfmt_misc shpchp sunrpc uinput lpfc ipr scsi_transport_fc [last unloaded: mlx4_core] [ 5748.046069] CPU: 1 PID: 0 Comm: swapper/1 Tainted: P O 3.18.0-rc2yw+ #161 [ 5748.046142] task: c0000027ef4f92d0 ti: c0000027ef574000 task.ti: c0000027ef574000 [ 5748.046215] NIP: c00000000079a9d4 LR: c00000000079a9d0 CTR: 0000000030044878 [ 5748.046286] REGS: c0000027ef5772d0 TRAP: 0700 Tainted: P O (3.18.0-rc2yw+) [ 5748.046357] MSR: 9000000000029032 <SF,HV,EE,ME,IR,DR,RI> CR: 28004024 XER: 00000000 [ 5748.046524] CFAR: c0000000008be150 SOFTE: 1 GPR00: c00000000079a9d0 c0000027ef577550 c00000000147acf0 000000000000003e GPR04: c000002004145888 c000002004156240 9000000000009032 0000000000000029 GPR08: c000000000d1acf0 0000000000000000 0000002003430000 0000000030044954 GPR12: 0000000028004022 c00000000fdc0900 0000000000000001 c0000000008f10a8 GPR16: c0000027ef795438 c0000027ef795838 c0000027ef795c38 0000000000000000 GPR20: c0000027ef795038 c0000000014b2200 0000000000000000 0000000000000000 GPR24: 0000000000000000 ffffffffffffffff 0000000000000000 0000000000000001 GPR28: 0000000000000004 c0000000014b2200 c000002781340000 0000000000000001 [ 5748.047435] NIP [c00000000079a9d4] .dev_watchdog+0x354/0x370 [ 5748.047496] LR [c00000000079a9d0] .dev_watchdog+0x350/0x370 [ 5748.047544] Call Trace: [ 5748.047570] [c0000027ef577550] [c00000000079a9d0] .dev_watchdog+0x350/0x370 (unreliable) [ 5748.047655] [c0000027ef577600] [c0000000001275a4] .call_timer_fn+0x64/0x190 [ 5748.047727] [c0000027ef5776b0] [c00000000012814c] .run_timer_softirq+0x2bc/0x3d0 [ 5748.047812] [c0000027ef5777b0] [c0000000000ad738] .__do_softirq+0x168/0x390 [ 5748.047885] [c0000027ef5778b0] [c0000000000adcc8] .irq_exit+0xc8/0x110 [ 5748.047958] [c0000027ef577930] [c00000000001d15c] .timer_interrupt+0x9c/0xd0 [ 5748.048030] [c0000027ef5779b0] [c000000000002658] decrementer_common+0x158/0x180 [ 5748.048116] --- interrupt: 901 at .arch_local_irq_restore+0x74/0x90 [ 5748.048116] LR = .arch_local_irq_restore+0x74/0x90 [ 5748.048224] [c0000027ef577ca0] [c0000027ef577d30] 0xc0000027ef577d30 (unreliable) [ 5748.048309] [c0000027ef577d10] [c00000000070887c] .cpuidle_enter_state+0xac/0x260 [ 5748.048393] [c0000027ef577dd0] [c0000000000f6c90] .cpu_startup_entry+0x410/0x460 [ 5748.048478] [c0000027ef577ed0] [c000000000040b40] .start_secondary+0x310/0x340 [ 5748.048562] [c0000027ef577f90] [c000000000008b6c] start_secondary_prolog+0x10/0x14 [ 5748.048643] Instruction dump: [ 5748.048680] 994d02d4 4bffff10 7fc3f378 4bfd5ff1 60000000 7fc4f378 7fe6fb78 7c651b78 [ 5748.048799] 3c62ff75 3863afb8 48123721 60000000 <0fe00000> 39200001 3d02fff0 99282b88 [ 5748.048920] ---[ end trace a04deb5eef8eba41 ]--- [ 5748.385369] mlx4_en: eth13: frag:0 - size:1518 prefix:0 align:0 stride:1536 [ 5760.841519] mlx4_en: eth13: CQE error - vendor syndrome: 0x6f syndrome: 0x2 [ 5781.165457] mlx4_en: eth13: CQE error - vendor syndrome: 0x6f syndrome: 0x2 [ 5781.315631] mlx4_en: eth13: CQE error - vendor syndrome: 0x6f syndrome: 0x2 [ 5782.165413] mlx4_en: eth13: CQE error - vendor syndrome: 0x6f syndrome: 0x2 >> >> But for mlx4_en, I am not sure I could raise the debug level with ethtool, >> since the ethernet driver may not work properly. Actually I am not sure how to >> raise the level with ethtool. Could you give me an example? > ># ethtool -s ens5f1d1 msglvl 0xffff > >> >>>> >>>> And this error is reported from VF always. After the error, the other network >>>> interface seems can't function. >>>> >>>>> >>>>>Amir. >>>> >>>> -- >>>> Richard Yang >>>> Help you, Help me >>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe netdev" in >>>> the body of a message to majordomo@...r.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> -- >> Richard Yang >> Help you, Help me >> >> -- >> To unsubscribe from this list: send the line "unsubscribe netdev" in >> the body of a message to majordomo@...r.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- Richard Yang Help you, Help me -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists