| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Wed, 20 Sep 2017 06:11:15 -0700
From: Eric Dumazet <eric.dumazet@...il.com>
To: Paweł Staszewski <pstaszewski@...are.pl>,
Wei Wang <weiwan@...gle.com>
Cc: Linux Kernel Network Developers <netdev@...r.kernel.org>,
edumazet@...gle.com
Subject: Re: Latest net-next from GIT panic
Sorry for top-posting, but this is to give context to Wei, since Pawel
used a top posting way to report his bisection.
Wei, can you take a look at Pawel report ?
Crash happens in dst_destroy() at following :
if (dst->dev)
dev_put(dst->dev); <<CRASH>>
dst->dev is not NULL, but netdev->pcpu_refcnt is NULL
65 ff 08 decl %gs:(%rax) // CRASH since rax = NULL
Pawel, please share your netdevices and routing setup ?
Thanks !
On Wed, 2017-09-20 at 14:49 +0200, Paweł Staszewski wrote:
> And the last one
>
> git bisect good
> Bisecting: 1 revision left to test after this (roughly 1 step)
> [1cfb71eeb12047bcdbd3e6730ffed66e810a0855] ipv6: take dst->__refcnt for
> insertion into fib6 tree
>
> With this have kernel panic same as always
>
> git bisect bad
> Bisecting: 0 revisions left to test after this (roughly 0 steps)
> [b838d5e1c5b6e57b10ec8af2268824041e3ea911] ipv4: mark DST_NOGC and
> remove the operation of dst_free()
>
>
>
> W dniu 2017-09-20 o 14:23, Paweł Staszewski pisze:
> > Almost there
> >
> > Bisecting: 6 revisions left to test after this (roughly 3 steps)
> > [ad65a2f05695aced349e308193c6e2a6b1d87112] ipv6: call dst_hold_safe()
> > properly
> >
> >
> >
> > W dniu 2017-09-20 o 13:02, Paweł Staszewski pisze:
> >> Ok resumed and soo far:
> >>
> >> Panic:
> >>
> >> # bad: [9cc9a5cb176ccb4f2cda5ac34da5a659926f125f] datapath: Avoid
> >> using stack larger than 1024.
> >> git bisect bad 9cc9a5cb176ccb4f2cda5ac34da5a659926f125f
> >>
> >> No panic:
> >>
> >> # good: [073cf9e20c333ab29744717a23f9e43ec7512a20] Merge branch
> >> 'udp-reduce-cache-pressure'
> >> git bisect good 073cf9e20c333ab29744717a23f9e43ec7512a20
> >>
> >>
> >> W dniu 2017-09-20 o 12:22, Paweł Staszewski pisze:
> >>> Soo far bisected and marked:
> >>>
> >>> git bisect start
> >>> # bad: [07dd6cc1fff160143e82cf5df78c1db0b6e03355] Linux 4.13.2
> >>> git bisect bad 07dd6cc1fff160143e82cf5df78c1db0b6e03355
> >>> # good: [5d7d2e03e0f01a992e3521b180c3d3e67905f269] Linux 4.12.13
> >>> git bisect good 5d7d2e03e0f01a992e3521b180c3d3e67905f269
> >>> # good: [6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c] Linux 4.12
> >>> git bisect good 6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c
> >>> # bad: [ac7b75966c9c86426b55fe1c50ae148aa4571075] Merge tag
> >>> 'pinctrl-v4.13-1' of
> >>> git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl
> >>> git bisect bad ac7b75966c9c86426b55fe1c50ae148aa4571075
> >>> # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch
> >>> 'next' of
> >>> git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security
> >>> git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31
> >>> # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch
> >>> 'next' of
> >>> git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security
> >>> git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31
> >>> # good: [e24dd9ee5399747b71c1d982a484fc7601795f31] Merge branch
> >>> 'next' of
> >>> git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security
> >>> git bisect good e24dd9ee5399747b71c1d982a484fc7601795f31
> >>>
> >>>
> >>>
> >>> W dniu 2017-09-20 o 12:21, Paweł Staszewski pisze:
> >>>> Ok kernel crashed with different panic that i didnt catch when i
> >>>> was doing bisect and now my bisection is broken :)
> >>>>
> >>>> git bisect good
> >>>> Bisecting: 1787 revisions left to test after this (roughly 11 steps)
> >>>> error: Your local changes to the following files would be
> >>>> overwritten by checkout:
> >>>> Documentation/00-INDEX
> >>>> Documentation/ABI/stable/sysfs-class-udc
> >>>> Documentation/ABI/testing/configfs-usb-gadget-uac1
> >>>> Documentation/ABI/testing/ima_policy
> >>>> Documentation/ABI/testing/sysfs-bus-iio
> >>>> Documentation/ABI/testing/sysfs-bus-iio-meas-spec
> >>>> Documentation/ABI/testing/sysfs-bus-iio-timer-stm32
> >>>> Documentation/ABI/testing/sysfs-class-net
> >>>> Documentation/ABI/testing/sysfs-class-power-twl4030
> >>>> Documentation/ABI/testing/sysfs-class-typec
> >>>> Documentation/DMA-API.txt
> >>>> Documentation/IRQ-domain.txt
> >>>> Documentation/Makefile
> >>>> Documentation/PCI/MSI-HOWTO.txt
> >>>> Documentation/RCU/00-INDEX
> >>>> Documentation/RCU/Design/Requirements/Requirements.html
> >>>> Documentation/RCU/checklist.txt
> >>>> Documentation/admin-guide/README.rst
> >>>> Documentation/admin-guide/devices.txt
> >>>> Documentation/admin-guide/index.rst
> >>>> Documentation/admin-guide/kernel-parameters.txt
> >>>> Documentation/admin-guide/pm/cpufreq.rst
> >>>> Documentation/admin-guide/pm/intel_pstate.rst
> >>>> Documentation/admin-guide/ras.rst
> >>>> Documentation/arm/Atmel/README
> >>>> Documentation/block/biodoc.txt
> >>>> Documentation/conf.py
> >>>> Documentation/core-api/assoc_array.rst
> >>>> Documentation/core-api/atomic_ops.rst
> >>>> Documentation/core-api/index.rst
> >>>> Documentation/crypto/asymmetric-keys.txt
> >>>> Documentation/dev-tools/index.rst
> >>>> Documentation/dev-tools/sparse.rst
> >>>> Documentation/devicetree/bindings/arm/amlogic.txt
> >>>> Documentation/devicetree/bindings/arm/atmel-at91.txt
> >>>> Documentation/devicetree/bindings/arm/ccn.txt
> >>>> Documentation/devicetree/bindings/arm/cpus.txt
> >>>> Documentation/devicetree/bindings/arm/gemini.txt
> >>>> Documentation/devicetree/bindings/arm/hisilicon/hisilicon.txt
> >>>> Documentation/devicetree/bindings/arm/keystone/keystone.txt
> >>>> Documentation/devicetree/bindings/arm/mediatek.txt
> >>>> Documentation/devicetree/bindings/arm/rockchip.txt
> >>>> Documentation/devicetree/bindings/arm/shmobile.txt
> >>>> Documentation/devicetree/bindings/arm/tegra.txt
> >>>> Documentation/devicetree/bindings/ata/ahci-fsl-qoriq.txt
> >>>> Documentation/devicetree/bindings/bus/brcm,gisb-arb.txt
> >>>> Documentation/devicetree/bindings/clock/brcm,iproc-clocks.txt
> >>>> Documentation/devicetree/bindings/cpufreq/ti-cpufreq.txt
> >>>> Documentation/devicetree/bindings/gpio/gpio_atmel.txt
> >>>> Documentation/devicetree/bindings/iio/adc/amlogic,meson-saradc.txt
> >>>> Documentation/devicetree/bindings/iio/adc/renesas,gyroadc.txt
> >>>> Documentation/devicetree/bindings/iio/adc/st,stm32-adc.txt
> >>>> Documentation/devicetree/bindings/iio/imu/st_lsm6dsx.txt
> >>>> Documentation/devicetree/bindings/interrupt-controller/allwinner,sunxi-nmi.txt
> >>>>
> >>>> Documentation/devicetree/bindings/interrupt-controller/aspeed,ast2400-vic.txt
> >>>>
> >>>> Documentation/devicetree/bindings/interrupt-controller/mediatek,sysirq.txt
> >>>>
> >>>> Documentation/devicetree/bindings/leds/common.txt
> >>>> Documentation/devicetree/bindings/mfd/hi6421.txt
> >>>> Documentation/devicetree/bindings/mfd/tps65910.txt
> >>>> Documentation/devicetree/bindings/mmc/fsl-esdhc.txt
> >>>> Documentation/devicetree/bindings/mmc/k3-dw-mshc.txt
> >>>> Documentation/devicetree/bindings/mmc/rockchip-dw-mshc.txt
> >>>> Documentation/devicetree/bindings/mmc/ti-omap-hsmmc.txt
> >>>> Documentation/devicetree/bindings/mtd/atmel-nand.txt
> >>>> Documentation/devicetree/bindings/net/dsa/b53.txt
> >>>> Documentation/devicetree/bindings/net/ethernet.txt
> >>>> Documentation/devicetree/bindings/net/macb.txt
> >>>> Documentation/devicetree/bindings/net/marvell-orion-mdio.txt
> >>>> Documentation/devicetree/bindings/net/ti,wilink-st.txt
> >>>> Documentation/devicetree/bindings/net/wireless/ti,wlcore.txt
> >>>> Documentation/devicetree/bindings/nvmem/rockchip-efuse.txt
> >>>> Documentation/devicetree/bindings/opp/opp.txt
> >>>> Documentation/devicetree/bindings/phy/bcm-ns-usb3-phy.txt
> >>>> Documentation/devicetree/bindings/phy/brcm-sata-phy.txt
> >>>> Documentation/devicetree/bindings/phy/meson8b-usb2-phy.txt
> >>>> Documentation/devicetree/bindings/phy/phy-rockchip-inno-usb2.txt
> >>>> Documentation/devicetree/bindings/power/rockchip-io-domain.txt
> >>>> Documentation/devicetree/bindings/power/supply/bq27xxx.txt
> >>>> Documentation/devicetree/bindings/property-units.txt
> >>>> Documentation/devicetree/bindings/regulator/regulator.txt
> >>>> Documentation/devicetree/bindings/serial/8
> >>>> error: The following untracked working tree files would be
> >>>> overwritten by checkout:
> >>>> Documentation/ABI/testing/sysfs-class-net-phydev
> >>>> Documentation/DocBook/.gitignore
> >>>> Documentation/DocBook/Makefile
> >>>> Documentation/DocBook/filesystems.tmpl
> >>>> Documentation/DocBook/kernel-hacking.tmpl
> >>>> Documentation/DocBook/kernel-locking.tmpl
> >>>> Documentation/DocBook/kgdb.tmpl
> >>>> Documentation/DocBook/libata.tmpl
> >>>> Documentation/DocBook/librs.tmpl
> >>>> Documentation/DocBook/lsm.tmpl
> >>>> Documentation/DocBook/mtdnand.tmpl
> >>>> Documentation/DocBook/networking.tmpl
> >>>> Documentation/DocBook/rapidio.tmpl
> >>>> Documentation/DocBook/s390-drivers.tmpl
> >>>> Documentation/DocBook/scsi.tmpl
> >>>> Documentation/DocBook/sh.tmpl
> >>>> Documentation/DocBook/stylesheet.xsl
> >>>> Documentation/DocBook/w1.tmpl
> >>>> Documentation/DocBook/z8530book.tmpl
> >>>> Documentation/Makefile.sphinx
> >>>> Documentation/RCU/trace.txt
> >>>> Documentation/devicetree/bindings/i2c/i2c-mt6577.txt
> >>>> Documentation/devicetree/bindings/misc/allwinner,syscon.txt
> >>>> Documentation/devicetree/bindings/net/cortina.txt
> >>>> Documentation/devicetree/bindings/net/dsa/ksz.txt
> >>>> Documentation/devicetree/bindings/net/dwmac-sun8i.txt
> >>>> Documentation/devicetree/bindings/net/qca,qca7000.txt
> >>>> Documentation/devicetree/bindings/power/max8903-charger.txt
> >>>> Documentation/devicetree/bindings/power_supply/maxim,max14656.txt
> >>>> Documentation/devicetree/bindings/ptp/brcm,ptp-dte.txt
> >>>> Documentation/devicetree/bindings/timer/moxa,moxart-timer.txt
> >>>> Documentation/doc-guide/docbook.rst
> >>>> Documentation/networking/tls.txt
> >>>> Documentation/prctl/no_new_privs.txt
> >>>> Documentation/prctl/seccomp_filter.txt
> >>>> Documentation/security/00-INDEX
> >>>> Documentation/security/IMA-templates.txt
> >>>> Documentation/security/LSM.txt
> >>>> Documentation/security/LoadPin.txt
> >>>> Documentation/security/SELinux.txt
> >>>> Documentation/security/Smack.txt
> >>>> Documentation/security/Yama.txt
> >>>> Documentation/security/apparmor.txt
> >>>> Documentation/security/conf.py
> >>>> Documentation/security/credentials.txt
> >>>> Documentation/security/keys-ecryptfs.txt
> >>>> Documentation/security/keys-request-key.txt
> >>>> Documentation/security/keys-trusted-encrypted.txt
> >>>> Documentation/security/keys.txt
> >>>> Documentation/security/self-protection.txt
> >>>> Documentation/security/tomoyo.txt
> >>>> Documentation/sphinx/convert_template.sed
> >>>> Documentation/sphinx/post_convert.sed
> >>>> Documentation/sphinx/tmplcvt
> >>>> Documentation/usb/typec.rst
> >>>> Documentation/usb/usb3-debug-port.rst
> >>>> arch/arm/boot/dts/rk1108-evb.dts
> >>>> arch/arm/boot/dts/rk1108.dtsi
> >>>> arch/arm/boot/dts/tegra20-whistler.dts
> >>>> arch/arm/mach-omap2/opp.c
> >>>> arch/arm/mach-omap2/pmu.c
> >>>> arch/ia64/include/asm/siginfo.h
> >>>> arch/m32r/include/uapi/asm/siginfo.h
> >>>> arch/microblaze/include/asm/bitops.h
> >>>> arch/microblaze/include/asm/bug.h
> >>>> arch/microblaze/include/asm/bugs.h
> >>>> arch/microblaze/include/asm/div64.h
> >>>> arch/microblaze/include/asm/emergency-restart.h
> >>>> arch/microblaze/include/asm/fb.h
> >>>> arch/microblaze/include/asm/hardirq.h
> >>>> arch/microblaze/include/asm/irq_regs.h
> >>>> arch/microblaze/include/asm/kdebug.h
> >>>> arch/microblaze/include/asm/kmap_types.h
> >>>> arch/microblaze/include/asm/linkage.h
> >>>> arch/microblaze/include/asm/local.h
> >>>> arch/microblaze/include/asm/local64.h
> >>>> arch/microblaze/include/asm/parport.h
> >>>> arch/microblaze/include/asm/percpu.h
> >>>> arch/microblaze/include/asm/serial.h
> >>>> arch/microblaze/include/asm/shmparam.h
> >>>> arch/microblaze/include/asm/topology.h
> >>>> arch/microblaze/include/asm/ucontext.h
> >>>> arch/microblaze/include/asm/vga.h
> >>>> arch/microblaze/include/asm/xor.h
> >>>> arch/microblaze/include/uapi/asm/bitsperlong.h
> >>>> arch/microblaze/include/uapi/asm/errno.h
> >>>> arch/microblaze/include/uapi/asm/fcntl.h
> >>>> arch/microblaze/include/uapi/asm/ioctl.h
> >>>> arch/microblaze/include/uapi/asm/ioctls.h
> >>>> arch/microblaze/include/uapi/asm/ipcbuf.h
> >>>> arch/microblaze/include/uapi/asm/kvm_para.h
> >>>> arch/microblaze/include/uapi/asm/mman.h
> >>>> arch/microblaze/include/uapi/asm/msgbuf.h
> >>>> arch/microblaze/include/uapi/asm/param.h
> >>>> arch/microblaze/include/uapi/asm/poll.h
> >>>> arch/microblaze/include/uapi/asm/resource.h
> >>>> arch/microblaze/include/uapi/asm/sembuf.h
> >>>> arch/microblaze/include/uapi/asm/shmbuf.h
> >>>> arch/microblaze/include/uapi/asm/siginfo.h
> >>>> arch/microblaze/include/uapi/asm/signal.h
> >>>> arch/microblaze/includ
> >>>> Aborting
> >>>>
> >>>>
> >>>>
> >>>> W dniu 2017-09-20 o 11:45, Paweł Staszewski pisze:
> >>>>> Ok looks like ending bisection
> >>>>>
> >>>>>
> >>>>> Latest bisected kernel when there is no kernel panic 4.12.0+ (from
> >>>>> next) - but only this warning:
> >>>>>
> >>>>> [ 309.030019] NETDEV WATCHDOG: enp4s0f0 (ixgbe): transmit queue 0
> >>>>> timed out
> >>>>> [ 309.030034] ------------[ cut here ]------------
> >>>>> [ 309.030040] WARNING: CPU: 35 PID: 0 at dev_watchdog+0xcf/0x139
> >>>>> [ 309.030041] Modules linked in: bonding ipmi_si
> >>>>> x86_pkg_temp_thermal
> >>>>> [ 309.030045] CPU: 35 PID: 0 Comm: swapper/35 Not tainted 4.12.0+ #5
> >>>>> [ 309.030046] task: ffff88086d98a000 task.stack: ffffc90003378000
> >>>>> [ 309.030048] RIP: 0010:dev_watchdog+0xcf/0x139
> >>>>> [ 309.030049] RSP: 0018:ffff88087fbc3ea8 EFLAGS: 00010246
> >>>>> [ 309.030050] RAX: 000000000000003d RBX: ffff88046b680000 RCX:
> >>>>> 0000000000000000
> >>>>> [ 309.030050] RDX: ffff88087fbd2f01 RSI: 0000000000000000 RDI:
> >>>>> ffff88087fbcda08
> >>>>> [ 309.030051] RBP: ffff88087fbc3eb8 R08: 0000000000000000 R09:
> >>>>> ffff88087ff80a04
> >>>>> [ 309.030051] R10: 0000000000000000 R11: ffff88086d98a001 R12:
> >>>>> 0000000000000000
> >>>>> [ 309.030052] R13: ffff88087fbc3ef8 R14: ffff88086d98a000 R15:
> >>>>> ffffffff81c06008
> >>>>> [ 309.030053] FS: 0000000000000000(0000)
> >>>>> GS:ffff88087fbc0000(0000) knlGS:0000000000000000
> >>>>> [ 309.030054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >>>>> [ 309.030054] CR2: 00007fba600f6098 CR3: 000000086b955000 CR4:
> >>>>> 00000000001406e0
> >>>>> [ 309.030055] Call Trace:
> >>>>> [ 309.030057] <IRQ>
> >>>>> [ 309.030059] ? netif_tx_lock+0x79/0x79
> >>>>> [ 309.030062] call_timer_fn.isra.24+0x17/0x77
> >>>>> [ 309.030063] run_timer_softirq+0x118/0x161
> >>>>> [ 309.030065] ? netif_tx_lock+0x79/0x79
> >>>>> [ 309.030066] ? ktime_get+0x2b/0x42
> >>>>> [ 309.030070] ? lapic_next_deadline+0x21/0x27
> >>>>> [ 309.030073] ? clockevents_program_event+0xa8/0xc5
> >>>>> [ 309.030076] __do_softirq+0xa8/0x19d
> >>>>> [ 309.030078] irq_exit+0x5d/0x6b
> >>>>> [ 309.030079] smp_apic_timer_interrupt+0x2a/0x36
> >>>>> [ 309.030082] apic_timer_interrupt+0x89/0x90
> >>>>> [ 309.030085] RIP: 0010:mwait_idle+0x4e/0x6a
> >>>>> [ 309.030086] RSP: 0018:ffffc9000337be98 EFLAGS: 00000246
> >>>>> ORIG_RAX: ffffffffffffff10
> >>>>> [ 309.030087] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
> >>>>> 0000000000000000
> >>>>> [ 309.030087] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> >>>>> ffff88086d98a000
> >>>>> [ 309.030088] RBP: ffffc9000337be98 R08: ffff88046f8279a0 R09:
> >>>>> ffff88046f827040
> >>>>> [ 309.030089] R10: ffff88086d98a000 R11: ffff88086d98a000 R12:
> >>>>> 0000000000000000
> >>>>> [ 309.030089] R13: ffff88086d98a000 R14: ffff88086d98a000 R15:
> >>>>> ffff88086d98a000
> >>>>> [ 309.030090] </IRQ>
> >>>>> [ 309.030094] arch_cpu_idle+0xa/0xc
> >>>>> [ 309.030095] default_idle_call+0x19/0x1b
> >>>>> [ 309.030102] do_idle+0xbc/0x196
> >>>>> [ 309.030104] cpu_startup_entry+0x1d/0x20
> >>>>> [ 309.030105] start_secondary+0xd8/0xdc
> >>>>> [ 309.030108] secondary_startup_64+0x9f/0x9f
> >>>>> [ 309.030109] Code: cc 75 bd eb 35 48 89 df c6 05 c3 dc 74 00 01
> >>>>> e8 3a 62 fe ff 44 89 e1 48 89 de 48 89 c2 48 c7 c7 0f 65 a4 81 31
> >>>>> c0 e8 3d 4c b5 ff <0f> ff 48 8b 83 e0 01 00 00 48 89 df ff 50 78
> >>>>> 48 8b 05 a0 bc 6a
> >>>>> [ 309.030128] ---[ end trace 9102cb25703ae2d9 ]---
> >>>>>
> >>>>>
> >>>>> I just marked it as good - cause this problem above is differend -
> >>>>> and im going to:
> >>>>>
> >>>>> git bisect good
> >>>>> Bisecting: 1787 revisions left to test after this (roughly 11 steps)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> W dniu 2017-09-20 o 10:44, Paweł Staszewski pisze:
> >>>>>> Trying to make video from ipmi :)
> >>>>>>
> >>>>>> with that results:
> >>>>>>
> >>>>>> https://bugzilla.kernel.org/attachment.cgi?id=258521
> >>>>>>
> >>>>>> catched two more lines where it starts - panic from 4.13.2.
> >>>>>>
> >>>>>>
> >>>>>> Now will try tro do some bisection
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> W dniu 2017-09-20 o 09:58, Paweł Staszewski pisze:
> >>>>>>> Hi
> >>>>>>>
> >>>>>>>
> >>>>>>> Will try bisecting tonight
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> W dniu 2017-09-20 o 05:24, Eric Dumazet pisze:
> >>>>>>>> On Wed, 2017-09-20 at 02:06 +0200, Paweł Staszewski wrote:
> >>>>>>>>> Just checked kernel 4.13.2 and same problem
> >>>>>>>>>
> >>>>>>>>> Just after start all 6 bgp sessions - and kernel starts to
> >>>>>>>>> learn routes
> >>>>>>>>> it panic.
> >>>>>>>>>
> >>>>>>>>> https://bugzilla.kernel.org/attachment.cgi?id=258509
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> Unfortunately we have not enough information from these traces.
> >>>>>>>>
> >>>>>>>> Can you get a full stack trace ?
> >>>>>>>>
> >>>>>>>> Alternatively, can you bisect ?
> >>>>>>>>
> >>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >
> >
>
Powered by blists - more mailing lists