Date:   Mon, 27 Nov 2017 10:35:48 +0100
From:   Sean Nyekjær <sean@...kjaer.dk>
To:     Thomas Petazzoni <thomas.petazzoni@...e-electrons.com>
Cc:     netdev@...r.kernel.org, Marcin Wojtas <mw@...ihalf.com>,
        Grégory Clement 
        <gregory.clement@...e-electrons.com>,
        Antoine Ténart <antoine.tenart@...e-electrons.com>,
        Simon Guinot <simon.guinot@...uanux.org>
Subject: Re: [BUG] mveta: mvneta_txq_bufs_free NULL pointer dereference

On 27 November 2017 at 10:00, Thomas Petazzoni
<thomas.petazzoni@...e-electrons.com> wrote:
> Hello,
>
> +Grégory Clement, Antoine Ténart, Simon Guinot, Marcin Wojtas.
>
> On Mon, 27 Nov 2017 09:47:10 +0100, Sean Nyekjær wrote:
>
>> I see you are the maintainer on mvneta :-)
>> I have an Espressobin board, i'm currently running with archlinux for
>> arm. I have been running with 4.13.x mainline, 4.13.x with arch
>> patches, 4.14.0 mainline and 4.14.1 with arch patches.
>>
>> You can see what patches that are applied here:
>> https://archlinuxarm.org/packages/aarch64/linux-espressobin/files
>>
>> To the issue (same with all the kernels) :-)
>> Every 10-14 days, sometimes faster the ethernet stops working. I have
>> a serial debug connected so i can check the logs. The kernel logs
>> contains nothing :-(
>> If i then try to down the interface nasty things happen.
>
> I'm not sure at all, but could you try to apply
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0d63785c6b94b5d2f095f90755825f90eea791f5
> and see if the problem is resolved ?
>
> I'm leaving the full kernel dump below for others to look at.
>
> Thanks!
>
> Thomas
>
>> Here is my kernel dump:
>>
>> [root@...ressobin ~]# ip link set dev eth0 down
>> [ 1339.493220] mvneta d0030000.ethernet eth0: TIMEOUT for TX stopped
>> status=0x0000ffff
>> [root@...ressobin ~]#
>> [ 1339.802218] br0: port 1(lan1) entered disabled state
>> [ 1339.874861] br0: port 2(lan0) entered disabled state
>> [ 1339.927740] alloc_contig_range: [7c198, 7c19d) PFNs busy
>> [ 1339.933991] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>> [ 1339.974840] br0: bridge flag offload is not supported 1(lan1)
>> [ 1340.033767] br0: bridge flag offload is not supported 2(lan0)
>> [ 1340.532339] mv88e6085 d0032004.mdio-mii:01 lan1: Link is Down
>> [ 1340.537957] mv88e6085 d0032004.mdio-mii:01 lan0: Link is Down
>> [ 1341.012314] mvneta d0030000.ethernet eth0: Link is Up - 1Gbps/Full
>> - flow control off
>> [ 1341.020267] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> [ 1341.035094] br0: port 1(lan1) entered blocking state
>> [ 1341.040148] br0: port 1(lan1) entered forwarding state
>> [ 1341.056721] br0: port 1(lan1) entered disabled state
>> [ 1341.077884] br0: port 2(lan0) entered blocking state
>> [ 1341.082795] br0: port 2(lan0) entered forwarding state
>> [ 1341.091679] br0: port 2(lan0) entered disabled state
>> [ 1341.107766] IPv6: ADDRCONF(NETDEV_UP): wan: link is not ready
>> [ 1342.058742] mv88e6085 d0032004.mdio-mii:01 lan1: Link is Down
>> [ 1342.143820] mv88e6085 d0032004.mdio-mii:01 lan0: Link is Down
>> [ 1344.139466] mv88e6085 d0032004.mdio-mii:01 lan1: Link is Up -
>> 100Mbps/Full - flow control off
>> [ 1344.148358] br0: port 1(lan1) entered blocking state
>> [ 1344.153309] br0: port 1(lan1) entered forwarding state
>> [ 1344.202470] Unable to handle kernel NULL pointer dereference at
>> virtual address 00000081
>> [ 1344.210490] Mem abort info:
>> [ 1344.213332]   Exception class = DABT (current EL), IL = 32 bits
>> [ 1344.219833]   SET = 0, FnV = 0
>> [ 1344.222466]   EA = 0, S1PTW = 0
>> [ 1344.226149] Data abort info:
>> [ 1344.229201]   ISV = 0, ISS = 0x00000006
>> [ 1344.233099]   CM = 0, WnR = 0
>> [ 1344.236131] user pgtable: 4k pages, 48-bit VAs, pgd = ffff80006ca11000
>> [ 1344.242788] [0000000000000081] *pgd=000000006ca31003,
>> *pud=000000006ca75003, *pmd=0000000000000000
>> [ 1344.252122] Internal error: Oops: 96000006 [#1] SMP
>> [ 1344.256794] Modules linked in: tun xt_nat veth ipt_MASQUERADE
>> nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat
>> nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype
>> iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter overlay
>> bridge stp llc
>> aes_ce_blk crypto_simd aes_ce_cipher crc32_ce crct10dif_ce ghash_ce
>> aes_arm64 sha2_ce sha256_arm64 sha1_ce sch_fq_codel ip_tables ipv6
>> [ 1344.293629] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.1-1-ARCH #1
>> [ 1344.300542] Hardware name: Globalscale Marvell ESPRESSOBin Board (DT)
>> [ 1344.307103] task: ffff000008d70580 task.stack: ffff000008d60000
>> [ 1344.313311] PC is at mvneta_txq_bufs_free.isra.24+0x68/0x170
>> [ 1344.318968] LR is at mvneta_txq_bufs_free.isra.24+0xd8/0x170
>> [ 1344.324537] pc : [<ffff000008685198>] lr : [<ffff000008685208>]
>> pstate: 80000145
>> [ 1344.332089] sp : ffff000008003d10
>> [ 1344.335684] x29: ffff000008003d10 x28: ffff000008d45000
>> [ 1344.341164] x27: ffff800077cb0028 x26: 0000000000000003
>> [ 1344.346737] x25: 0000000000000001 x24: ffff800077d36a00
>> [ 1344.352219] x23: ffff800077d32938 x22: 0000000000000005
>> [ 1344.357713] x21: ffff000008926110 x20: 00000000000000b6
>> [ 1344.363281] x19: ffff800077cb0208 x18: 000000000000002e
>> [ 1344.368947] x17: 0000ffff9127b8d0 x16: ffff0000080ce160
>> [ 1344.374340] x15: 0000000000000008 x14: 0000000000000000
>> [ 1344.379824] x13: 0000000000000001 x12: 0000000000000000
>> [ 1344.384859] x11: 0000000000000000 x10: 0000000000000000
>> [ 1344.390699] x9 : ffff000008d67000 x8 : 00000001000197df
>> [ 1344.395825] x7 : 0000000000000000 x6 : 0000000000000000
>> [ 1344.401663] x5 : 0000000000000001 x4 : 0000000000000000
>> [ 1344.407234] x3 : ffff800078b3a000 x2 : ffff00000a43d060
>> [ 1344.412715] x1 : 0000000000000003 x0 : 0000000000000003
>> [ 1344.418110] Process swapper/0 (pid: 0, stack limit = 0xffff000008d60000)
>> [ 1344.425026] Call trace:
>> [ 1344.427277] Exception stack(0xffff000008003bd0 to 0xffff000008003d10)
>> [ 1344.434372] 3bc0:
>> 0000000000000003 0000000000000003
>> [ 1344.442550] 3be0: ffff00000a43d060 ffff800078b3a000
>> 0000000000000000 0000000000000001
>> [ 1344.450374] 3c00: 0000000000000000 0000000000000000
>> 00000001000197df ffff000008d67000
>> [ 1344.458460] 3c20: 0000000000000000 0000000000000000
>> 0000000000000000 0000000000000001
>> [ 1344.466189] 3c40: 0000000000000000 0000000000000008
>> ffff0000080ce160 0000ffff9127b8d0
>> [ 1344.474633] 3c60: 000000000000002e ffff800077cb0208
>> 00000000000000b6 ffff000008926110
>> [ 1344.482808] 3c80: 0000000000000005 ffff800077d32938
>> ffff800077d36a00 0000000000000001
>> [ 1344.490805] 3ca0: 0000000000000003 ffff800077cb0028
>> ffff000008d45000 ffff000008003d10
>> [ 1344.498897] 3cc0: ffff000008685208 ffff000008003d10
>> ffff000008685198 0000000080000145
>> [ 1344.506896] 3ce0: ffff800077cb0208 00000000000000b6
>> 0001000000000000 0000000000000005
>> [ 1344.515255] 3d00: ffff000008003d10 ffff000008685198
>> [ 1344.520207] [<ffff000008685198>] mvneta_txq_bufs_free.isra.24+0x68/0x170
>> [ 1344.527142] [<ffff0000086873c8>] mvneta_poll+0x4f0/0xad8
>> [ 1344.532528] [<ffff0000087d04fc>] net_rx_action+0x184/0x418
>> [ 1344.538461] [<ffff000008081798>] __do_softirq+0x130/0x32c
>> [ 1344.543594] [<ffff0000080cee58>] irq_exit+0xc8/0x100
>> [ 1344.548812] [<ffff00000812a52c>] __handle_domain_irq+0x6c/0xc0
>> [ 1344.554651] [<ffff000008081560>] gic_handle_irq+0x80/0x184
>> [ 1344.560492] Exception stack(0xffff000008d63db0 to 0xffff000008d63ef0)
>> [ 1344.567233] 3da0:
>> ffff000008d45000 0000000000000000
>> [ 1344.575317] 3dc0: ffff000008d63ef0 0000000000784718
>> 0000800073275000 ffff000008d63f00
>> [ 1344.583403] 3de0: 0000800073275000 0000000000000001
>> ffff000008d70fe0 ffff000008d63e80
>> [ 1344.591403] 3e00: 0000000000000a00 0000000000000000
>> 0000000000000000 0000000000000001
>> [ 1344.599666] 3e20: 0000000000000000 0000000000000008
>> ffff0000080ce160 0000ffff9127b8d0
>> [ 1344.607843] 3e40: 000000000000002e ffff000008d45000
>> ffff000008d69000 ffff000008d69000
>> [ 1344.615839] 3e60: ffff000008d4f148 ffff000008d69bec
>> 0000000000000000 0000000000000000
>> [ 1344.623836] 3e80: ffff000008d70580 000000007ff963f8
>> 0000000000c80018 ffff000008d63ef0
>> [ 1344.631922] 3ea0: ffff00000808521c ffff000008d63ef0
>> ffff000008085220 0000000000000145
>> [ 1344.640097] 3ec0: ffff80007bfffb00 ffff000008cea028
>> ffffffffffffffff 0000000000000000
>> [ 1344.648181] 3ee0: ffff000008d63ef0 ffff000008085220
>> [ 1344.653037] [<ffff000008082fb0>] el1_irq+0xb0/0x140
>> [ 1344.657891] [<ffff000008085220>] arch_cpu_idle+0x30/0x188
>> [ 1344.663911] [<ffff00000810eed0>] do_idle+0x128/0x1e8
>> [ 1344.669041] [<ffff00000810f13c>] cpu_startup_entry+0x2c/0x30
>> [ 1344.674972] [<ffff00000890516c>] rest_init+0xb4/0xc0
>> [ 1344.679945] [<ffff000008c80cf0>] start_kernel+0x394/0x3a8
>> [ 1344.685594] Code: 93407c01 8b011442 f8617879 b4000079 (b9408321)
>> [ 1344.691523] ---[ end trace 0e5abdfc76ee83e5 ]---
>> [ 1344.696733] Kernel panic - not syncing: Fatal exception in interrupt
>> [ 1344.703310] SMP: stopping secondary CPUs
>> [ 1344.707369] Kernel Offset: disabled
>> [ 1344.710613] CPU features: 0x002008
>> [ 1344.714294] Memory Limit: none
>> [ 1344.717086] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
>>
>> I you want more logs or some other details about my setup i'll be
>> happy to help :-)
>> Also with testing a possible fix.
>>
>> Thanks,
>> Sean Nyekjaer
>
>
>
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com

I will apply the patch right away, and report back.

BR
Sean Nyekjaer

Powered by blists - more mailing lists