Date:	Sun, 17 Aug 2008 15:50:01 +0300
From:	Denys Fedoryshchenko <denys@...p.net.lb>
To:	Jarek Poplawski <jarkao2@...il.com>
Cc:	netdev@...r.kernel.org
Subject: Re: panic 2.6.27-rc3-git2, qdisc_dequeue_head

On Sunday 17 August 2008, Jarek Poplawski wrote:

> Denys, I would like to be sure: do you mean here the last patch
> attached below? So, the alternative shaper works with previous
> patches, but not this one? If so, than no need to test this patch
> anymore, only confirm this, please.
No.
Alternative shaper didn't work without applying latest patch (just i had old 
kernel, and i decide just to try alternative shaper).

Latest warning with all patches applied. I will test alternative shaper with 
all patches applied now.



>
> Jarek P.
>
> > Here is panic i got when i try to run alternative shaper.
> >
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] BUG: unable to handle kernel
> > Aug 17 15:01:54 192.168.20.66 paging request
> > Aug 17 15:01:54 192.168.20.66 at 50639a62
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] IP:
> > Aug 17 15:01:54 192.168.20.66 [<c025fd4e>] net_tx_action+0x78/0xd8
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Oops: 0000 [#1]
> > Aug 17 15:01:54 192.168.20.66 SMP
> > Aug 17 15:01:54 192.168.20.66
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Modules linked in:
> > Aug 17 15:01:54 192.168.20.66 cls_flow
> > Aug 17 15:01:54 192.168.20.66 act_police
> > Aug 17 15:01:54 192.168.20.66 sch_ingress
> > Aug 17 15:01:54 192.168.20.66 sch_prio
> > Aug 17 15:01:54 192.168.20.66 netconsole
> > Aug 17 15:01:54 192.168.20.66 configfs
> > Aug 17 15:01:54 192.168.20.66 nf_nat_pptp
> > Aug 17 15:01:54 192.168.20.66 nf_nat_proto_gre
> > Aug 17 15:01:54 192.168.20.66 nf_conntrack_pptp
> > Aug 17 15:01:54 192.168.20.66 nf_conntrack_proto_gre
> > Aug 17 15:01:54 192.168.20.66 iTCO_wdt
> > Aug 17 15:01:54 192.168.20.66 e752x_edac
> > Aug 17 15:01:54 192.168.20.66 edac_core
> > Aug 17 15:01:54 192.168.20.66 cls_u32
> > Aug 17 15:01:54 192.168.20.66 em_meta
> > Aug 17 15:01:54 192.168.20.66 cls_basic
> > Aug 17 15:01:54 192.168.20.66 xt_dscp
> > Aug 17 15:01:54 192.168.20.66 xt_DSCP
> > Aug 17 15:01:54 192.168.20.66 ipt_REJECT
> > Aug 17 15:01:54 192.168.20.66 ts_bm
> > Aug 17 15:01:54 192.168.20.66 xt_string
> > Aug 17 15:01:54 192.168.20.66 ipt_ttl
> > Aug 17 15:01:54 192.168.20.66 ifb
> > Aug 17 15:01:54 192.168.20.66 cls_fw
> > Aug 17 15:01:54 192.168.20.66 sch_tbf
> > Aug 17 15:01:54 192.168.20.66 sch_htb
> > Aug 17 15:01:54 192.168.20.66 act_ipt
> > Aug 17 15:01:54 192.168.20.66 act_mirred
> > Aug 17 15:01:54 192.168.20.66 xt_MARK
> > Aug 17 15:01:54 192.168.20.66 pppoe
> > Aug 17 15:01:54 192.168.20.66 pppox
> > Aug 17 15:01:54 192.168.20.66 ppp_generic
> > Aug 17 15:01:54 192.168.20.66 slhc
> > Aug 17 15:01:54 192.168.20.66 xt_TCPMSS
> > Aug 17 15:01:54 192.168.20.66 xt_mark
> > Aug 17 15:01:54 192.168.20.66 xt_tcpudp
> > Aug 17 15:01:54 192.168.20.66 iptable_mangle
> > Aug 17 15:01:54 192.168.20.66 iptable_nat
> > Aug 17 15:01:54 192.168.20.66 nf_nat
> > Aug 17 15:01:54 192.168.20.66 nf_conntrack_ipv4
> > Aug 17 15:01:54 192.168.20.66 nf_conntrack
> > Aug 17 15:01:54 192.168.20.66 rtc_cmos
> > Aug 17 15:01:54 192.168.20.66 rtc_core
> > Aug 17 15:01:54 192.168.20.66 rtc_lib
> > Aug 17 15:01:54 192.168.20.66 iptable_filter
> > Aug 17 15:01:54 192.168.20.66 ip_tables
> > Aug 17 15:01:54 192.168.20.66 x_tables
> > Aug 17 15:01:54 192.168.20.66 8021q
> > Aug 17 15:01:54 192.168.20.66 garp
> > Aug 17 15:01:54 192.168.20.66 stp
> > Aug 17 15:01:54 192.168.20.66 llc
> > Aug 17 15:01:54 192.168.20.66 tun
> > Aug 17 15:01:54 192.168.20.66 loop
> > Aug 17 15:01:54 192.168.20.66 tulip
> > Aug 17 15:01:54 192.168.20.66 r8169
> > Aug 17 15:01:54 192.168.20.66 sky2
> > Aug 17 15:01:54 192.168.20.66 via_velocity
> > Aug 17 15:01:54 192.168.20.66 via_rhine
> > Aug 17 15:01:54 192.168.20.66 sis900
> > Aug 17 15:01:54 192.168.20.66 ne2k_pci
> > Aug 17 15:01:54 192.168.20.66 8390
> > Aug 17 15:01:54 192.168.20.66 skge
> > Aug 17 15:01:54 192.168.20.66 tg3
> > Aug 17 15:01:54 192.168.20.66 libphy
> > Aug 17 15:01:54 192.168.20.66 8139too
> > Aug 17 15:01:54 192.168.20.66 e1000
> > Aug 17 15:01:54 192.168.20.66 e100
> > Aug 17 15:01:54 192.168.20.66 usb_storage
> > Aug 17 15:01:54 192.168.20.66 mtdblock
> > Aug 17 15:01:54 192.168.20.66 mtd_blkdevs
> > Aug 17 15:01:54 192.168.20.66 usbhid
> > Aug 17 15:01:54 192.168.20.66 uhci_hcd
> > Aug 17 15:01:54 192.168.20.66 ehci_hcd
> > Aug 17 15:01:54 192.168.20.66 ohci_hcd
> > Aug 17 15:01:54 192.168.20.66 usbcore
> > Aug 17 15:01:54 192.168.20.66
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Pid: 4, comm: ksoftirqd/0
> > Not tainted (2.6.27-rc3-git3-build-0031 #7) Aug 17 15:01:54 192.168.20.66
> > [ 7222.089634] EIP: 0060:[<c025fd4e>] EFLAGS: 00010206 CPU: 0 Aug 17
> > 15:01:54 192.168.20.66 [ 7222.089634] EIP is at net_tx_action+0x78/0xd8
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] EAX: 01deb6a8 EBX: 506399ee
> > ECX: 00000002 EDX: 00000001 Aug 17 15:01:54 192.168.20.66 [ 7222.089634]
> > ESI: f6429224 EDI: f6429238 EBP: c0861fd4 ESP: c0861fc4 Aug 17 15:01:54
> > 192.168.20.66 [ 7222.089634]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS:
> > 0068 Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Process ksoftirqd/0
> > (pid: 4, ti=c0861000 task=f78399b0 task.ti=f7841000) Aug 17 15:01:54
> > 192.168.20.66
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Stack:
> > Aug 17 15:01:54 192.168.20.66 506399ee
> > Aug 17 15:01:54 192.168.20.66 c039ea88
> > Aug 17 15:01:54 192.168.20.66 c085a554
> > Aug 17 15:01:54 192.168.20.66 00000003
> > Aug 17 15:01:54 192.168.20.66 c0861ff8
> > Aug 17 15:01:54 192.168.20.66 c012626b
> > Aug 17 15:01:54 192.168.20.66 c085d700
> > Aug 17 15:01:54 192.168.20.66 c085d700
> > Aug 17 15:01:54 192.168.20.66
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]
> > Aug 17 15:01:54 192.168.20.66 00000006
> > Aug 17 15:01:54 192.168.20.66 00000000
> > Aug 17 15:01:54 192.168.20.66 f7841fac
> > Aug 17 15:01:54 192.168.20.66 f7841000
> > Aug 17 15:01:54 192.168.20.66 c01261f0
> > Aug 17 15:01:54 192.168.20.66 f7841fbc
> > Aug 17 15:01:54 192.168.20.66 c01060b3
> > Aug 17 15:01:54 192.168.20.66
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Call Trace:
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c012626b>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 __do_softirq+0x7b/0xf4
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01261f0>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 __do_softirq+0x0/0xf4
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01060b3>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 do_softirq+0x65/0xb6
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c0125cfc>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 ksoftirqd+0x59/0xc8
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c0125ca3>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 ksoftirqd+0x0/0xc8
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01320e5>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 k
> > Aug 17 15:01:54 192.168.20.66 thread+0x3b/0x61
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c01320aa>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 kthread+0x0/0x61
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  [<c010451b>]
> > Aug 17 15:01:54 192.168.20.66 ?
> > Aug 17 15:01:54 192.168.20.66 kernel_thread_helper+0x7/0x10
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634]  =======================
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] Code:
> > Aug 17 15:01:54 192.168.20.66 1e
> > Aug 17 15:01:54 192.168.20.66 ec
> > Aug 17 15:01:54 192.168.20.66 ff
> > Aug 17 15:01:54 192.168.20.66 89
> > Aug 17 15:01:54 192.168.20.66 d8
> > Aug 17 15:01:54 192.168.20.66 89
> > Aug 17 15:01:54 192.168.20.66 fb
> > Aug 17 15:01:54 192.168.20.66 e8
> > Aug 17 15:01:54 192.168.20.66 6e
> > Aug 17 15:01:54 192.168.20.66 c2
> > Aug 17 15:01:54 192.168.20.66 ff
> > Aug 17 15:01:54 192.168.20.66 ff
> > Aug 17 15:01:54 192.168.20.66 85
> > Aug 17 15:01:54 192.168.20.66 db
> > Aug 17 15:01:54 192.168.20.66 75
> > Aug 17 15:01:54 192.168.20.66 d9
> > Aug 17 15:01:54 192.168.20.66 83
> > Aug 17 15:01:54 192.168.20.66 3e
> > Aug 17 15:01:54 192.168.20.66 00
> > Aug 17 15:01:54 192.168.20.66 74
> > Aug 17 15:01:54 192.168.20.66 70
> > Aug 17 15:01:54 192.168.20.66 fa
> > Aug 17 15:01:54 192.168.20.66 e8
> > Aug 17 15:01:54 192.168.20.66 e7
> > Aug 17 15:01:54 192.168.20.66 ca
> > Aug 17 15:01:54 192.168.20.66 ed
> > Aug 17 15:01:54 192.168.20.66 ff
> > Aug 17 15:01:54 192.168.20.66 8b
> > Aug 17 15:01:54 192.168.20.66 1e
> > Aug 17 15:01:54 192.168.20.66 c7
> > Aug 17 15:01:54 192.168.20.66 06
> > Aug 17 15:01:54 192.168.20.66 00
> > Aug 17 15:01:54 192.168.20.66 00
> > Aug 17 15:01:54 192.168.20.66 00
> > Aug 17 15:01:54 192.168.20.66 00
> > Aug 17 15:01:54 192.168.20.66 e8
> > Aug 17 15:01:54 192.168.20.66 c1
> > Aug 17 15:01:54 192.168.20.66 dd
> > Aug 17 15:01:54 192.168.20.66 ed
> > Aug 17 15:01:54 192.168.20.66 ff
> > Aug 17 15:01:54 192.168.20.66 fb
> > Aug 17 15:01:54 192.168.20.66 eb
> > Aug 17 15:01:54 192.168.20.66 56
> > Aug 17 15:01:54 192.168.20.66 unparseable log message: "<8b> "
> > Aug 17 15:01:54 192.168.20.66 43
> > Aug 17 15:01:54 192.168.20.66 74
> > Aug 17 15:01:54 192.168.20.66 89
> > Aug 17 15:01:54 192.168.20.66 45
> > Aug 17 15:01:54 192.168.20.66 f0
> > Aug 17 15:01:54 192.168.20.66 8d
> > Aug 17 15:01:54 192.168.20.66 73
> > Aug 17 15:01:54 192.168.20.66 24
> > Aug 17 15:01:54 192.168.20.66 f0
> > Aug 17 15:01:54 192.168.20.66 80
> > Aug 17 15:01:54 192.168.20.66 63
> > Aug 17 15:01:54 192.168.20.66 24
> > Aug 17 15:01:54 192.168.20.66 fd
> > Aug 17 15:01:54 192.168.20.66 f6
> > Aug 17 15:01:54 192.168.20.66 43
> > Aug 17 15:01:54 192.168.20.66 08
> > Aug 17 15:01:54 192.168.20.66 01
> > Aug 17 15:01:54 192.168.20.66 8d
> > Aug 17 15:01:54 192.168.20.66 7b
> > Aug 17 15:01:54 192.168.20.66 38
> > Aug 17 15:01:54 192.168.20.66
> > Aug 17 15:01:54 192.168.20.66 [ 7222.089634] EIP: [<c025fd4e>]
> > Aug 17 15:01:54 192.168.20.66 net_tx_action+0x78/0xd8
> > Aug 17 15:01:54 192.168.20.66 SS:ESP 0068:c0861fc4
> > Aug 17 15:01:54 192.168.20.66 [ 7222.094000] Kernel panic - not syncing:
> > Fatal exception in interrupt Aug 17 15:01:54 192.168.20.66 [ 7222.094626]
> > Rebooting in 5 seconds..
> >
> > On Sunday 17 August 2008, Jarek Poplawski wrote:
> > > On Sun, Aug 17, 2008 at 01:01:55PM +0300, Denys Fedoryshchenko wrote:
> > > > Ok, now things finally organized.
> > > > fixing patch #1 was missing.
> > > >
> > > > Rebooting system to this kernel. But strange thing, without fixing
> > > > patch #1 it didn't crash for whole night, and 1-3 hours now (with
> > > > same options as before).
> > > >
> > > > Let's test now complete set of patches. I will keep around 2 hours on
> > > > most loaded pppoe NAS, then distribute to 2-3 servers more if it
> > > > doesn't crash.
> > >
> > > BTW, after you complete this testing (no hurry) I would be glad if you
> > > could try one more patch which I send earlier to the list. IMHO, it's
> > > needed to fix some other locking problems. This patch could be applied
> > > and tested as an addition to all currently tested patches (but let's
> > > first be sure they really work).
> > >
> > > Thanks,
> > > Jarek P.
> > >
> > > ------------->
> > >
> > > pkt_sched: Destroy qdiscs under rtnl_lock again.
> > >
> > > We don't need to trigger __qdisc_destroy() as an RCU callback because
> > > the use of qdisc isn't controlled by RCU alone: after querying RCU
> > > with synchronize_rcu() in dev_deactivate() we additionaly wait in a
> > > loop checking some flags. After the loop is done there could be no
> > > outstanding use of the qdisc, so call_rcu() doesn't make any sense.
> > >
> > > On the other hand, current calling Qdisc's ->destroy() from a softirq
> > > context without locking (rtnl) can break various things like:
> > > qdisc_put_rtab(), tcf_destroy_chain() (e.g. u32_destroy()), and
> > > probably more.
> > >
> > >
> > > Signed-off-by: Jarek Poplawski <jarkao2@...il.com>
> > >
> > > ---
> > >
> > >  net/sched/sch_generic.c |    8 ++------
> > >  1 files changed, 2 insertions(+), 6 deletions(-)
> > >
> > > diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
> > > index 4685746..e7379d2 100644
> > > --- a/net/sched/sch_generic.c
> > > +++ b/net/sched/sch_generic.c
> > > @@ -518,12 +518,8 @@ void qdisc_reset(struct Qdisc *qdisc)
> > >  }
> > >  EXPORT_SYMBOL(qdisc_reset);
> > >
> > > -/* this is the rcu callback function to clean up a qdisc when there
> > > - * are no further references to it */
> > > -
> > > -static void __qdisc_destroy(struct rcu_head *head)
> > > +static void __qdisc_destroy(struct Qdisc *qdisc)
> > >  {
> > > -	struct Qdisc *qdisc = container_of(head, struct Qdisc, q_rcu);
> > >  	const struct Qdisc_ops  *ops = qdisc->ops;
> > >
> > >  #ifdef CONFIG_NET_SCHED
> > > @@ -554,7 +550,7 @@ void qdisc_destroy(struct Qdisc *qdisc)
> > >  	if (qdisc->parent)
> > >  		list_del(&qdisc->list);
> > >
> > > -	call_rcu(&qdisc->q_rcu, __qdisc_destroy);
> > > +	__qdisc_destroy(qdisc);
> > >  }
> > >  EXPORT_SYMBOL(qdisc_destroy);
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > > the body of a message to majordomo@...r.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > > the body of a message to majordomo@...r.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists