netdev - Re: [ipvs] BUG: soft lockup detected on CPU#3!

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20070529010824.GB3873@verge.net.au>
Date:	Tue, 29 May 2007 10:08:25 +0900
From:	Horms <horms@...ge.net.au>
To:	Sebastien Estienne <sebastien.estienne@...il.com>
Cc:	wensong@...ux-vs.org, ja@....bg, nedev <netdev@...r.kernel.org>
Subject: Re: [ipvs] BUG: soft lockup detected on CPU#3!

On Mon, May 28, 2007 at 04:20:32PM +0200, Sebastien Estienne wrote:
> On 5/28/07, Horms <horms@...ge.net.au> wrote:
> >On Sat, May 26, 2007 at 11:22:40AM +0900, Horms wrote:
> >> On Fri, May 25, 2007 at 09:30:52AM +0000, Sebastien Estienne wrote:
> >> >
> >> > I didn't try 2.6.21 yet, but using ubuntu dapper kernel (2.6.15) i
> >> > can't reproduce the bug.
> >> > When i was using feisty kernel (2.6.20), i can reproduce in less than 5
> >> > minutes.
> >> >
> >> > I'm using lvs to loadbalance some mysql servers, i wrote a deamon that
> >> > check the synchro of the mysql replication on each slave and adjust
> >> > the wieght on the lvs every 500ms
> >>
> >> It does look a lot like there is some sort of locking problem in there.
> >> Would it be possible to send your kernel config, as the locking
> >> deatails to change a little with different configs.
> >
> 
> About the kernel .config, i'm using the vanilla kernel "-server" from
> ubuntu feisty
> 
> >If you also have some details of you ipvs configuration,
> >that might help narrow down which code-paths to investigate.
> >
> 
> i attached the output of ipvsadm-save
> 
> i'm adjusting the weight every 500ms by generating lines like this:
> -e -t 10.33.1.231:3306 -r 10.33.1.1 -w 100

Thanks.

> and piping all the needed changes in ipvadm -R
> 
> it can represent something like 20 to 40 updates in one time.
> 
> i also noticed that sometimes when i execute "ipvsadm" the display get
> locked in the middle for a second and then finish.

That is probably related. Though the fact that it eventually
exits seems to indicate that you're not hitting II.

> >II.
> >
> >ip_vs_set_ctl() does seem to leak svc->usecnt in one corner case,
> >but I doubt that is what you are seeing - if it was your ipvsadm
> >command(s) would hang. The problem is a bit wordy to describe,
> >but this fix should illustrate the problem.
> >
> >--- linux-2.6.orig/net/ipv4/ipvs/ip_vs_ctl.c
> >+++ linux-2.6/net/ipv4/ipvs/ip_vs_ctl.c
> >@@ -2000,7 +2000,7 @@ do_ip_vs_set_ctl(struct sock *sk, int cm
> >        if (cmd != IP_VS_SO_SET_ADD
> >            && (svc == NULL || svc->protocol != usvc->protocol)) {
> >                ret = -ESRCH;
> >-               goto out_unlock;
> >+               goto out_svc;
> >        }
> >
> >        switch (cmd) {
> >@@ -2034,9 +2034,9 @@ do_ip_vs_set_ctl(struct sock *sk, int cm
> >                ret = -EINVAL;
> >        }
> >
> >+  out_svc:
> >        if (svc)
> >                ip_vs_service_put(svc);
> >-
> >   out_unlock:
> >        mutex_unlock(&__ip_vs_mutex);
> >   out_dec:

-- 
Horms
  H: http://www.vergenet.net/~horms/
  W: http://www.valinux.co.jp/en/

-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html