linux-kernel - Re: KVM: Softlockups in guests while running kernbench

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <87ipg4drgd.fsf@abhimanyu.in.ibm.com>
Date:	Thu, 10 May 2012 20:16:26 +0530
From:	Nikunj A Dadhania <nikunj@...ux.vnet.ibm.com>
To:	Avi Kivity <avi@...hat.com>, Ingo Molnar <mingo@...e.hu>,
	Peter Zijlstra <peterz@...radead.org>
Cc:	Andrea Arcangeli <aarcange@...hat.com>, KVM <kvm@...r.kernel.org>,
	LKML <linux-kernel@...r.kernel.org>
Subject: Re: KVM: Softlockups in guests while running kernbench

On Thu, 10 May 2012 15:39:04 +0530, Nikunj A Dadhania <nikunj@...ux.vnet.ibm.com> wrote:
> On Thu, 10 May 2012 12:22:00 +0300, Avi Kivity <avi@...hat.com> wrote:
> > On 05/10/2012 11:15 AM, Nikunj A Dadhania wrote:
> > > I am running a 3.4.0-rc4 based kernel(commit: af3a3ab2), guest config
> > > attached.
> > >
> > > During my tests, I saw few softlockups inside the guests while running
> > > kernbench inside the guest. I can reproduce this repeatedly.
> > >
> > > Test Setup:
> > >
> > > - Create 4 VMs (8vcpu, 1GB RAM)
> > > - Run kernbench inside each guests (kernbench -f -M -H -o 16) in parallel
> > >
> > 
> > How overcommitted are you?  
> 8 Physical CPU and 32 vCPUs
> 
> > What does the host hardware look like?  
> Machine : IBM xSeries with Intel(R) Xeon(R) x5570 2.93GHz CPU with 8 core , 64GB RAM
> Threading is disabled.
> 
> > Is there any load on the host?
> No
> 
> > 
> > The traces themselves always point a couple of instructions after
> > interrupts are enabled.  Can you post a few more?
> > 

I had a discussion with Avi on IRC, he suggested running a trace on the
host for the sched: tracepoints

So when I see the console of the guest, i can see the soft-lockup
messages. When I start tracing, soft-lockups disappear. 

Kernbench is started from the host; ps output:
13359 pts/1    S+     0:00 perf stat -a -- ssh root@....168.123.11 cd /root/linux-src; kernbench -f -M -H -o 16 > /root/kbench.log
13362 pts/1    S+     0:00 ssh root@....168.123.11 cd /root/linux-src; kernbench -f -M -H -o 16 > /root/kbench.log
13373 pts/1    S+     0:00 perf stat -a -- ssh root@....168.123.12 cd /root/linux-src; kernbench -f -M -H -o 16 > /root/kbench.log
13376 pts/1    S+     0:00 ssh root@....168.123.12 cd /root/linux-src; kernbench -f -M -H -o 16 > /root/kbench.log
13387 pts/1    S+     0:00 perf stat -a -- ssh root@....168.123.13 cd /root/linux-src; kernbench -f -M -H -o 16 > /root/kbench.log
13390 pts/1    S+     0:00 ssh root@....168.123.13 cd /root/linux-src; kernbench -f -M -H -o 16 > /root/kbench.log
13401 pts/1    S+     0:00 perf stat -a -- ssh root@....168.123.14 cd /root/linux-src; kernbench -f -M -H -o 16 > /root/kbench.log
13402 pts/1    S+     0:00 ssh root@....168.123.14 cd /root/linux-src; kernbench -f -M -H -o 16 > /root/kbench.log

I have tried following commands trace-cmd:
1) trace-cmd record -e kvm -e sched -b 100000  -p 13362
2) trace-cmd record -e sched -b 100000  -p 13362
3) trace-cmd record -e kvm -b 100000  -p 13362

I have tried trace-cmd on qemu pid as well, I see similar behaviour in
that case.

Peterz/Ingo, Any clue on debugging this would help.

Regards
Nikunj

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/