linux-kernel - Re: Major KVM issues with kernel 4.5 on the host

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20160423184341.GA21755@torres.zugschlus.de>
Date:	Sat, 23 Apr 2016 20:43:41 +0200
From:	Marc Haber <mh+linux-kernel@...schlus.de>
To:	Borislav Petkov <bp@...en8.de>
Cc:	Paolo Bonzini <pbonzini@...hat.com>, linux-kernel@...r.kernel.org,
	kvm ML <kvm@...r.kernel.org>
Subject: Re: Major KVM issues with kernel 4.5 on the host

On Sat, Apr 23, 2016 at 06:04:29PM +0200, Borislav Petkov wrote:
> On Thu, Apr 21, 2016 at 10:04:33PM +0200, Marc Haber wrote:
> > Yes, but there are two symptoms. The VM either suffers file system
> > issues (garbage read from files, or an aborted ext4 journal and
> > following ro remount) or it stops dead in its tracks.
> 
> Stops dead? What does that mean exactly? Box is wedged solid and it
> doesn't react to any key presses?

No ping, no reaction on serial console, no reaction on virtual
console, no syslog entries.

> Because if so, this could really be a DRAM going bad and a correctable
> error turning into an uncorrectable. How old is the DRAM in that box?
> Judging by your CPU, it should be a couple of years...

Uncorrectable errors would still be identified by the ECC hardware,
and the box wouldn't be perfectly fine with an "old" kernel.

> > The box reports about one correctable error per week, so I probably
> > have a faulty DIMM, but since the issue only surfaces in VMs while the
> > host system is in perfect working order...
> 
> So it could be that correctable error turns into an uncorrectable one at
> some point. But then you should be getting an exception...

Yes, that would be in the logs.

> > And yes, I am pondering to simply replace the box with an Intel CPU.
> 
> Your CPU is fine, from what I've seen so far.

But we still postulate that the issue does only show on older AMD
CPUs. Otherwise, I wouldn't be the only one making this experience.

> > I go the way of Debian packages since it is easier to handle the
> > crypto file systems when the machine is booting up.
> 
> As long as you're testing the correct bisection kernels...

I am reasonably sure about that, yes.

> > And yes, I think about doing a test reinstall on unencrypted disk to
> > find out whether encryption plays a role, but I currently need the
> > machine to urgently to take it out of serice for half a month, and,
> > again, the host system is in perfect working order, it is just VMs
> > that barf.
> 
> Yeah, I can't reproduce it here and I have a very similar box to yours
> which is otherwise idle, more or less.
> 
> Another fact which points to potentially DIMM going bad...

Do you want me to memtest for 24 hours?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421