linux-kernel - Re: Oops in VMA code

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <96D27CEC-8492-49F2-913F-F587DEC5E95E@suse.de>
Date:	Thu, 16 Jun 2011 08:20:43 +0200
From:	Alexander Graf <agraf@...e.de>
To:	Linus Torvalds <torvalds@...ux-foundation.org>
Cc:	Benjamin Herrenschmidt <benh@...nel.crashing.org>,
	linux-mm@...ck.org,
	"linux-kernel@...r.kernel.org List" <linux-kernel@...r.kernel.org>
Subject: Re: Oops in VMA code


On 16.06.2011, at 07:59, Linus Torvalds wrote:

> On Wed, Jun 15, 2011 at 10:32 PM, Alexander Graf <agraf@...e.de> wrote:
>> 
>> 0xc000000000190580 <find_vma_prev+44>:  ld      r9,16(r9)
>> 0xc000000000190584 <find_vma_prev+48>:  mr      r26,r11
>> 0xc000000000190588 <find_vma_prev+52>:  cmpdi   cr7,r9,0
>> 0xc00000000019058c <find_vma_prev+56>:  mr      r11,r26
>> 0xc000000000190590 <find_vma_prev+60>:  beq     cr7,0xc0000000001905c4 <find_vma_prev+112>
>> 0xc000000000190594 <find_vma_prev+64>:  addi    r26,r9,-56
>> 0xc000000000190598 <find_vma_prev+68>:  ld      r0,16(r26)
>> 0xc00000000019059c <find_vma_prev+72>:  cmpld   cr7,r31,r0
>> 0xc0000000001905a0 <find_vma_prev+76>:  blt     cr7,0xc000000000190580 <find_vma_prev+44>
> 
> That's the inner loop in find_vma_prev(), and yes, it was inlined into
> do_munmap.
> 
> And the fault happens in that "ld r0,16(r26)", and it looks like you
> have memory corruption.
> 
> r26 has the value 0xc00090026236bbb0, and that "90" byte in the middle
> there looks bogus. It's not a valid pointer any more, but if that "9"
> had been a zero, it would have been.

Please see my reply to Ben here.

> So it looks like the rbtree has become corrupt, and it _looks_ like
> it's just a couple of bits that are set in what otherwise looks like a
> reasonable pointer. It *could* be a two-bit error that wasn't
> corrected (I assume you have ECC or parity on your RAM or caches), so
> it's theoretically possible that it's hardware, but generally memory
> corruption is due to software bugs, so that's a pretty far-fetched
> thing.

I'm not running on ECC memory IIRC, but this really doesn't look like a memory bit flip. Maybe somewhere else which resulted in that code to overwrite memory here, but I tend to not want to blame hardware for failures. Usually these bugs are software made :)

> At a guess, there's not a lot more to be had from the oops. The
> corruption probably came from some totally unrelated code. Without
> more of a pattern, it's pretty much impossible to even guess.
> 
> It may be that somebody can see something I'm missing, but unless you
> can find an ECC error report in your logs and say "oh, that's it", I
> suspect that you're better off ignoring it, and hoping that it will
> happen again (and again) so that we'd get enough of a pattern to start
> making any educated guesses about what's going on.
> 
> That's why I often google oops reports - one report may not give much
> of a pattern, but if google finds lots of them that all look roughly
> similar, you end up possibly seeing what the common issue is.

Yup, so let's keep this documented for now. Actually, the more I think about it the more it looks like simple random memory corruption by someone else in the kernel - and that's basically impossible to track and will give completely different bugs next time around :(.

Either way, thanks a lot for looking at it!


Alex

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/