netdev - Re: kernel panic in skb_copy

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1372666283.14691.8.camel@zakaz.uk.xensource.com>
Date:	Mon, 1 Jul 2013 09:11:23 +0100
From:	Ian Campbell <Ian.Campbell@...rix.com>
To:	Joe Jin <joe.jin@...cle.com>
CC:	Alex Bligh <alex@...x.org.uk>,
	Eric Dumazet <eric.dumazet@...il.com>,
	Frank Blaschka <frank.blaschka@...ibm.com>,
	"David S. Miller" <davem@...emloft.net>,
	<linux-kernel@...r.kernel.org>, <netdev@...r.kernel.org>,
	<zheng.x.li@...cle.com>, Xen Devel <xen-devel@...ts.xen.org>,
	Jan Beulich <JBeulich@...e.com>,
	"Stefano Stabellini" <stefano.stabellini@...citrix.com>,
	Konrad Rzeszutek Wilk <konrad.wilk@...cle.com>
Subject: Re: kernel panic in skb_copy_bits

On Mon, 2013-07-01 at 11:18 +0800, Joe Jin wrote:
> > A workaround is to turn off O_DIRECT use by Xen as that ensures
> > the pages are copied. Xen 4.3 does this by default.
> > 
> > I believe fixes for this are in 4.3 and 4.2.2 if using the
> > qemu upstream DM. Note these aren't real fixes, just a workaround
> > of a kernel bug.
> 
> The guest is pvm, and disk model is xvbd, guest config file as below:

Do you know which disk backend? The workaround Alex refers to went into
qdisk but I think blkback could still suffer from a variant of the
retransmit issue if you run it over iSCSI.

> > To fix on a local build of xen you will need something like this:
> > https://github.com/abligh/qemu-upstream-4.2-testing/commit/9a97c011e1a682eed9bc7195a25349eaf23ff3f9
> > and something like this (NB: obviously insert your own git
> > repo and commit numbers)
> > https://github.com/abligh/xen/commit/f5c344afac96ced8b980b9659fb3e81c4a0db5ca
> > 
> 
> I think this only for pvhvm/hvm?

No, the underlying issue affects any PV device which is run over a
network protocol (NFS, iSCSI etc). In effect a delayed retransmit can
cross over the deayed ack and cause I/O to be completed while
retransmits are pending, such as is described in
http://www.spinics.net/lists/linux-nfs/msg34913.html (the original NFS
variant). The problem is that because Xen PV drivers often unmap the
page on I/O completion you get a crash (page fault) on the retransmit.

The issue also affects native but in that case the symptom is "just" a
corrupt packet on the wire. I tried to address this with my "skb
destructor" series but unfortunately I got bogged down on the details,
then I had to take time out to look into some other stuff and never
managed to get back into it. I'd be very grateful if there was someone
who could pick up that work (Alex gave some useful references in another
reply to this thread)

Some PV disk backends (e.g. blktap2) have worked around this by using
grant copy instead of grant map, others (e.g. qdisk) have disabled
O_DIRECT so that the pages are copied into the dom0 page cache and
transmitted from there.

We were discussing recently the possibility of mapping all ballooned out
pages to a single read-only scratch page instead of leaving them empty
in the page tables, this would cause the Xen case to revert to the
native case. I think Thanos was going to take a look into this.

Ian.

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html