linux-kernel - Re: Hang in 5.17.4+ that appears to be due to Xen

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <YoaJ6KB+GRXHG3Mv@itl-email>
Date:   Thu, 19 May 2022 14:17:46 -0400
From:   Demi Marie Obenour <demi@...isiblethingslab.com>
To:     Juergen Gross <jgross@...e.com>,
        Xen developer discussion <xen-devel@...ts.xenproject.org>
Cc:     Boris Ostrovski <boris.ostrovsky@...cle.com>,
        Marek Marczykowski-Górecki 
        <marmarek@...isiblethingslab.com>, linux-kernel@...r.kernel.org,
        Jani Nikula <jani.nikula@...ux.intel.com>,
        Joonas Lahtinen <joonas.lahtinen@...ux.intel.com>,
        Rodrigo Vivi <rodrigo.vivi@...el.com>,
        Tvrtko Ursulin <tvrtko.ursulin@...ux.intel.com>,
        David Airlie <airlied@...ux.ie>,
        Daniel Vetter <daniel@...ll.ch>,
        Intel Graphics Development <intel-gfx@...ts.freedesktop.org>,
        DRI Development <dri-devel@...ts.freedesktop.org>
Subject: Re: Hang in 5.17.4+ that appears to be due to Xen

On Mon, May 16, 2022 at 08:48:17AM +0200, Juergen Gross wrote:
> On 14.05.22 17:55, Demi Marie Obenour wrote:
> > In https://github.com/QubesOS/qubes-issues/issues/7481, a user reported
> > that Xorg locked up when resizing a VM window.  While I do not have the
> > same hardware the user does and thus cannot reproduce the bug, the stack
> > trace seems to indicate a deadlock between xen_gntdev and i915.  It
> > appears that gnttab_unmap_refs_sync() is waiting for i915 to free the
> > pages, while i915 is waiting for the MMU notifier that called
> > gnttab_unmap_refs_sync() to return.  Result: deadlock.
> > 
> > The problem appears to be that a mapped grant in PV mode will stay in
> > the “invalidating” state until it is freed.  While MMU notifiers are
> > allowed to sleep, it appears that they cannot wait for the page to be
> > freed, as is happening here.  That said, I am not very familiar with
> > this code, so my diagnosis might be incorrect.
> 
> All I can say for now is that your patch seems to be introducing a use after
> free issue, as the parameters of the delayed work might get freed now before
> the delayed work is being executed.
> 
> I don't know why this is happening only with rather recent kernels, as the
> last gntdev changes in this area have been made in kernel 4.13.
> 
> I'd suggest to look at i915, as quite some work has happened in the code
> visible in your stack backtraces rather recently. Maybe it would be possible
> to free the pages in i915 before calling the MMU notifier?

Honestly, I would rather fix this in gntdev, regardless of where the
actual bug lies.  GPU drivers get little testing under Xen, so if
something like this happens again, it is likely to remain undiscovered
until a Qubes user files a bug report.  This results in a bad experience
for Qubes users.  I would much rather code that works on bare metal work
in Xen dom0 too.  I have had random hangs in the past (with various
kernel versions) that might be due to similar problems.

Furthermore, similar problems can arise whenever a driver removes an
MMU notifier on userspace pages that someone else has references to.  It
is hard for me to see how this is the fault of the driver that removed
the MMU notifier.  I find it much more plausible that the correct fix is
on the Xen side: allocate the delayed work parameters on the heap, and
free them after the work is finished.  This eliminates this entire class
of bugs.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

Download attachment "signature.asc" of type "application/pgp-signature" (834 bytes)