netdev - Re: [RFCv3 PATCH 1/6] uacce: Add documents for WarpDrive/uacce

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Mon, 19 Nov 2018 21:02:29 +0200
From:   Leon Romanovsky <leon@...nel.org>
To:     Jerome Glisse <jglisse@...hat.com>
Cc:     Jason Gunthorpe <jgg@...pe.ca>,
        Kenneth Lee <liguozhu@...ilicon.com>,
        Tim Sell <timothy.sell@...sys.com>, linux-doc@...r.kernel.org,
        Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
        Zaibo Xu <xuzaibo@...wei.com>, zhangfei.gao@...mail.com,
        linuxarm@...wei.com, haojian.zhuang@...aro.org,
        Christoph Lameter <cl@...ux.com>,
        Hao Fang <fanghao11@...wei.com>,
        Gavin Schenk <g.schenk@...elmann.de>,
        RDMA mailing list <linux-rdma@...r.kernel.org>,
        Zhou Wang <wangzhou1@...ilicon.com>,
        Doug Ledford <dledford@...hat.com>,
        Uwe Kleine-König 
        <u.kleine-koenig@...gutronix.de>,
        David Kershner <david.kershner@...sys.com>,
        Kenneth Lee <nek.in.cn@...il.com>,
        Johan Hovold <johan@...nel.org>,
        Cyrille Pitchen <cyrille.pitchen@...e-electrons.com>,
        Sagar Dharia <sdharia@...eaurora.org>,
        Jens Axboe <axboe@...nel.dk>, guodong.xu@...aro.org,
        linux-netdev <netdev@...r.kernel.org>,
        Randy Dunlap <rdunlap@...radead.org>,
        linux-kernel@...r.kernel.org, Vinod Koul <vkoul@...nel.org>,
        linux-crypto@...r.kernel.org,
        Philippe Ombredanne <pombredanne@...b.com>,
        Sanyog Kale <sanyog.r.kale@...el.com>,
        "David S. Miller" <davem@...emloft.net>,
        linux-accelerators@...ts.ozlabs.org
Subject: Re: [RFCv3 PATCH 1/6] uacce: Add documents for WarpDrive/uacce

On Mon, Nov 19, 2018 at 01:42:16PM -0500, Jerome Glisse wrote:
> On Mon, Nov 19, 2018 at 11:27:52AM -0700, Jason Gunthorpe wrote:
> > On Mon, Nov 19, 2018 at 11:48:54AM -0500, Jerome Glisse wrote:
> >
> > > Just to comment on this, any infiniband driver which use umem and do
> > > not have ODP (here ODP for me means listening to mmu notifier so all
> > > infiniband driver except mlx5) will be affected by same issue AFAICT.
> > >
> > > AFAICT there is no special thing happening after fork() inside any of
> > > those driver. So if parent create a umem mr before fork() and program
> > > hardware with it then after fork() the parent might start using new
> > > page for the umem range while the old memory is use by the child. The
> > > reverse is also true (parent using old memory and child new memory)
> > > bottom line you can not predict which memory the child or the parent
> > > will use for the range after fork().
> > >
> > > So no matter what you consider the child or the parent, what the hw
> > > will use for the mr is unlikely to match what the CPU use for the
> > > same virtual address. In other word:
> > >
> > > Before fork:
> > >     CPU parent: virtual addr ptr1 -> physical address = 0xCAFE
> > >     HARDWARE:   virtual addr ptr1 -> physical address = 0xCAFE
> > >
> > > Case 1:
> > >     CPU parent: virtual addr ptr1 -> physical address = 0xCAFE
> > >     CPU child:  virtual addr ptr1 -> physical address = 0xDEAD
> > >     HARDWARE:   virtual addr ptr1 -> physical address = 0xCAFE
> > >
> > > Case 2:
> > >     CPU parent: virtual addr ptr1 -> physical address = 0xBEEF
> > >     CPU child:  virtual addr ptr1 -> physical address = 0xCAFE
> > >     HARDWARE:   virtual addr ptr1 -> physical address = 0xCAFE
> >
> > IIRC this is solved in IB by automatically calling
> > madvise(MADV_DONTFORK) before creating the MR.
> >
> > MADV_DONTFORK
> >   .. This is useful to prevent copy-on-write semantics from changing the
> >   physical location of a page if the parent writes to it after a
> >   fork(2) ..
>
> This would work around the issue but this is not transparent ie
> range marked with DONTFORK no longer behave as expected from the
> application point of view.
>
> Also it relies on userspace doing the right thing (which is not
> something i usualy trust :)).

The good thing that we didn't see anyone who succeeded to run
IB stack without our user space, which does right thing under
the hood :).

>
> Cheers,
> Jérôme

Download attachment "signature.asc" of type "application/pgp-signature" (802 bytes)