netdev - Re: XDP_TX bug report on mlx4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20160918235943.GA18685@gmail.com>
Date:   Sun, 18 Sep 2016 19:59:43 -0400
From:   Brenden Blanco <bblanco@...mgrid.com>
To:     Jesper Dangaard Brouer <brouer@...hat.com>
Cc:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Tariq Toukan <tariqt@...lanox.com>,
        Tom Herbert <tom@...bertland.com>,
        Saeed Mahameed <saeedm@...lanox.com>,
        Rana Shahout <rana.shahot@...il.com>,
        Eran Ben Elisha <eranbe@...lanox.com>
Subject: Re: XDP_TX bug report on mlx4

On Fri, Sep 16, 2016 at 09:24:43PM +0200, Jesper Dangaard Brouer wrote:
> On Fri, 16 Sep 2016 12:17:27 -0700
> Brenden Blanco <bblanco@...mgrid.com> wrote:
> 
> > On Fri, Sep 16, 2016 at 09:03:40PM +0200, Jesper Dangaard Brouer wrote:
> > > Hi Brenden,
> > > 
> > > I've discovered a bug with XDP_TX recycling of pages in the mlx4 driver.
> > > 
> > > If I increase the number of RX and TX queues/channels via ethtool cmd:
> > >  ethtool -L mlx4p1 rx 10 tx 10
> > > 
> > > Then when running the xdp2 program, which does XDP_TX, the kernel will
> > > crash with page errors, because the page refcnt goes to zero or even
> > > minus.  I've noticed pages delivered to mlx4_en_rx_recycle() can have
> > > a page refcnt of zero, which is wrong, they should always have 1 (for
> > > XDP).
> > > 
> > > Debugging it further, I find that this can happen when mlx4_en_rx_recycle()
> > > is called from mlx4_en_recycle_tx_desc().  This is the TX cleanup function,
> > > associated with TX ring queues used for XDP_TX only. No others than the
> > > XDP_TX action should be able to place packets into these TX rings
> > > which call mlx4_en_recycle_tx_desc().  
> > 
> > Sounds pretty straightforward, let me look into it.
> 
> Here is some debug info I instrumented my kernel with, and I've
> attached my minicom output with a warning and a panic.

Thanks for the info.

I've also spent this weekend trying to debug (pretty easy to reproduce),
but with no conclusive answer. I was investigating the sequence in
mlx4_en_stop_port to see if rx might still be running through the
function, on the theory that the xdp tx might race with
mlx4_en_free_tx_buf. For instance, I tried moving the napi_synchronize
loop to be just below the msleep(10). No improvement.

Unfortunately, I'm out of options, since my one test box has decided not
to reboot itself automatically, and I'll be out of email communication
(for 3 weeks) before anybody can physicially resuscitate it (tomorrow).
> 
> Enable some driver debug printks via::
>  ethtool -s mlx4p1 msglvl drv on
> 
> Debug normal situation::
> 
[...]