netdev - Re: [PATCH 1/1] enic: prevent waking up stopped tx queues over watchdog reset

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20200212.094356.734697556409905980.davem@davemloft.net>
Date:   Wed, 12 Feb 2020 09:43:56 -0800 (PST)
From:   David Miller <davem@...emloft.net>
To:     firo.yang@...e.com
Cc:     netdev@...r.kernel.org, pkaustub@...co.com, _govind@....com,
        benve@...co.com, firogm@...il.com
Subject: Re: [PATCH 1/1] enic: prevent waking up stopped tx queues over
 watchdog reset

From: Firo Yang <firo.yang@...e.com>
Date: Wed, 12 Feb 2020 06:09:17 +0100

> Recent months, our customer reported several kernel crashes all
> preceding with following message:
> NETDEV WATCHDOG: eth2 (enic): transmit queue 0 timed out
> Error message of one of those crashes:
> BUG: unable to handle kernel paging request at ffffffffa007e090
> 
> After analyzing severl vmcores, I found that most of crashes are
> caused by memory corruption. And all the corrupted memory areas
> are overwritten by data of network packets. Moreover, I also found
> that the tx queues were enabled over watchdog reset.
> 
> After going through the source code, I found that in enic_stop(),
> the tx queues stopped by netif_tx_disable() could be woken up over
> a small time window between netif_tx_disable() and the
> napi_disable() by the following code path:
> napi_poll->
>   enic_poll_msix_wq->
>      vnic_cq_service->
>         enic_wq_service->
>            netif_wake_subqueue(enic->netdev, q_number)->
>               test_and_clear_bit(__QUEUE_STATE_DRV_XOFF, &txq->state)
> In turn, upper netowrk stack could queue skb to ENIC NIC though
> enic_hard_start_xmit(). And this might introduce some race condition.
> 
> Our customer comfirmed that this kind of kernel crash doesn't occur over
> 90 days since they applied this patch.
> 
> Signed-off-by: Firo Yang <firo.yang@...e.com>

Applied and queued up for -stable, thanks.