netdev - Re: [PATCH net] net: enforce xmit_recursion for devices with a queue

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20190410094507.GA16112@bistromath.localdomain>
Date:   Wed, 10 Apr 2019 11:45:07 +0200
From:   Sabrina Dubroca <sd@...asysnail.net>
To:     Eric Dumazet <eric.dumazet@...il.com>
Cc:     netdev@...r.kernel.org, Jianlin Shi <jishi@...hat.com>,
        Stefano Brivio <sbrivio@...hat.com>
Subject: Re: [PATCH net] net: enforce xmit_recursion for devices with a queue

Hi Eric,

2019-03-14, 10:51:49 -0700, Eric Dumazet wrote:
> On 03/14/2019 10:40 AM, Sabrina Dubroca wrote:
> > 2019-03-14, 07:56:10 -0700, Eric Dumazet wrote:
> >> On 03/14/2019 07:15 AM, Sabrina Dubroca wrote:
> >>> 2019-03-14, 05:58:03 -0700, Eric Dumazet wrote:
> >>>> On 03/14/2019 03:15 AM, Sabrina Dubroca wrote:
> >>>>> Commit 745e20f1b626 ("net: add a recursion limit in xmit path")
> >>>>> introduced a recursion limit, but it only applies to devices without a
> >>>>> queue. Virtual devices with a queue (either because they don't have
> >>>>> the IFF_NO_QUEUE flag, or because the administrator added one) can
> >>>>> still cause an unbounded recursion, via __dev_queue_xmit ->
> >>>>> __dev_xmit_skb -> qdisc_run -> __qdisc_run -> qdisc_restart ->
> >>>>> sch_direct_xmit -> dev_hard_start_xmit . Jianlin reported this in a
> >>>>> setup with 16 gretap devices stacked on top of one another.
> >>>>>
> >>>>> This patch prevents the stack overflow by incrementing xmit_recursion in
> >>>>> code paths that can call dev_hard_start_xmit() (like commit 745e20f1b626
> >>>>> did). If the recursion limit is exceeded, the packet is enqueued and the
> >>>>> qdisc is scheduled.
> >>>>>
> >>>>> Reported-by: Jianlin Shi <jishi@...hat.com>
> >>>>> Signed-off-by: Sabrina Dubroca <sd@...asysnail.net>
> >>>>> Reviewed-by: Stefano Brivio <sbrivio@...hat.com>
> >>>>
> >>>> Hi Sabrina, thanks for the patch.
> >>>>
> >>>> Can't we detect this in the control path instead ?
> >>>
> >>> I don't see how. You could have a perfectly reasonable set of gretap
> >>> devices that trigger this situation from simply reshuffling the IP
> >>> addresses:
> >>>
> >>> gretap$x remote 1.1.$((x-1)).{1,2}
> >>> (all those addresses set on a single veth device)
> >>>
> >>> Then you move those addresses to the corresponding device
> >>> (1.1.${x}.{1,2} on gretap$x), and your machine crashes.
> >>>
> >>
> >> If this only can be done with gretap, why gretap cant implement the protection,
> >> outside of the fast path ?
> > 
> > It's not just gretap. VXLAN will do the same as long as you add a
> > qdisc. I expect other types of tunnels to behave like that.
> > 
> 
> It might make sense to add a helper using dev_queue_xmit()
> for tunnel users.
> 
> Then remove the xmit recursion stuff out of the dev_queue_xmit()
> 
> Lets make the fast path fast again.

I've been looking into this. Ignoring devices that link themselves to
their upper/lower (which would be easy to handle at setup time), we're
left with IP/UDP tunnels (also easy to handle, they all xmit via a few
output functions), and then a couple of weird cases:

 - vti/xfrmi call dst_output
 - ppp can end up calling ip_local_out, ip_queue_xmit, or ip6_xmit
 - some devices call dev_queue_xmit. we'd need a recursion-limited
   wrapper for those, without putting the penalty for users that
   respect the rules (core, linked devices). I don't see a way to
   enforce that (ipv6 needs dev_queue_xmit to be EXPORT'ed), and human
   review is not perfect, so some devices could slip in using the
   wrong dev_queue_xmit.

I'm not convinced this change is worth the risk.

-- 
Sabrina