linux-kernel - Re: [PATCH bpf] bpf: cgroup: prevent out-of-order release of cgroup bpf

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20200106222042.GA18722@tower.dhcp.thefacebook.com>
Date:   Mon, 6 Jan 2020 22:20:46 +0000
From:   Roman Gushchin <guro@...com>
To:     Alexei Starovoitov <alexei.starovoitov@...il.com>
CC:     "tj@...nel.org" <tj@...nel.org>,
        "bpf@...r.kernel.org" <bpf@...r.kernel.org>,
        Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        Kernel Team <Kernel-team@...com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "stable@...r.kernel.org" <stable@...r.kernel.org>
Subject: Re: [PATCH bpf] bpf: cgroup: prevent out-of-order release of cgroup
 bpf

On Mon, Jan 06, 2020 at 02:07:48PM -0800, Alexei Starovoitov wrote:
> On Sat, Jan 04, 2020 at 03:00:46AM +0000, Roman Gushchin wrote:
> > On Fri, Jan 03, 2020 at 06:31:14PM -0800, Alexei Starovoitov wrote:
> > > On Sat, Jan 04, 2020 at 01:13:24AM +0000, Roman Gushchin wrote:
> > > > On Fri, Jan 03, 2020 at 04:35:25PM -0800, Alexei Starovoitov wrote:
> > > > > On Fri, Dec 27, 2019 at 01:50:34PM -0800, Roman Gushchin wrote:
> > > > > > Before commit 4bfc0bb2c60e ("bpf: decouple the lifetime of cgroup_bpf
> > > > > > from cgroup itself") cgroup bpf structures were released with
> > > > > > corresponding cgroup structures. It guaranteed the hierarchical order
> > > > > > of destruction: children were always first. It preserved attached
> > > > > > programs from being released before their propagated copies.
> > > > > > 
> > > > > > But with cgroup auto-detachment there are no such guarantees anymore:
> > > > > > cgroup bpf is released as soon as the cgroup is offline and there are
> > > > > > no live associated sockets. It means that an attached program can be
> > > > > > detached and released, while its propagated copy is still living
> > > > > > in the cgroup subtree. This will obviously lead to an use-after-free
> > > > > > bug.
> > > > > ...
> > > > > > @@ -65,6 +65,9 @@ static void cgroup_bpf_release(struct work_struct *work)
> > > > > >  
> > > > > >  	mutex_unlock(&cgroup_mutex);
> > > > > >  
> > > > > > +	for (p = cgroup_parent(cgrp); p; p = cgroup_parent(p))
> > > > > > +		cgroup_bpf_put(p);
> > > > > > +
> > > > > 
> > > > > The fix makes sense, but is it really safe to walk cgroup hierarchy
> > > > > without holding cgroup_mutex?
> > > > 
> > > > It is, because we're holding a reference to the original cgroup and going
> > > > towards the root. On each level the cgroup is protected by a reference
> > > > from their child cgroup.
> > > 
> > > cgroup_bpf_put(p) can make bpf.refcnt zero which may call cgroup_bpf_release()
> > > on another cpu which will do cgroup_put() and this cpu p = cgroup_parent(p)
> > > would be use-after-free?
> > > May be not due to the way work_queues are implemented.
> > > But it feels dangerous to have such delicate release logic.
> > 
> > If I understand your concern correctly: you assume that parent's
> > cgroup_bpf_release() can be finished prior to the child's one and
> > the final cgroup_put() will release the parent?
> > 
> > If so, it's not possible, because the child hold a reference to the
> > parent (independent to all cgroup bpf stuff), which exists at least
> > until the final cgroup_put() in cgroup_bpf_release(). Please, look
> > at css_free_rwork_fn() for details.
> > 
> > > Why not to move the loop under the mutex and make things obvious?
> > 
> > Traversing the cgroup tree to the root cgroup without additional
> > locking seems pretty common to me. You can find a ton of examples in
> > mm/memcontrol.c. So it doesn't look scary or adventurous to me.
> > 
> > I think it doesn't matter that much here, so I'm ok with putting it
> > under the mutex, but IMO it won't make the code any safer.
> > 
> > 
> > cc Tejun for the second opinion on cgroup locking
> 
> Checked with TJ offline. This seems fine.
> 
> I tweaked commit log:
> - extra 'diff' lines were confusing 'git am'
> - commit description shouldn't be split into multiline

Hm, I thought we don't break it only on the "Fixes:" line. Maybe it's
subtree-dependent :)

> 
> And applied to bpf tree. Thanks

Thank you!