netdev - Re: Race condition when creating multiple namespaces?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <m1ei58co08.fsf@fess.ebiederm.org>
Date:	Mon, 11 Apr 2011 17:27:35 -0700
From:	ebiederm@...ssion.com (Eric W. Biederman)
To:	Hans Schillstrom <hans@...illstrom.com>
Cc:	netdev@...r.kernel.org, Daniel Lezcano <daniel.lezcano@...e.fr>
Subject: Re: Race condition when creating multiple namespaces?

Hans Schillstrom <hans@...illstrom.com> writes:

> Hello
> I'v been strugling with this for some time now
>
> When creating multiple namespaces using lxc-start,  un-initialized network namespace parts will be called by the new process in the namespace.
> ex. when using conntrack or ipvsadm to quickly,  (a sleep 2 "solves" the problem).
> (From what I can see syscall clone() is used in lx-start  i.e. do_fork will be called later on.)
> Actually I was debugging ip_vs when closing multiple ns  when I fell into this one.
>
> I have a loop that create 33 containers whith lxc-start ... -- test.sh
> the first thing the new conatiner does in test.sh is
> #!/bin/bash
> iptables -t mangle -A PREROUTING -m conntrack --ctstate RELATED,ESTABLISHED -j CONNMARK --restore-mark
> nc -l -p1234
>
> This results in NULL ptr in ip_conntrack_net_init(struct *net)

Ouch!

> and in anoither test test.sh looks like this
> #!/bin/bash
> ipvsadm --start-daemon=master --mcast-interface=lo
> nc -l -p1234
>
> And this results in an uniitialized spinlock in ip_vs_sync
>
> I put a printk in nsproxy: copy_namespaces() and could see a dozens of them
> before anything appears from ipvs or conntrack.
>
> My feeling is that when you start up user processes in a new name space, 
> all kernel related init should have been done (you should not need to add a sleep to get it working)
>
> All test  made by using todays net-next-2.6 (2.6.39-rc1)
>
> Note:
> That neither conntrack or ip_vs modules where loaded,
> if modules where loaded before creating new namespaces it all works...
>
> Finally the question,
> Should it really work to load modules within a namespace , 
> that is a part of netns ?

>From an implementation point of view kernel modules are not in a
namespace, so there should be no difference between being in a namespace
and loading a kernel networking module and not being in a namespace and
loading in a kernel module.

It does sound like you have hit a module loading race, and perhaps
a race that is confined to network namespaces.

My head is in another problem so I won't be able to look at this for
a bit.  But if you are getting into ip_conntrack_net_init with
a NULL network namespace something spectacularly bad is happening.

In particular it looks like you must be hitting a bug in for_each_net.
Which would pretty much have to be a race in adding or removing from
net_namespace_list.

I took a quick skim through the code and whenever we modify the
net_namespace we hold but the net_mutex and inside it the rtnl_lock so I
don't immediate see how you could be getting a NULL net into
ip_conntrack_net_init.

Is there a codepath besides register_pernet_subsys that is calling
ip_conntrack_net_init?

Do you have any local modifications that could be messing up register_pernet_subsys?

Eric
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html