netdev - Re: [PATCH] net: limit a number of namespaces which can be cleaned up concurrently

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <871szk9rl9.fsf@x220.int.ebiederm.org>
Date:   Thu, 13 Oct 2016 10:49:38 -0500
From:   ebiederm@...ssion.com (Eric W. Biederman)
To:     Andrei Vagin <avagin@...nvz.org>
Cc:     netdev@...r.kernel.org, containers@...ts.linux-foundation.org,
        "David S. Miller" <davem@...emloft.net>
Subject: Re: [PATCH] net: limit a number of namespaces which can be cleaned up concurrently

Andrei Vagin <avagin@...nvz.org> writes:

> From: Andrey Vagin <avagin@...nvz.org>
>
> The operation of destroying netns is heavy and it is executed under
> net_mutex. If many namespaces are destroyed concurrently, net_mutex can
> be locked for a long time. It is impossible to create a new netns during
> this period of time.

This may be the right approach or at least the right approach to bound
net_mutex hold times but I have to take exception to calling network
namespace cleanup heavy.

The only particularly time consuming operation I have ever found are calls to
synchronize_rcu/sycrhonize_sched/synchronize_net.

Ideally we can search those out calls in the network namespace cleanup
operations and figuroue out how to eliminate those operations or how to
stack them.

> In our days when userns allows to create network namespaces to
> unprivilaged users, it may be a real problem.

Sorting out syncrhonize_rcu calls will be a much larger
and much more effective improvement than your patch here.

> On my laptop (fedora 24, i5-5200U, 12GB) 1000 namespaces requires about
> 300MB of RAM and are being destroyed for 8 seconds.
>
> In this patch, a number of namespaces which can be cleaned up
> concurrently is limited by 32. net_mutex is released after handling each
> portion of net namespaces and then it is locked again to handle the next
> one. It allows other users to lock it without waiting for a long
> time.
>
> I am not sure whether we need to add a sysctl to costomize this limit.
> Let me know if you think it's required.

We definitely don't need an extra sysctl.

Eric


> Cc: "David S. Miller" <davem@...emloft.net>
> Cc: "Eric W. Biederman" <ebiederm@...ssion.com>
> Signed-off-by: Andrei Vagin <avagin@...nvz.org>
> ---
>  net/core/net_namespace.c | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/net/core/net_namespace.c b/net/core/net_namespace.c
> index 989434f..33dd3b7 100644
> --- a/net/core/net_namespace.c
> +++ b/net/core/net_namespace.c
> @@ -406,10 +406,20 @@ static void cleanup_net(struct work_struct *work)
>  	struct net *net, *tmp;
>  	struct list_head net_kill_list;
>  	LIST_HEAD(net_exit_list);
> +	int i = 0;
>  
>  	/* Atomically snapshot the list of namespaces to cleanup */
>  	spin_lock_irq(&cleanup_list_lock);
> -	list_replace_init(&cleanup_list, &net_kill_list);
> +	list_for_each_entry_safe(net, tmp, &cleanup_list, cleanup_list)
> +		if (++i == 32)
> +			break;
> +	if (i == 32) {
> +		list_cut_position(&net_kill_list,
> +				  &cleanup_list, &net->cleanup_list);
> +		queue_work(netns_wq, work);
> +	} else {
> +		list_replace_init(&cleanup_list, &net_kill_list);
> +	}
>  	spin_unlock_irq(&cleanup_list_lock);
>  
>  	mutex_lock(&net_mutex);