linux-kernel - Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAJE_dJw-HhRqrr-9cO-EevLa6RWin3gMMutYZLE9CwPFA+dHfA@mail.gmail.com>
Date:	Wed, 11 Jun 2014 18:03:13 -0300
From:	Rafael Tinoco <rafael.tinoco@...onical.com>
To:	"Eric W. Biederman" <ebiederm@...ssion.com>
Cc:	Paul McKenney <paulmck@...ux.vnet.ibm.com>,
	Dave Chiluk <chiluk@...onical.com>,
	linux-kernel@...r.kernel.org, davem@...emloft.net,
	Christopher Arges <chris.j.arges@...onical.com>,
	Jay Vosburgh <jay.vosburgh@...onical.com>
Subject: Re: Possible netns creation and execution performance/scalability
 regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

Eric,

I'll test the patch with the same testcase and let you all know.

Really appreciate everybody's efforts.

On Wed, Jun 11, 2014 at 5:55 PM, Eric W. Biederman
<ebiederm@...ssion.com> wrote:
> "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com> writes:
>
>> On Wed, Jun 11, 2014 at 01:27:07PM -0500, Dave Chiluk wrote:
>>> On 06/11/2014 11:18 AM, Paul E. McKenney wrote:
>>> > On Wed, Jun 11, 2014 at 10:46:00AM -0500, David Chiluk wrote:
>>> >> Now think about what happens when a gateway goes down, the namespaces
>>> >> need to be migrated, or a new machine needs to be brought up to replace
>>> >> it.  When we're talking about 3000 namespaces, the amount of time it
>>> >> takes simply to recreate the namespaces becomes very significant.
>>> >>
>>> >> The script is a stripped down example of what exactly is being done on
>>> >> the neutron gateway in order to create namespaces.
>>> >
>>> > Are the namespaces torn down and recreated one at a time, or is there some
>>> > syscall, ioctl(), or whatever that allows bulk tear down and recreating?
>>> >
>>> >                                                    Thanx, Paul
>>>
>>> In the normal running case, the namespaces are created one at a time, as
>>> new customers create a new set of VMs on the cloud.
>>>
>>> However, in the case of failover to a new neutron gateway the namespaces
>>> are created all at once using the ip command (more or less serially).
>>>
>>> As far as I know there is no syscall or ioctl that allows bulk tear down
>>> and recreation.  if such a beast exists that might be helpful.
>>
>> The solution might be to create such a beast.  I might be able to shave
>> a bit of time off of this benchmark, but at the cost of significant
>> increases in RCU's CPU consumption.  A bulk teardown/recreation API could
>> reduce the RCU grace-period overhead by several orders of magnitude by
>> having a single RCU grace period cover a few thousand changes.
>>
>> This is why other bulk-change syscalls exist.
>>
>> Just out of curiosity, what syscalls does the ip command use?
>
> You can look in iproute2 ip/ipnetns.c
>
> But rought ip netns add does:
>
> unshare(CLONE_NEWNET);
> mkdir /var/run/netns/<name>
> mount --bind /proc/self/ns/net /var/run/netns/<name>
>
> I don't know if there is any sensible way to batch that work.
>
> (The unshare gets you into copy_net_ns in net/core/net_namespace.c
>  and to find all of the code it can call you have to trace all
>  of the register_pernet_subsys and register_pernet_device calls).
>
> At least for creation I would like to see if we can make all of the
> rcu_callback synchronize_rcu calls go away.  That seems preferable
> to batching at creation time.
>
> Eric



-- 
-- 
Rafael David Tinoco
Software Sustaining Engineer @ Canonical
Canonical Technical Services Engineering Team
# Email: rafael.tinoco@...onical.com (GPG: 87683FC0)
# Phone: +55.11.9.6777.2727 (Americas/Sao_Paulo)
# LP: ~inaddy | IRC: tinoco | Skype: rafael.tinoco
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/