lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Date: Sat, 07 May 2011 14:22:06 +0200 From: Eric Dumazet <eric.dumazet@...il.com> To: Alex Bligh <alex@...x.org.uk> Cc: netdev@...r.kernel.org Subject: Re: Scalability of interface creation and deletion Le samedi 07 mai 2011 à 12:08 +0100, Alex Bligh a écrit : > I am trying to track down why interface creation slows down badly with > large numbers of interfaces (~1,000 interfaces) and why deletion is so > slow. Use case: restarting routers needs to be fast; some failover methods > require interface up/down; some routers need lots of interfaces. > > I have written a small shell script to create and delete a number of > interfaces supplied on the command line (script appended below). It > is important to run this with udev, udev-bridge etc. disabled. In > my environment > (Ubuntu 2.6.32-28-generic, Lucid). I did this by > * service upstart-udev-bridge stop > * service udev stop > * unshare -n bash > If you don't do this, you are simply timing your distro's interface > scripts. > > Note the "-n" parameter creates the supplied number of veth pair > interfaces. As these are pairs, there are twice as many interfaces actually > created. > > So, the results which are pretty repeatable are as follows: > > 100 pairs 500 pairs > Interface creation 14ms 110ms > Interface deletion 160ms 148ms > > Now I don't think interface deletion has in fact got faster: simply > the overhead of loading the script is spread over more processes. > But there are two obvious conclusions: > > 1. Interface creation slows down hugely with more interfaces sysfs is the problem, a very well known one. (sysfs_refresh_inode(), try : $ time ls /sys/class/net >/dev/null real 0m0.002s user 0m0.000s sys 0m0.001s $ modprobe dummy numdummies=1000 $ time ls /sys/class/net >/dev/null real 0m0.041s user 0m0.003s sys 0m0.002s > 2. Interface deletion is normally much slower than interface creation > > strace -T -ttt on the "ip" command used to do this does not show the delay > where I thought it would be - cataloguing the existing interfaces. Instead, > it's the final send() to the netlink socket which does the relevant action > which appears to be slow, for both addition and detion. Adding the last > interface takes 200ms in that syscall, the first is quick (symptomatic of a > slowdown); for deletion the last send syscall is quick. > > Poking about in net/core/dev.c, I see that interface names are hashed using > a hash with a maximum of 256 entries. However, these seem to be hash > buckets supporting multiple entries so I can't imagine a chain of 4 entries > is problematic. Its not. > > I am having difficulty seeing what might be the issue in interface > creation. Any ideas? > Actually a lot, just make git log net/core/dev.c and you'll see many commits to make this faster. > In interface deletion, my attention is drawn to netdev_wait_allrefs, > which does this: > refcnt = netdev_refcnt_read(dev); > Here refcnt is 0, or there is a bug somewhere. (It happens, we fix bugs once in a while) > while (refcnt != 0) { > ... > msleep(250); > > refcnt = netdev_refcnt_read(dev); > .... > } > > I am guessing that this is going to do the msleep 50% of the time, > explaining 125ms of the observed time. How would people react to > exponential backoff instead (untested): > > int backoff = 10; > refcnt = netdev_refcnt_read(dev); > > while (refcnt != 0) { > ... > msleep(backoff); > if ((backoff *= 2) > 250) > backoff = 250; > > refcnt = netdev_refcnt_read(dev); > .... > } > > Welcome to the club. This is what is discussed on netdev since many years. Lot of work had been done to make it better. Interface deletion needs several rcu synch calls, they are very expensive. This is the price to pay to have lockless network stack in fast paths. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists