netdev - Re: netem: the reorder discussion

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20130823084450.6baf2835@nehalam.linuxnetplumber.net>
Date:	Fri, 23 Aug 2013 08:44:50 -0700
From:	Stephen Hemminger <stephen@...workplumber.org>
To:	Eric Dumazet <eric.dumazet@...il.com>
Cc:	Teco Boot <teco@...-net.nl>, Ferry Huberts <mailings@...ie.com>,
	Netem <netem@...ts.linux-foundation.org>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	"core-users@...itd.nrl.navy.mil" <core-users@...itd.nrl.navy.mil>
Subject: Re: netem: the reorder discussion

On Fri, 23 Aug 2013 08:12:36 -0700
Eric Dumazet <eric.dumazet@...il.com> wrote:

> On Fri, 2013-08-23 at 16:16 +0200, Teco Boot wrote:
> > I had reorder problems with netem qdisc. I make use of the NRL CORE
> > network emulator, which sets up virtual routers and links, build with
> > network namespaces and netem. Typical CORE config for a n1<-->n2 link
> > with rate (1024Kbps), delay (50ms) and no jitter:
> >   qdisc tbf 1: dev n1.eth0.222 root refcnt 2 rate 1024Kbit burst 2999b
> > lat 488.6ms 
> >   qdisc netem 10: dev n1.eth0.222 parent 1:1 limit 1000 delay 50.0ms
> >   qdisc tbf 1: dev n2.eth0.222 root refcnt 2 rate 1024Kbit burst 2999b
> > lat 488.6ms 
> >   qdisc netem 10: dev n2.eth0.222 parent 1:1 limit 1000 delay 50.0ms
> > 
> > I added jitter to this bi-directional link, e.g. 20ms. Now the delay
> > for each packet is 50ms +/- 20ms is 30ms to 70ms.
> > However, this has some unexpected results: packets may be reordered.
> > That is because the actual delay is calculated for each packet. Some
> > packets have a larger delay, some have a smaller. If the inter-packet
> > spacing is smaller than the time differences set by netem (up to 2x
> > configured jitter), packets are reordered. In this example with
> > +/20ms, this is the case with packet rate larger than 25pps.
> > Reordering has bad effects on transport protocol throughputs.
> > Reordering is less common on the Internet, so I don't want to emulate
> > such. I don't say there is no reordering, I just say I don't want this
> > netem behavior.
> > 
> > The netem guide mentions this unexpected results. It was caused by a
> > change in version 1.1 (2.6.15). For people like me that do not want
> > this, there is a work-around mentioned. However, this doesn't work
> > anymore since somewhere before 2.6.31.
> > http://www.linuxfoundation.org/collaborate/workgroups/networking/netem
> > https://lists.linux-foundation.org/pipermail/netem/2011-April/001507.html
> > 
> > So what to do? The recent patch from Eric Dumazet eliminates
> > jitter_reordering when netem rate is configured. Maybe not the clean
> > approach, but acceptable. Ferry his patch eliminates
> > jitter_reordering. Some people may use reorder by jitter, so this
> > "feature" should not get removed. So Ferry his patch is not accepted.
> > On the other hand, people like me are very confused by current
> > behavior.
> > 
> > What we could do is keep existing features and describe what netem
> > currently does. That is:
> >  - netem with delay and jitter may reorder packets, if inter packet
> > spacing is smaller than jitter
> >  - reordering caused by delay and jitter can be turned off by using
> > netem rate. rate can be set to very high value is no shaping is
> > wanted.
> 
> As long as one can define the expected behavior, you can add whatever
> new netem parameter.
> 
> One could envision adding flow separation (skb->sk or rxhashing), so
> that each flow can have his own local queue, to guarantee no reorders
> per flow _if_ this is needed, even if per flow delays/jitter is/are
> configured.
> 
> We also use netem to test on large scale, where the reordering stuff
> needs fixes in transport stacks (And yes, we are working on TCP stack
> to permit higher levels of reorders)

I am happy with any solution that is:
  * allows both always ordering and reordering based on random jitter of delay
  * documented

I do get worried that people's tests get different results because of
netem behavior changes. Researchers like to have repeatable results.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html