Message-ID: <1324654013.10184.597.camel@denise.theartistscloset.com>
Date:	Fri, 23 Dec 2011 10:26:53 -0500
From:	"John A. Sullivan III" <jsullivan@...nsourcedevel.com>
To:	Eric Dumazet <eric.dumazet@...il.com>
Cc:	netdev@...r.kernel.org
Subject: Re: SFQ on HFSC leaf does not seem to work

Thanks for asking those questions as I had questions about them but felt
I was doing enough spamming of the list.  I am also just in the midst of
gather some of the other information you requested.  I'll respond in
line - John

On Fri, 2011-12-23 at 15:59 +0100, Eric Dumazet wrote:
> Le vendredi 23 décembre 2011 à 09:38 -0500, John A. Sullivan III a
> écrit :
> 
> > Thanks very much, Eric.  gso and gso only was enabled but disabling it
> > does not seem to have solved the problem when I activate netem:
> > 
> 
> And your kernel version is ?
root@...tswitch01:~# uname -a
Linux testswitch01 2.6.32-5-686 #1 SMP Mon Oct 3 04:15:24 UTC 2011 i686
GNU/Linux

This is Debian Squeeze i386

As described, I'm running two netcats - one going to the default queue
along with the pings and the other going to a different queue.
> 
> > root@...tswitch01:~# ./tcplay
> > root@...tswitch01:~# man ethtool
> > root@...tswitch01:~# ethtool -k eth1
> > Offload parameters for eth1:
> > rx-checksumming: on
> > tx-checksumming: on
> > scatter-gather: on
> > tcp-segmentation-offload: off
> > udp-fragmentation-offload: off
> > generic-segmentation-offload: on
> > generic-receive-offload: off
> > large-receive-offload: off
> > ntuple-filters: off
> > receive-hashing: off
> > root@...tswitch01:~# ethtool -K eth1 gso off
> > root@...tswitch01:~# ethtool -k eth1
> > Offload parameters for eth1:
> > rx-checksumming: on
> > tx-checksumming: on
> > scatter-gather: on
> > tcp-segmentation-offload: off
> > udp-fragmentation-offload: off
> > generic-segmentation-offload: off
> > generic-receive-offload: off
> > large-receive-offload: off
> > ntuple-filters: off
> > receive-hashing: off
> > ip ro flush cache
> > 
> > 64 bytes from 192.168.223.84: icmp_req=16 ttl=64 time=42.6 ms
> > 64 bytes from 192.168.223.84: icmp_req=17 ttl=64 time=39.1 ms
> > 64 bytes from 192.168.223.84: icmp_req=18 ttl=64 time=45.5 ms
> > 64 bytes from 192.168.223.84: icmp_req=19 ttl=64 time=406 ms
> > 64 bytes from 192.168.223.84: icmp_req=20 ttl=64 time=919 ms
> > 64 bytes from 192.168.223.84: icmp_req=21 ttl=64 time=920 ms
> > 64 bytes from 192.168.223.84: icmp_req=22 ttl=64 time=1013 ms
> > 64 bytes from 192.168.223.84: icmp_req=23 ttl=64 time=1158 ms
> > 64 bytes from 192.168.223.84: icmp_req=24 ttl=64 time=1521 ms
> > 64 bytes from 192.168.223.84: icmp_req=25 ttl=64 time=1915 ms
> > 64 bytes from 192.168.223.84: icmp_req=26 ttl=64 time=2371 ms
> > 64 bytes from 192.168.223.84: icmp_req=27 ttl=64 time=2797 ms
> > 64 bytes from 192.168.223.84: icmp_req=28 ttl=64 time=3161 ms
> > 64 bytes from 192.168.223.84: icmp_req=29 ttl=64 time=3162 ms
> > 64 bytes from 192.168.223.84: icmp_req=30 ttl=64 time=3163 ms
> > 
> > Just in case something is amiss in my methodology, I have four ssh
> > sessions open to the test firewall; ssh is in a separate prioritized
> > queue.  In one session I run:
> > 	ping 192.168.223.84
> > Then, in another, I do:
> > 	nc 192.168.223.100 443 >/dev/null - this should go into a non-default,
> 
> So you _receive_ trafic ?
Yes
> 
> Are you aware you dont have SFQ in your ingress setup, only egress  ?
Yes.  This is a problem I have with netem on ingress traffic.  I use the
filter on ffff: to redirect to ifb0 for the ingress traffic shaping.  I
cannot figure out a way to redirect a second time to ifb1 for the netem
qdisc.  I tried putting two action mirred statements in the filter but
that did not work.  Unlike eth1, I cannot attach a filter further down
the ifb0 hfsc hierarchy because one can't redirect one ifb into another
ifb.  Thus, the only way I could figure out how to do inbound netem was
to replace the terminal qdisc with netem rather than SFQ.  I'd love to
be able to do that differently.  I tried attaching netem to the SFQ but
that failed (I assume because SFQ is classless) and I tried the other
way around, attaching SFQ to netem since you mentioned netem could take
a class but that did not work either.
> 
> > prioritized queue.
> > Pings are OK at this point.
> > Then, in a third, I do:
> > 	nc 192.168.223.100 80 >/dev/null - this goes into the default queue,
> 
> same here ?
Yes.
> 
> > the same as ping, and is when the trouble starts.
> > 
> > I did alter the queue lengths in a recommendation from Dave Taht.  Here
> > is my current script with netem:
> > 
> > tc qdisc add dev eth1 root handle 1: hfsc default 20
> > tc class add dev eth1 parent 1: classid 1:1 hfsc sc rate 1490kbit ul
> > rate 1490kbit
> > tc class add dev eth1 parent 1:1 classid 1:20 hfsc rt rate 400kbit ls
> > rate 200kbit
> > tc qdisc add dev eth1 parent 1:20 handle 1201 sfq perturb 60 limit 30
> > tc class add dev eth1 parent 1:1 classid 1:10 hfsc rt umax 16kbit dmax
> > 50ms rate 200kbit ls rate 1000kbit
> > tc qdisc add dev eth1 parent 1:10 handle 1101 sfq perturb 60 limit 30
> > tc class add dev eth1 parent 1:1 classid 1:30 hfsc rt umax 1514b dmax
> > 20ms rate 20kbit
> > tc qdisc add dev eth1 parent 1:30 handle 1301 sfq perturb 60 limit 30
> > iptables -t mangle -A POSTROUTING -p 6 --syn --dport 443 -j CONNMARK
> > --set-mark 0x10
> > iptables -t mangle -A PREROUTING -p 6 --syn --dport 822 -j CONNMARK
> > --set-mark 0x11
> > iptables -t mangle -A POSTROUTING -o eth1 -p 6 -j CONNMARK
> > --restore-mark
> > modprobe ifb
> > ifconfig ifb0 up
> > ifconfig ifb1 up
> > tc qdisc add dev ifb0 root handle 1: hfsc default 20
> > tc class add dev ifb0 parent 1: classid 1:1 hfsc sc rate 1490kbit ul
> > rate 1490kbit
> > tc class add dev ifb0 parent 1:1 classid 1:20 hfsc rt rate 400kbit ls
> > rate 200kbit
> > tc qdisc add dev ifb0 parent 1:20 handle 1201 netem delay 25ms 5ms
> > distribution normal loss 0.1% 30%
> > tc class add dev ifb0 parent 1:1 classid 1:10 hfsc rt umax 16kbit dmax
> > 50ms rate 200kbit ls rate 1000kbit
> > tc qdisc add dev ifb0 parent 1:10 handle 1101 netem delay 25ms 5ms
> > distribution normal loss 0.1% 30%
> > tc class add dev ifb0 parent 1:1 classid 1:30 hfsc rt umax 1514b dmax
> > 20ms rate 20kbit
> > tc qdisc add dev ifb0 parent 1:30 handle 1301 netem delay 25ms 5ms
> > distribution normal loss 0.1% 30%
> > tc filter add dev ifb0 parent 1:0 protocol ip prio 1 handle 6: u32
> > divisor 1
> > tc filter add dev ifb0 parent 1:0 protocol ip prio 1 u32 match ip
> > protocol 6 0xff link 6: offset at 0 mask 0x0f00 shift 6 plus 0 eat
> 
> 
> > tc filter add dev ifb0 parent 1:0 protocol ip prio 1 u32 ht 6:0 match
> > tcp src 443 0x00ff flowid 1:10
> 
> why "src 443 0x00ff" ? It should be "src 443 0xffff"
That's what I tried at first but nothing matched the filter.  I assumed
it was because it objected to a value in the dst field so I masked it
off and it worked.
> 
> > tc filter add dev ifb0 parent 1:0 protocol ip prio 1 u32 ht 6:0 match
> > tcp dst 822 0xff00 flowid 1:30
> 
> same here : "dst 822 0xffff"
Same as above.  No filter matches when using that mask.
> 
> > tc qdisc add dev ifb1 root handle 2 netem delay 25ms 5ms distribution
> > normal loss 0.1% 30%
> > tc qdisc add dev eth1 ingress
> > tc filter add dev eth1 parent ffff: protocol ip prio 50 u32 match u32 0
> > 0 action mirred egress redirect dev ifb0
> > tc filter add dev eth1 parent 1:1 protocol ip prio 1 handle 0x11 fw
> > flowid 1:30
> > tc filter add dev eth1 parent 1:1 protocol ip prio 1 handle 0x10 fw
> > flowid 1:10
> > tc filter add dev eth1 parent 1:1 protocol ip prio 2 u32 match u32 0 0
> > flowid 1:20
> > tc filter add dev eth1 parent 1:0 protocol ip prio 1 u32 match u32 0 0
> > flowid 1:1 action mirred egress redirect dev ifb1
> > ip link set eth1 txqueuelen 100
> > ip link set ifb1 txqueuelen 100
> > ip link set ifb0 txqueuelen 100
> > 
> > I'd love to solve this.  Just when I thought I was all finished having
> > cracked the multiple filter problem to add netem to hfsc, I hit this.
> > Thanks again - John
> > 
> 
> Add some SFQ to your ingress too...
<grin> how with netem?  Thanks very much again - John
> 
> 
> 
PS - I also manually disabled gro in case that was a problem even though
it was showing off already.  It made no difference.

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists