netdev - Re: [PATCH next 1/3] ipvlan: Defer multicast / broadcast processing to a work-queue

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Mon, 27 Apr 2015 11:14:12 -0700
From:	Mahesh Bandewar <maheshb@...gle.com>
To:	Dan Williams <dcbw@...hat.com>
Cc:	netdev <netdev@...r.kernel.org>,
	Eric Dumazet <edumazet@...gle.com>,
	David Miller <davem@...emloft.net>
Subject: Re: [PATCH next 1/3] ipvlan: Defer multicast / broadcast processing
 to a work-queue

On Fri, Apr 24, 2015 at 3:59 PM, Dan Williams <dcbw@...hat.com> wrote:
> On Fri, 2015-04-24 at 15:40 -0700, Mahesh Bandewar wrote:
>> On Fri, Apr 24, 2015 at 1:15 PM, Dan Williams <dcbw@...hat.com> wrote:
>> > On Thu, 2015-04-23 at 14:29 -0700, Mahesh Bandewar wrote:
>> >> Processing multicast / broadcast in fast path is performance draining
>> >> and having more links means more clonning and bringing performance
>> >> down further.
>> >>
>> >> Broadcast; in particular, need to be given to all the virtual links.
>> >> Earlier tricks of enabling broadcast bit for IPv4 only interfaces are not
>> >> really working since it fails autoconf. Which means enabling braodcast
>> >> for all the links if protocol specific hacks do not have to be added into
>> >> the driver.
>> >>
>> >> This patch defers all (incoming as well as outgoing) multicast traffic to
>> >> a work-queue leaving only the unicast traffic in the fast-path. Now if we
>> >> need to apply any additional tricks to further reduce the impact of this
>> >> (multicast / broadcast) type of traffic, it can be implemented while
>> >> processing this work without affecting the fast-path.
>> >
>> > These patches appear to work for me for the L2 + DHCP use-case, however
>> > I experienced some quite odd behavior when pinging the ipvlan interface
>> > from another machine.  I did this:
>> >
>> > ip link add link eno1 type ipvlan mode l2
>> > ip netns add ipv
>> > ip link set dev ipvlan0 netns ipv
>> > ip netns exec ipv /sbin/dhclient -B -4 -1 -v
>> > -pf /run/dhclient-ipvlan0.pid -C adafdasdfasf ipvlan0
>> > ip netns exec ping 4.2.2.1 <success>
>> >
>> > However, when pinging from another machine, I got very inconsistent ping
>> > replies:
>> >
>> > 64 bytes from 192.168.1.38: icmp_seq=1 ttl=64 time=11.4 ms
>> > 64 bytes from 192.168.1.38: icmp_seq=16 ttl=64 time=64.9 ms
>> > 64 bytes from 192.168.1.38: icmp_seq=25 ttl=64 time=87.9 ms
>> > 64 bytes from 192.168.1.38: icmp_seq=30 ttl=64 time=242 ms
>> > 64 bytes from 192.168.1.38: icmp_seq=35 ttl=64 time=40.1 ms
>> > 64 bytes from 192.168.1.38: icmp_seq=36 ttl=64 time=60.9 ms
>> >
>> We know that there is that PAUSE frame leak but that should not cause
>> this behavior if those are present in your network. The sched_work()
>> which is wrong (as pointed by Eric) especially when the machine is
>> busy and that might trigger something like this. Almost every 10th -
>> 15th ping packet seems to be processed correctly.
>>
>> I did get consistent results as opposed what you have shown here, but
>> will dig some more to see if something obviously wrong here.
>>
>> > But I cannot reproduce that in a second run (though I haven't rebooted
>> > to test cleanly again).
>> >
>> > And oddly, the dhclient process takes a consistent 5% CPU and wireshark
>> > running on eno1 (not even the ipvlan interface) jumps to 100% CPU along
>> > with the dumpcap process taking another 25%, none of which are normal.
>> > This is a 4-core i4790 box, so something is wrong here; is something
>> > holding onto a spinlock for way too long?
>> >
>> > But at least it handles the packets ok, so I say progress!  Happy to
>> > help track down the CPU usage issue if you want to give me patches to
>> > test.
>> >
>> Which patch(es) you are referring to?
>
> None that yet exist; simply that if any of the issues I described
> triggered thoughts or patches on your end, I'm happy to test them.  I
> will try to characterize the issues I have seen more next week and
> report back.
>
OK found the issue! RX frames  are never drained from the queue and
are processed again and again causing the CPU-usage spike you have
observed. I'll integrate the fix into v2

--mahesh..

> Dan
>
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html