netdev - Re: [PATCH 0/3]: net: dsa: mt7530: support MT7530 in the MT7621 SoC

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <871s6gbpbt.fsf@notabene.neil.brown.name>
Date:   Mon, 17 Dec 2018 18:11:02 +1100
From:   NeilBrown <neil@...wn.name>
To:     Florian Fainelli <f.fainelli@...il.com>,
        David Miller <davem@...emloft.net>
Cc:     bjorn@...k.no, gerg@...nel.org, sean.wang@...iatek.com,
        andrew@...n.ch, vivien.didelot@...oirfairelinux.com,
        netdev@...r.kernel.org, blogic@...nwrt.org, opensource@...rst.com
Subject: Re: [PATCH 0/3]: net: dsa: mt7530: support MT7530 in the MT7621 SoC

On Sun, Dec 16 2018, Florian Fainelli wrote:

> On December 16, 2018 3:19:22 PM PST, NeilBrown <neil@...wn.name> wrote:
>>On Sun, Dec 16 2018, David Miller wrote:
>>
>>> From: NeilBrown <neil@...wn.name>
>>> Date: Mon, 17 Dec 2018 09:08:54 +1100
>>>
>>>> In my 4.4 kernel, the build_skb() call in (the equivalent of)
>>>> mtk_poll_rx() takes about 1.2usec and the call to napi_gro_receive()
>>>> takes about 3usec.
>>>> 
>>>> In my 4.20 kernel, these calls take about 30 and 24 usec
>>respectively.
>>>> This easily explains the slowdown.
>>>
>>> That's a huge difference.
>>>
>>> Nothing jumps out as a possible cause except perhaps retpoline or
>>> something like that.
>>
>>I'll keep that in mind - thanks.
>>
>>My guess was CPU-cache invalidation.
>>I just checked and the other CPU core (there are two - each
>>hyper-threaded - "other" meaning not the one that handles ethernet
>>interrupts) gets several thousand "IPI resched" interrupts while
>>running a 10 second (226MByte) iperf3 receive test.
>>About 17KB transferred per IPI.
>>I cannot see where build_skb() would do cache invalidation though.
>
> It doesn't the driver is responsible for that. How is coherency maintained between cores?

I suspect so - yes.  Coherency only needs explicit management with DMA
is used.  This wasn't the problem.

Further investigation showed that the problem was that I had
CONFIG_SLUB_DEBUG set.  That was probably useful in some earlier
debugging exercise, but it clearly isn't useful when performance-testing
the network.
I removed that and I have much nicer numbers - not quite the consistent
900+ that I saw with 4.4, but a lot closer.

Thanks for the encouragement, and sorry of the noise.

NeilBrown


>
> The IPI could be due to receive packet steering, is the MAC multi queue aware on the RX path?
> -- 
> Florian

Download attachment "signature.asc" of type "application/pgp-signature" (833 bytes)