netdev - Re: tx queue hashing hot-spots and poor performance (multiq, ixgbe)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <96ff3930904300351x60842bc1me0431c856f631773@mail.gmail.com>
Date:	Thu, 30 Apr 2009 12:51:16 +0200
From:	Jens Låås <jelaas@...il.com>
To:	David Miller <davem@...emloft.net>
Cc:	andrew@...dna.net, netdev@...r.kernel.org
Subject: Re: tx queue hashing hot-spots and poor performance (multiq, ixgbe)

2009/4/30, David Miller <davem@...emloft.net>:
> From: Jens Låås <jelaas@...il.com>
>  Date: Thu, 30 Apr 2009 11:07:35 +0200
>
>
>  > RX-side looks good. TX-side looks like what we also got with vanilla linux.
>  >
>  > What we do is patch all drivers with a custom select_queue function
>  > that selects the same outgoing queue as the incoming queue. With a one
>  > to one mapping of queues to CPUs you can also use the processor id.
>  >
>  > This way we get performance.
>
>
> I don't understand why this can even be necessary.
>
>  With the current code, the RX queue of a packet becomes
>  the hash for the TX queue.
>
>  If all the TX activity is happening on one TX queue then
>  there is a bug somewhere.

If I remember correctly we got use of several tx-queues and not one.
The hashed distribution missed a few of the tx-queues though. And it
also looked like some rx-queues got mapped on top of the same
tx-queue.

At the time we reasoned this behaviour was expected from the hashed randomizing.
We may certainly have misunderstood this and a one to one mapping
should be expected.

Hopefully the case where we have several devices and want
TX-completion to match rx-queue can also be solved. (The assumption
that tx-completion needs to run on the same CPU may also be proved
wrong. But we havent seen this in tests sofar.)

The main problem though was that the mapping is randomized. We wanted
to set smp_affinity correctly for tx to match rx. That was actually
the main reason for our local hacks.

>
>  Either the receiving device isn't invoking skb_record_rx_queue()
>  correctly, or there is some bug in how we compute the TX hash.
>
>  Everyone adds their own hacks, but that absolutely should not be
>  necessary, the kernel is essentially doing what you are adding
>  hacks for.
>
>  The only possible problems are bugs in the code, and we should find
>  those bugs instead of constantly talking about 'local select_queue
>  hacks we add to our cool driver for performance' :-/

We certainly dont consider the hacks cool in any way. They were only
for a specific purpose and a specific kernel-version.

Cheers,
Jens
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html