netdev - Re: epoll_wait() performance

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <26e9b84bef24d46da9504aae2ca444d0d258c621.camel@redhat.com>
Date:   Wed, 27 Nov 2019 18:50:02 +0100
From:   Paolo Abeni <pabeni@...hat.com>
To:     David Laight <David.Laight@...LAB.COM>,
        Jesper Dangaard Brouer <brouer@...hat.com>
Cc:     'Marek Majkowski' <marek@...udflare.com>,
        linux-kernel <linux-kernel@...r.kernel.org>,
        network dev <netdev@...r.kernel.org>,
        kernel-team <kernel-team@...udflare.com>
Subject: Re: epoll_wait() performance

Hi,

Thanks for the additional details.

On Wed, 2019-11-27 at 17:30 +0000, David Laight wrote:
> From: Paolo Abeni
> > Sent: 27 November 2019 16:27
> ...
> > @David: If I read your message correctly, the pkt rate you are dealing
> > with is quite low... are we talking about tput or latency? I guess
> > latency could be measurably higher with recvmmsg() in respect to other
> > syscall. How do you measure the releative performances of recvmmsg()
> > and recv() ? with micro-benchmark/rdtsc()? Am I right that you are
> > usually getting a single packet per recvmmsg() call?
> 
> The packet rate per socket is low, typically one packet every 20ms.
> This is RTP, so telephony audio.
> However we have a lot of audio channels and hence a lot of sockets.
> So there are can be 1000s of sockets we need to receive the data from.
> The test system I'm using has 16 E1 TDM links each of which can handle
> 31 audio channels.
> Forwarding all these to/from RTP (one of the things it might do) is 496
> audio channels - so 496 RTP sockets and 496 RTCP ones.
> Although the test I'm doing is pure RTP and doesn't use TDM.

Oks, I think this is not exactly the preferred recvmmsg() use case ;)

> What I'm measuring is the total time taken to receive all the packets
> (on all the sockets) that are available to be read every 10ms.
> So poll + recv + add_to_queue.
> (The data processing is done by other threads.)
> I use the time difference (actually CLOCK_MONOTONIC - from rdtsc)
> to generate a 64 entry (self scaling) histogram of the elapsed times.
> Then look for the histograms peak value.
> (I need to work on the max value, but that is a different (more important!) problem.)
> Depending on the poll/recv method used this takes 1.5 to 2ms
> in each 10ms period.
> (It is faster if I run the cpu at full speed, but it usually idles along
> at 800MHz.)
> 
> If I use recvmmsg() I only expect to see one packet because there
> is (almost always) only one packet on each socket every 20ms.
> However there might be more than one, and if there is they
> all need to be read (well at least 2 of them) in that block of receives.

I would wild guess that recvmmsg() would be faster than 2 recv() when
there are exactly 2 pkts to read and the user-space provides exactly 2
msg entries, but likely non very relevant for the overall scenario.

Sorry, I don't have any good suggestion here.

Cheers,

Paolo