netdev - Re: [RFC PATCH 00/24] Introducing AF

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CALDO+SYcnc+kN3pVQCdUrFqnonQToo4YBe3bfqfiZoNQpvL2Nw@mail.gmail.com>
Date:   Wed, 28 Mar 2018 08:05:20 -0700
From:   William Tu <u9012063@...il.com>
To:     Jesper Dangaard Brouer <brouer@...hat.com>
Cc:     Björn Töpel <bjorn.topel@...il.com>,
        "Karlsson, Magnus" <magnus.karlsson@...el.com>,
        Alexander Duyck <alexander.h.duyck@...el.com>,
        Alexander Duyck <alexander.duyck@...il.com>,
        John Fastabend <john.fastabend@...il.com>,
        Alexei Starovoitov <ast@...com>,
        Willem de Bruijn <willemdebruijn.kernel@...il.com>,
        Daniel Borkmann <daniel@...earbox.net>,
        Linux Kernel Network Developers <netdev@...r.kernel.org>,
        Björn Töpel <bjorn.topel@...el.com>,
        michael.lundkvist@...csson.com,
        "Brandeburg, Jesse" <jesse.brandeburg@...el.com>,
        Anjali Singhai Jain <anjali.singhai@...el.com>,
        jeffrey.b.shaw@...el.com, ferruh.yigit@...el.com,
        qi.z.zhang@...el.com, dendibakh@...il.com
Subject: Re: [RFC PATCH 00/24] Introducing AF_XDP support

Hi Jesper,
Thanks for the comments.

>> I assume this xdpsock code is small and should all fit into the icache.
>> However, doing another perf stat on xdpsock l2fwd shows
>>
>> 13,720,109,581      stalled-cycles-frontend   # 60.01% frontend cycles
>> idle     (23.82%)
>>
>> <not supported>      stalled-cycles-backend
>>       7,994,837      branch-misses           # 0.16% of all branches
>>        (23.80%)
>>     996,874,424      bus-cycles      # 99.679 M/sec      (23.80%)
>>  18,942,220,445      ref-cycles      # 1894.067 M/sec    (28.56%)
>>     100,983,226      LLC-loads       # 10.097 M/sec      (23.80%)
>>       4,897,089      LLC-load-misses # 4.85% of all LL-cache hits     (23.80%)
>>      66,659,889      LLC-stores      # 6.665 M/sec       (9.52%)
>>           8,373 LLC-store-misses     # 0.837 K/sec  (9.52%)
>>     158,178,410      LLC-prefetches       # 15.817 M/sec  (9.52%)
>>       3,011,180      LLC-prefetch-misses  # 0.301 M/sec   (9.52%)
>>   8,190,383,109      dTLB-loads       # 818.971 M/sec     (9.52%)
>>      20,432,204      dTLB-load-misses # 0.25% of all dTLB cache hits   (9.52%)
>>   3,729,504,674      dTLB-stores       # 372.920 M/sec     (9.52%)
>>         992,231  dTLB-store-misses         # 0.099 M/sec    (9.52%)
>> <not supported>      dTLB-prefetches
>> <not supported>      dTLB-prefetch-misses
>>          11,619 iTLB-loads            # 0.001 M/sec (9.52%)
>>       1,874,756      iTLB-load-misses # 16135.26% of all iTLB cache hits (14.28%)
>
> What was the sample period for this perf stat?
>
10 seconds.
root@...-smartnic:~/net-next/tools/perf# ./perf stat -C 6 sleep 10

>> I have super high iTLB-load-misses. This is probably the cause of high
>> frontend stalled.
>
> It looks very strange that your iTLB-loads are 11,619, while the
> iTLB-load-misses are much much higher 1,874,756.
>
Does it mean cpu try to load the code, then fail, then load again and
fail again...
So the number of iTLB loads is larger than misses.
Maybe it's related to high nmi rate, where the nmi handler clear my iTLB?
Let me try to remove the nmi interference first.

>> Do you know any way to improve iTLB hit rate?
>
> The xdpsock code should be small enough to fit in the iCache, but it
> might be layout in memory in an unfortunate way.  You could play with
> rearranging the C-code (look at the objdump alignments).
>
> If you want to know the details about code alignment issue, and how to
> troubleshoot them, you should read this VERY excellent blog post by
> Denis Bakhvalov:
> https://dendibakh.github.io/blog/2018/01/18/Code_alignment_issues

Thanks for the link.
William