netdev - Re: Kernel 4.19 network performance - forwarding/routing normal users traffic

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <13d8e510ec3287ac0680dfaa311b10d79353c5e7.camel@mellanox.com>
Date:   Fri, 9 Nov 2018 04:52:01 +0000
From:   Saeed Mahameed <saeedm@...lanox.com>
To:     "dsahern@...il.com" <dsahern@...il.com>,
        "pstaszewski@...are.pl" <pstaszewski@...are.pl>,
        "brouer@...hat.com" <brouer@...hat.com>
CC:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "yoel@...knet.dk" <yoel@...knet.dk>
Subject: Re: Kernel 4.19 network performance - forwarding/routing normal users
 traffic

On Thu, 2018-11-08 at 17:42 -0700, David Ahern wrote:
> On 11/8/18 5:40 PM, Paweł Staszewski wrote:
> > 
> > W dniu 08.11.2018 o 17:32, David Ahern pisze:
> > > On 11/8/18 9:27 AM, Paweł Staszewski wrote:
> > > > > > What hardware is this?
> > > > > > 
> > > > mellanox connectx 4
> > > > ethtool -i enp175s0f0
> > > > driver: mlx5_core
> > > > version: 5.0-0
> > > > firmware-version: 12.21.1000 (SM_2001000001033)
> > > > expansion-rom-version:
> > > > bus-info: 0000:af:00.0
> > > > supports-statistics: yes
> > > > supports-test: yes
> > > > supports-eeprom-access: no
> > > > supports-register-dump: no
> > > > supports-priv-flags: yes
> > > > 
> > > > ethtool -i enp175s0f1
> > > > driver: mlx5_core
> > > > version: 5.0-0
> > > > firmware-version: 12.21.1000 (SM_2001000001033)
> > > > expansion-rom-version:
> > > > bus-info: 0000:af:00.1
> > > > supports-statistics: yes
> > > > supports-test: yes
> > > > supports-eeprom-access: no
> > > > supports-register-dump: no
> > > > supports-priv-flags: yes
> > > > 
> > > > > > Start with:
> > > > > > 
> > > > > > echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
> > > > > > cat /sys/kernel/debug/tracing/trace_pipe
> > > > >   cat /sys/kernel/debug/tracing/trace_pipe
> > > > >           <idle>-0     [045] ..s. 68469.467752:
> > > > > xdp_devmap_xmit:
> > > > > ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0
> > > > > drops=1
> > > > > from_ifindex=4 to_ifindex=5 err=-6
> > > FIB lookup is good, the redirect is happening, but the mlx5
> > > driver does
> > > not like it.
> > > 
> > > I think the -6 is coming from the mlx5 driver and the packet is
> > > getting
> > > dropped. Perhaps this check in mlx5e_xdp_xmit:
> > > 
> > >         if (unlikely(sq_num >= priv->channels.num))
> > >                  return -ENXIO;
> > I removed that part and recompiled - but after running now xdp_fwd
> > i
> > have kernel pamic :)
> 

hh, no please don't do such thing :)

It must be because the tx netdev has less tx queues than the rx netdev.
or the rx netdev rings are bound to a high cpu indexes.

anyway, best practice is to open #cores RX/TX netdev on both sides

ethtool -L enp175s0f0  combined $(nproc) 
ethtool -L enp175s0f1  combined $(nproc)

> Jesper or one of the Mellanox folks needs to respond about the config
> needed to run XDP with this NIC. I don't have a 40G or 100G card to
> play
> with.