netdev - Re: VRF Issue Since kernel 5

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20200401203523.vafhsqb3uxfvvvxq@topsnens>
Date:   Wed, 1 Apr 2020 22:35:23 +0200
From:   Maximilian Bosch <maximilian@...sch.me>
To:     netdev@...r.kernel.org
Subject: Re: VRF Issue Since kernel 5

Hi!

> This should work:
>     make -C tools/testing/selftests/net nettest
>     PATH=$PWD/tools/testing/selftests/net:$PATH
>     tools/testing/selftests/net/fcnal-test.sh

Thanks, will try this out later.

> If you want that ssh connection to work over a VRF you either need to
> set the shell context:
>     ip vrf exec <NAME> su - $USER
> 

Yes, using `ip vrf exec` is basically my current workaround.

> or add 'ip vrf exec' before the ssh. If it is an incoming connection to
> a server the ssh server either needs to be bound to the VRF or you need
> 'net.ipv4.tcp_l3mdev_accept = 1'

Does this mean that the `*l3mdev_accept`-parameters only "fix" this
issue if the VRF is on the server I connect to?

In my case the VRF is on my local machine and I try to connect through
the VRF to the server.

> The tcp reset suggests you are doing an outbound connection but the
> lookup for what must be the SYN-ACK is not finding the local socket -
> and that is because of the missing 'ip vrf exec' above.

I only experience this behavior on a 5.x kernel, not on e.g. 4.19
though. I may be wrong, but isn't this a breaking change for userspace
applications in the end?

Thanks!

  Maximilian

On Wed, Apr 01, 2020 at 01:18:28PM -0600, David Ahern wrote:
> On 4/1/20 12:16 PM, Maximilian Bosch wrote:
> > Hi!
> > 
> > First of all, sorry for my delayed response!
> > 
> >> functional test script under tools/testing/selftests/net covers VRF
> >> tests and it ran clean for 5.4 last time I checked. There were a few
> >> changes that went into 4.20 or 5.0 that might be tripping up this use
> >> case, but I need a lot more information.
> > 
> > I recently started an attempt to get those tests running on my machine
> > (and a Fedora VM after that), however I had several issues with
> > timeouts (when running `sudo -E make -C tools/testing/selftests TARGETS="net"
> > run_tests`).
> > 
> > May I ask if there are further things I need to take care of to get
> > those tests successfully running?
> 
> This should work:
>     make -C tools/testing/selftests/net nettest
>     PATH=$PWD/tools/testing/selftests/net:$PATH
>     tools/testing/selftests/net/fcnal-test.sh
> 
> > 
> >> are you saying wireguard worked with VRF in the past but is not now?
> > 
> > No. WireGuard traffic is still working fine. The only issue is
> > TCP-traffic through a VRF (which worked with 4.19, but doesn't anymore
> > with 5.4 and 5.5).
> > 
> >> 'ip vrf exec' loads a bpf program and that requires locked memory, so
> >> yes, you need to increase it.
> > 
> > Thanks a lot for the explanation!
> > 
> >> Let's start with lookups:
> >>
> >> perf record -e fib:* -a -g
> >> <run test that fails, ctrl-c>
> >> perf script
> > 
> > For the record, please note that I'm now on Linux 5.5.13.
> > 
> > I ran the following command:
> > 
> > ```
> > sudo perf record -e fib:* -a -g -- ssh root@...60.36.231 -o ConnectTimeout=10s
> > ```
> 
> If you want that ssh connection to work over a VRF you either need to
> set the shell context:
>     ip vrf exec <NAME> su - $USER
> 
> or add 'ip vrf exec' before the ssh. If it is an incoming connection to
> a server the ssh server either needs to be bound to the VRF or you need
> 'net.ipv4.tcp_l3mdev_accept = 1'
> 
> > 
> > The full output can be found here:
> > 
> > https://gist.githubusercontent.com/Ma27/a6f83e05f6ffede21c2e27d5c7d27098/raw/4852d97ee4860f7887e16f94a8ede4b4406f07bc/perf-report.txt
> 
> seems like you have local rule ahead of the l3mdev rule. The order
> should be:
> 
> # ip ru ls
> 1000:	from all lookup [l3mdev-table]
> 32765:	from all lookup local
> 32766:	from all lookup main
> 
> That is not the problem, I just noticed some sub-optimal lookups.
> 
> The tcp reset suggests you are doing an outbound connection but the
> lookup for what must be the SYN-ACK is not finding the local socket -
> and that is because of the missing 'ip vrf exec' above.

Download attachment "signature.asc" of type "application/pgp-signature" (489 bytes)