Date:   Thu, 7 Apr 2022 14:28:56 -0700
From:   Ben Greear <greearb@...delatech.com>
To:     David Ahern <dsahern@...il.com>,
        Stephen Suryaputra <ssuryaextr@...il.com>
Cc:     netdev@...r.kernel.org
Subject: Re: Matching unbound sockets for VRF

On 4/5/22 7:32 AM, David Ahern wrote:
> On 4/4/22 6:41 AM, Stephen Suryaputra wrote:
>> On Sun, Apr 03, 2022 at 10:24:36AM -0600, David Ahern wrote:
>>> On 3/27/22 6:57 AM, Stephen Suryaputra wrote:
>>>>
>>>> The reproducer script is attached.
>>>>
>>>
>>> h0 has the mgmt vrf, the l3mdev settings yet is running the client in
>>> *default* vrf. Add 'ip vrf exec mgmt' before the 'nc' and it works.
>>
>> Yes. With "ip vrf exec mgmt" nc would work. We know that. See more
>> below.
>>
>>> Are you saying that before Mike and Robert's changes you could get a
>>> client to run in default VRF and work over mgmt VRF? If so it required
>>> some ugly routing tricks (the last fib rule you installed) and is a bug
>>> relative to the VRF design.
>>
>> Yes, before Mike and Robert's changes the client ran fine because of the
>> last fib rule. We did that because some of our applications are:
>> 1) Pre-dates "ip vrf exec"
>> 2) LD_PRELOAD trick from the early days doesn't work
>>
>> On the case (2) above, one concrete example is NFS mounting our images:
>> applications and kernel modules. We had to run less than full-blown
>> utilities and also the mount command uses glibc RPC functions
>> (pmap_getmaps(), clntudp_create(), clnt_call(), etc, etc.). We analyzed
>> it back then that because these functions are in glibc and call socket()
>> from within glibc, the LD_PRELOAD doesn't work.
>>
>>  From the thread of Mike and Robert's changes, the conclusion is that the
>> previous behavior is a bug but we have been relying on it for a while,
>> since the early days of VRFs, and an upgrade that includes the changes
>> caused some applications to not work anymore.
>>
>> I'm asking if Mike and Robert's changes should be controlled by an
>> option, e.g. sysctl, and be the default. But can be reverted back to the
>> previous behavior.
>>
> 
> It has been 3-1/2 years since that patch. Rather than add more checks to
> try to manage unintended app behavior, why not work on making your apps
> consistent with the intent of the VRF design? If adding `ip vrf exec
> VRF` before commands works, that is a very simple solution and the
> reason for the command (handle code that is not VRF aware).
> 
> I'm guessing that option will not work for all cases (e.g., NFS which I
> think Ben has asked about as well, cc'ed), but working towards making
> the code align with VRF design is the longer term win.

NFS certainly wouldn't work.  It builds its sockets in the kernel in a convoluted
call path.  I tried to make NFS work with VRF at one time, found it very painful
and not worth the effort, especially since I figured patches would never make it
upstream.

We have out-of-tree patches to at least make NFS work with source based routing,
but those patches were not accepted upstream (2011 timeframe may be last I tried),
so stock kernels + NFS plus interesting routing
pretty much doesn't work at all as far as I can tell.

If binding NFS to source IPs is interesting in this day and time, I can try reposting
my patches, or you can grab them from one of my trees:

Somewhere around 800 patches down from HEAD, first few patches after the upstream
rebase point:

https://github.com/greearb/linux-ct-5.17

https://github.com/greearb/nfs-utils-ct

Thanks,
Ben

-- 
Ben Greear <greearb@...delatech.com>
Candela Technologies Inc  http://www.candelatech.com

Powered by blists - more mailing lists