Message-ID: <49271935-e0bf-948c-829b-18ddd9203926@6wind.com>
Date: Fri, 15 Sep 2023 17:54:47 +0200
From: Nicolas Dichtel <nicolas.dichtel@...nd.com>
To: David Ahern <dsahern@...nel.org>, Hangbin Liu <liuhangbin@...il.com>
Cc: Thomas Haller <thaller@...hat.com>, Benjamin Poirier
 <bpoirier@...dia.com>, Stephen Hemminger <stephen@...workplumber.org>,
 Ido Schimmel <idosch@...sch.org>, netdev@...r.kernel.org,
 "David S . Miller" <davem@...emloft.net>, Eric Dumazet
 <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
 Paolo Abeni <pabeni@...hat.com>
Subject: Re: [PATCH net-next] ipv4/fib: send RTM_DELROUTE notify when flush
 fib

Le 15/09/2023 à 05:07, David Ahern a écrit :
> On 9/14/23 9:43 AM, Nicolas Dichtel wrote:
>> Le 13/09/2023 à 16:53, Nicolas Dichtel a écrit :
>>> Le 13/09/2023 à 16:43, David Ahern a écrit :
>>>> On 9/13/23 8:11 AM, Nicolas Dichtel wrote:
>>>>> The compat_mode was introduced for daemons that doesn't support the nexthop
>>>>> framework. There must be a notification (RTM_DELROUTE) when a route is deleted
>>>>> due to a carrier down event. Right now, the backward compat is broken.
>>>>
>>>> The compat_mode is for daemons that do not understand the nexthop id
>>>> attribute, and need the legacy set of attributes for the route - i.e,
>>> Yes, it's my point.
>>> On my system, one daemon understands and configures nexthop id and another one
>>> doesn't understand nexthop id. This last daemon removes routes when an interface
>>> is put down but not when the carrier is lost.
>>> The kernel doc [1] says:
>>> 	Further, updates or deletes of a nexthop configuration generate route
>>> 	notifications for each fib entry using the nexthop.
>>> So, my understanding is that a RTM_DELROUTE msg should be sent when a nexthop is
>>> removed due to a carrier lost event.
>>>
>>> [1]
>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/ip-sysctl.rst#n2116
>>
>> I dug a bit more about these (missing) notifications. I will try to describe
>> what should be done for cases where there is no notification:
>>
>> When an interface is set down:
>>  - the single (!multipath) routes associated with this interface should be
>>    removed;
>>  - for multipath routes:
>>    + if all nh use this interface: the routes are deleted;
>>    + if only some nh uses this interface :
>>      ~ if all other nh already point to a down interface: the routes are deleted;
>>      ~ if at least one nh points to an up interface:
>>        o the nh are *temporarily* disabled if it's a plain nexthop;
>>        o the nh is *definitely* removed if it's a nexthop object;
>> When the interface is set up later, disabled nh are restored (ie only plain
>> nexthop of multipath routes).
>>
>> When an interface loses its carrier:
>>  - for routes using plain nexthop: nothing happens;
>>  - for routes using nexthop objects:
>>    + for single routes: they are deleted;
>>    + for multipath routes, the nh is definitely removed if it's a nexthop
>>      object (ie the route is deleted if there is no other nexthop in the group);
>> When an interface recovers its carrier, there is nothing to do.
>>
>> When the last ipv4 address of an interface is removed:
>>  - for routes using nexthop objects: nothing happens;
>>  - for routes using plain nexthop: the same rules as 'interface down' applies.
>> When an ipv4 address is added again on the interface, disabled nh are restored
>> (ie only plain nexthop of multipath routes).
>>
>> I bet I miss some cases.
>>
>> Conclusions:
>>  - legacy applications (that are not aware of nexthop objects) cannot maintain a
>>    routing cache (even with compat_mode enabled);
>>  - fixing only the legacy applications (aka compat_mode) seems too
>>    complex;
>>  - even if an application is aware of nexthop objects, the rules to maintain a
>>    cache are far from obvious.
>>
>> I don't understand why there is so much reluctance to not send a notification
>> when a route is deleted. This would fix all cases.
>> I understand that the goal was to save netlink traffic, but in this case, the
>> daemons that are interested in maintaining a routing cache have to fully parse
>> their cache to mark/remove routes. For big routing tables, this will cost a lot
>> of cpu, so I wonder if it's really a gain for the system. On such systems, there
>> is probably more than one daemon in this case, so even more cpu to spend for
>> these operations.
>>
>> As Thomas said, this discussion has come up for more than a decade. And with the
>> nexthop objects support, it's even more complex. There is obviously something to do.
>>
>> At least, I would have expected an RTM_DELNEXTHOP msg for each deleted nexthop.
>> But this wouldn't solve the routing cache sync for legacy applications.
>>
> 
> The split nexthop API is about efficiency. Do not send route
Efficiency for the kernel, but complexity for userspace daemons. I'm not sure
the system wins something when several daemons implement the same complex logic.

> notifications when they are easily derived from other events -- e.g.,
> link down or carrier down. The first commit for the nexthop code:
> 
> commit ab84be7e54fc3d9b248285f1a14067558d858819
> Author: David Ahern <dsahern@...il.com>
> Date:   Fri May 24 14:43:04 2019 -0700
> 
>     net: Initial nexthop code
> 
>     Barebones start point for nexthops. Implementation for RTM commands,
>     notifications, management of rbtree for holding nexthops by id, and
>     kernel side data structures for nexthops and nexthop config.
> 
>     Nexthops are maintained in an rbtree sorted by id. Similar to routes,
>     nexthops are configured per namespace using netns_nexthop struct added
>     to struct net.
> 
>     Nexthop notifications are sent when a nexthop is added or deleted,
>     but NOT if the delete is due to a device event or network namespace
>     teardown (which also involves device events). Applications are
>     expected to use the device down event to flush nexthops and any
>     routes used by the nexthops.
> 
> Intent is stated.
I understand, but are there numbers of the gain? In terms of cpu? memory? Some
systems could have a lot of routes (~900k for an ipv4 full route), but they
usually have a small number of nexthop. If you look at the big picture, is it
really worth saving some netlink messages and putting the load/complexity in the
userspace daemons?

It was the initial intent, no problem. But if people complain, maybe it could be
improved to fit everybody's use case.

> 
> The only compatibility with legacy API is around expanding the nhid to a
> full set of nexthop attributes to aid a transition to the new API.
As I said in another reply, the kernel doc [1] seems also to explain this:
	Further, updates or deletes of a nexthop configuration generate route
	notifications for each fib entry using the nexthop.

Without this, legacy daemons are broken. The compat_mode only hides some
behavior changes if these updates are not notified.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/ip-sysctl.rst#n2116


Regards,
Nicolas

Powered by blists - more mailing lists