Message-ID: <0e146bcf-d8b1-84a3-f8a4-976fc8414b25@kernel.org>
Date: Thu, 14 Sep 2023 21:07:25 -0600
From: David Ahern <dsahern@...nel.org>
To: nicolas.dichtel@...nd.com, Hangbin Liu <liuhangbin@...il.com>
Cc: Thomas Haller <thaller@...hat.com>, Benjamin Poirier
 <bpoirier@...dia.com>, Stephen Hemminger <stephen@...workplumber.org>,
 Ido Schimmel <idosch@...sch.org>, netdev@...r.kernel.org,
 "David S . Miller" <davem@...emloft.net>, Eric Dumazet
 <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
 Paolo Abeni <pabeni@...hat.com>
Subject: Re: [PATCH net-next] ipv4/fib: send RTM_DELROUTE notify when flush
 fib

On 9/14/23 9:43 AM, Nicolas Dichtel wrote:
> Le 13/09/2023 à 16:53, Nicolas Dichtel a écrit :
>> Le 13/09/2023 à 16:43, David Ahern a écrit :
>>> On 9/13/23 8:11 AM, Nicolas Dichtel wrote:
>>>> The compat_mode was introduced for daemons that doesn't support the nexthop
>>>> framework. There must be a notification (RTM_DELROUTE) when a route is deleted
>>>> due to a carrier down event. Right now, the backward compat is broken.
>>>
>>> The compat_mode is for daemons that do not understand the nexthop id
>>> attribute, and need the legacy set of attributes for the route - i.e,
>> Yes, it's my point.
>> On my system, one daemon understands and configures nexthop id and another one
>> doesn't understand nexthop id. This last daemon removes routes when an interface
>> is put down but not when the carrier is lost.
>> The kernel doc [1] says:
>> 	Further, updates or deletes of a nexthop configuration generate route
>> 	notifications for each fib entry using the nexthop.
>> So, my understanding is that a RTM_DELROUTE msg should be sent when a nexthop is
>> removed due to a carrier lost event.
>>
>> [1]
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/ip-sysctl.rst#n2116
> 
> I dug a bit more about these (missing) notifications. I will try to describe
> what should be done for cases where there is no notification:
> 
> When an interface is set down:
>  - the single (!multipath) routes associated with this interface should be
>    removed;
>  - for multipath routes:
>    + if all nh use this interface: the routes are deleted;
>    + if only some nh uses this interface :
>      ~ if all other nh already point to a down interface: the routes are deleted;
>      ~ if at least one nh points to an up interface:
>        o the nh are *temporarily* disabled if it's a plain nexthop;
>        o the nh is *definitely* removed if it's a nexthop object;
> When the interface is set up later, disabled nh are restored (ie only plain
> nexthop of multipath routes).
> 
> When an interface loses its carrier:
>  - for routes using plain nexthop: nothing happens;
>  - for routes using nexthop objects:
>    + for single routes: they are deleted;
>    + for multipath routes, the nh is definitely removed if it's a nexthop
>      object (ie the route is deleted if there is no other nexthop in the group);
> When an interface recovers its carrier, there is nothing to do.
> 
> When the last ipv4 address of an interface is removed:
>  - for routes using nexthop objects: nothing happens;
>  - for routes using plain nexthop: the same rules as 'interface down' applies.
> When an ipv4 address is added again on the interface, disabled nh are restored
> (ie only plain nexthop of multipath routes).
> 
> I bet I miss some cases.
> 
> Conclusions:
>  - legacy applications (that are not aware of nexthop objects) cannot maintain a
>    routing cache (even with compat_mode enabled);
>  - fixing only the legacy applications (aka compat_mode) seems too
>    complex;
>  - even if an application is aware of nexthop objects, the rules to maintain a
>    cache are far from obvious.
> 
> I don't understand why there is so much reluctance to not send a notification
> when a route is deleted. This would fix all cases.
> I understand that the goal was to save netlink traffic, but in this case, the
> daemons that are interested in maintaining a routing cache have to fully parse
> their cache to mark/remove routes. For big routing tables, this will cost a lot
> of cpu, so I wonder if it's really a gain for the system. On such systems, there
> is probably more than one daemon in this case, so even more cpu to spend for
> these operations.
> 
> As Thomas said, this discussion has come up for more than a decade. And with the
> nexthop objects support, it's even more complex. There is obviously something to do.
> 
> At least, I would have expected an RTM_DELNEXTHOP msg for each deleted nexthop.
> But this wouldn't solve the routing cache sync for legacy applications.
> 

The split nexthop API is about efficiency. Do not send route
notifications when they are easily derived from other events -- e.g.,
link down or carrier down. The first commit for the nexthop code:

commit ab84be7e54fc3d9b248285f1a14067558d858819
Author: David Ahern <dsahern@...il.com>
Date:   Fri May 24 14:43:04 2019 -0700

    net: Initial nexthop code

    Barebones start point for nexthops. Implementation for RTM commands,
    notifications, management of rbtree for holding nexthops by id, and
    kernel side data structures for nexthops and nexthop config.

    Nexthops are maintained in an rbtree sorted by id. Similar to routes,
    nexthops are configured per namespace using netns_nexthop struct added
    to struct net.

    Nexthop notifications are sent when a nexthop is added or deleted,
    but NOT if the delete is due to a device event or network namespace
    teardown (which also involves device events). Applications are
    expected to use the device down event to flush nexthops and any
    routes used by the nexthops.

Intent is stated.

The only compatibility with legacy API is around expanding the nhid to a
full set of nexthop attributes to aid a transition to the new API.

This new API also gave an opportunity to bring equivalency between IPv4
and IPv6.

Let's not weaken that model at all.

Powered by blists - more mailing lists