Message-ID: <DM6PR11MB465763E8D261CEC6B215358C9BAFA@DM6PR11MB4657.namprd11.prod.outlook.com>
Date: Thu, 9 Nov 2023 23:21:11 +0000
From: "Kubalewski, Arkadiusz" <arkadiusz.kubalewski@...el.com>
To: Jiri Pirko <jiri@...nulli.us>
CC: Vadim Fedorenko <vadim.fedorenko@...ux.dev>, "netdev@...r.kernel.org"
	<netdev@...r.kernel.org>, "Michalik, Michal" <michal.michalik@...el.com>,
	"Olech, Milena" <milena.olech@...el.com>, "pabeni@...hat.com"
	<pabeni@...hat.com>, "kuba@...nel.org" <kuba@...nel.org>
Subject: RE: [PATCH net 3/3] dpll: fix register pin with unregistered parent
 pin

>From: Jiri Pirko <jiri@...nulli.us>
>Sent: Thursday, November 9, 2023 7:04 PM
>
>Thu, Nov 09, 2023 at 05:02:48PM CET, arkadiusz.kubalewski@...el.com wrote:
>>>From: Vadim Fedorenko <vadim.fedorenko@...ux.dev>
>>>Sent: Thursday, November 9, 2023 11:56 AM
>>>To: Kubalewski, Arkadiusz <arkadiusz.kubalewski@...el.com>; Jiri Pirko
>>>
>>>On 09/11/2023 09:59, Kubalewski, Arkadiusz wrote:
>>>>> From: Jiri Pirko <jiri@...nulli.us>
>>>>> Sent: Wednesday, November 8, 2023 4:08 PM
>>>>>
>>>>> Wed, Nov 08, 2023 at 11:32:26AM CET, arkadiusz.kubalewski@...el.com
>>>>> wrote:
>>>>>> In case of multiple kernel module instances using the same dpll
>>>>>>device:
>>>>>> if only one registers dpll device, then only that one can register
>>>>>
>>>>> They why you don't register in multiple instances? See mlx5 for a
>>>>> reference.
>>>>>
>>>>
>>>> Every registration requires ops, but for our case only PF0 is able to
>>>> control dpll pins and device, thus only this can provide ops.
>>>> Basically without PF0, dpll is not able to be controlled, as well
>>>> as directly connected pins.
>>>>
>>>But why do you need other pins then, if FP0 doesn't exist?
>>>
>>
>>In general we don't need them at that point, but this is a corner case,
>>where users for some reason decided to unbind PF 0, and I treat this state
>>as temporary, where dpll/pins controllability is temporarily broken.
>
>So resolve this broken situation internally in the driver, registering
>things only in case PF0 is present. Some simple notification infra would
>do. Don't drag this into the subsystem internals.
>

Thanks for your feedback, but this is already wrong advice.

Our HW/FW is designed in different way than yours, it doesn't mean it is wrong.
As you might recall from our sync meetings, the dpll subsystem is to unify
approaches and reduce the code in the drivers, where your advice is exactly
opposite, suggested fix would require to implement extra synchronization of the
dpll and pin registration state between driver instances, most probably with
use of additional modules like aux-bus or something similar, which was from the
very beginning something we tried to avoid.
Only ice uses the infrastructure of muxed pins, and this is broken as it
doesn't allow unbind the driver which have registered dpll and pins without
crashing the kernel, so a fix is required in dpll subsystem, not in the driver.

Thank you!
Arkadiusz

>
>>
>>The dpll at that point is not registered, all the direct pins are also
>>not registered, thus not available to the users.
>>
>>When I do dump at that point there are still 3 pins present, one for each
>>PF, although they are all zombies - no parents as their parent pins are
>>not
>>registered (as the other patch [1/3] prevents dump of pin parent if the
>>parent is not registered). Maybe we can remove the REGISTERED mark for all
>>the muxed pins, if all their parents have been unregistered, so they won't
>>be visible to the user at all. Will try to POC that.
>>
>>>>>
>>>>>> directly connected pins with a dpll device. If unregistered parent
>>>>>> determines if the muxed pin can be register with it or not, it forces
>>>>>> serialized driver load order - first the driver instance which
>>>>>> registers the direct pins needs to be loaded, then the other
>>>>>> instances
>>>>>> could register muxed type pins.
>>>>>>
>>>>>> Allow registration of a pin with a parent even if the parent was not
>>>>>> yet registered, thus allow ability for unserialized driver instance
>>>>>
>>>>> Weird.
>>>>>
>>>>
>>>> Yeah, this is issue only for MUX/parent pin part, couldn't find better
>>>> way, but it doesn't seem to break things around..
>>>>
>>>
>>>I just wonder how do you see the registration procedure? How can parent
>>>pin exist if it's not registered? I believe you cannot get it through
>>>DPLL API, then the only possible way is to create it within the same
>>>driver code, which can be simply re-arranged. Am I wrong here?
>>>
>>
>>By "parent exist" I mean the parent pin exist in the dpll subsystem
>>(allocated on pins xa), but it doesn't mean it is available to the users,
>>as it might not be registered with a dpll device.
>>
>>We have this 2 step init approach:
>>1. dpll_pin_get(..) -> allocate new pin or increase reference if exist
>>2.1. dpll_pin_register(..) -> register with a dpll device
>>2.2. dpll_pin_on_pin_register -> register with a parent pin
>>
>>Basically:
>>- PF 0 does 1 & 2.1 for all the direct inputs, and steps: 1 & 2.2 for its
>>  recovery clock pin,
>>- other PF's only do step 1 for the direct input pins (as they must get
>>  reference to those in order to register recovery clock pin with them),
>>  and steps: 1 & 2.2 for their recovery clock pin.
>>
>>
>>Thank you!
>>Arkadiusz
>>
>>>> Thank you!
>>>> Arkadiusz
>>>>
>>>>>
>>>>>> load order.
>>>>>> Do not WARN_ON notification for unregistered pin, which can be
>>>>>> invoked
>>>>>> for described case, instead just return error.
>>>>>>
>>>>>> Fixes: 9431063ad323 ("dpll: core: Add DPLL framework base functions")
>>>>>> Fixes: 9d71b54b65b1 ("dpll: netlink: Add DPLL framework base
>>>>>> functions")
>>>>>> Signed-off-by: Arkadiusz Kubalewski <arkadiusz.kubalewski@...el.com>
>>>>>> ---
>>>>>> drivers/dpll/dpll_core.c    | 4 ----
>>>>>> drivers/dpll/dpll_netlink.c | 2 +-
>>>>>> 2 files changed, 1 insertion(+), 5 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/dpll/dpll_core.c b/drivers/dpll/dpll_core.c
>>>>>> index
>>>>>> 4077b562ba3b..ae884b92d68c 100644
>>>>>> --- a/drivers/dpll/dpll_core.c
>>>>>> +++ b/drivers/dpll/dpll_core.c
>>>>>> @@ -28,8 +28,6 @@ static u32 dpll_xa_id;
>>>>>> 	WARN_ON_ONCE(!xa_get_mark(&dpll_device_xa, (d)->id,
>>>>>> DPLL_REGISTERED))
>>>>>> #define ASSERT_DPLL_NOT_REGISTERED(d)	\
>>>>>> 	WARN_ON_ONCE(xa_get_mark(&dpll_device_xa, (d)->id,
>>>>>> DPLL_REGISTERED))
>>>>>> -#define ASSERT_PIN_REGISTERED(p)	\
>>>>>> -	WARN_ON_ONCE(!xa_get_mark(&dpll_pin_xa, (p)->id,
>>>>>> DPLL_REGISTERED))
>>>>>>
>>>>>> struct dpll_device_registration {
>>>>>> 	struct list_head list;
>>>>>> @@ -641,8 +639,6 @@ int dpll_pin_on_pin_register(struct dpll_pin
>>>>>> *parent,
>>>>>> struct dpll_pin *pin,
>>>>>> 	    WARN_ON(!ops->state_on_pin_get) ||
>>>>>> 	    WARN_ON(!ops->direction_get))
>>>>>> 		return -EINVAL;
>>>>>> -	if (ASSERT_PIN_REGISTERED(parent))
>>>>>> -		return -EINVAL;
>>>>>>
>>>>>> 	mutex_lock(&dpll_lock);
>>>>>> 	ret = dpll_xa_ref_pin_add(&pin->parent_refs, parent, ops,
>>>>>> priv); diff
>>>>>> --git a/drivers/dpll/dpll_netlink.c b/drivers/dpll/dpll_netlink.c
>>>>>> index
>>>>>> 963bbbbe6660..ff430f43304f 100644
>>>>>> --- a/drivers/dpll/dpll_netlink.c
>>>>>> +++ b/drivers/dpll/dpll_netlink.c
>>>>>> @@ -558,7 +558,7 @@ dpll_pin_event_send(enum dpll_cmd event, struct
>>>>>> dpll_pin *pin)
>>>>>> 	int ret = -ENOMEM;
>>>>>> 	void *hdr;
>>>>>>
>>>>>> -	if (WARN_ON(!xa_get_mark(&dpll_pin_xa, pin->id,
>>>>>> DPLL_REGISTERED)))
>>>>>> +	if (!xa_get_mark(&dpll_pin_xa, pin->id, DPLL_REGISTERED))
>>>>>> 		return -ENODEV;
>>>>>>
>>>>>> 	msg = genlmsg_new(NLMSG_GOODSIZE, GFP_KERNEL);
>>>>>> --
>>>>>> 2.38.1
>>>>>>
>>>
>>

Powered by blists - more mailing lists