linux-kernel - Re: [PATCH] RDMA/device: Fix a race between mad_client and cm

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <e029db0a-c515-e61c-d34e-f7f054d51e88@sangfor.com.cn>
Date: Sat, 6 Jan 2024 10:12:17 +0800
From: Ding Hui <dinghui@...gfor.com.cn>
To: Jason Gunthorpe <jgg@...pe.ca>
Cc: leon@...nel.org, wenglianfa@...wei.com, gustavoars@...nel.org,
 linux-rdma@...r.kernel.org, linux-kernel@...r.kernel.org,
 Shifeng Li <lishifeng1992@....com>, Shifeng Li <lishifeng@...gfor.com.cn>
Subject: Re: [PATCH] RDMA/device: Fix a race between mad_client and cm_client
 init

On 2024/1/4 20:37, Jason Gunthorpe wrote:
> On Thu, Jan 04, 2024 at 02:48:14PM +0800, Shifeng Li wrote:
> 
>> The root cause is that mad_client and cm_client may init concurrently
>> when devices_rwsem write semaphore is downgraded in enable_device_and_get() like:
> 
> That can't be true, the module loader infrastructue ensures those two
> things are sequential.
> 

Please consider the sequence again and notice that:

1. We agree that dependencies ensure mad_client be registered before cm_client.
2. But the mad_client.add() is not invoked in ib_register_client(), since
    there is no DEVICE_REGISTERED device at that time.
    Instead, it will be delayed until the device driver init (e.g. mlx5_core)
    in enable_device_and_get().
3. The ib_cm and mlx5_core can be loaded concurrently, after setting DEVICE_REGISTERED
    and downgrade_write(&devices_rwsem) in enable_device_and_get(), there is a chance
    that cm_client.add() can be invoked before mad_client.add().


         T1(ib_core init)      |      T2(device driver init)        |        T3(ib_cm init)
---------------------------------------------------------------------------------------------------
ib_register_client mad_client
   assign_client_id
     add clients CLIENT_REGISTERED
     (with clients_rwsem write)
   down_read(&devices_rwsem);
   xa_for_each_marked (&devices, DEVICE_REGISTERED)
     nop # no devices
   up_read(&devices_rwsem);

                               ib_register_device
                                 enable_device_and_get
                                   down_write(&devices_rwsem);
                                   set DEVICE_REGISTERED
                                   downgrade_write(&devices_rwsem);
                                                                     ib_register_client cm_client
                                                                       assign_client_id
                                                                         add clients CLIENT_REGISTERED
                                                                         (with clients_rwsem write)
                                                                       down_read(&devices_rwsem);
                                                                       xa_for_each_marked (&devices, DEVICE_REGISTERED)
                                                                         add_client_context
                                                                           down_write(&device->client_data_rwsem);
                                                                           get CLIENT_DATA_REGISTERED
                                                                           downgrade_write(&device->client_data_rwsem);
                                                                           cm_client.add
                                                                             cm_add_one
                                                                               ib_register_mad_agent
                                                                                 ib_get_mad_port
                                                                                   __ib_get_mad_port return NULL!
                                                                           set CLIENT_DATA_REGISTERED
                                                                           up_read(&device->client_data_rwsem);
                                                                       up_read(&devices_rwsem);
                                 down_read(&clients_rwsem);
                                 xa_for_each_marked (&clients, CLIENT_REGISTERED)
                                   add_client_context [mad]
                                     mad_client.add
                                   add_client_context [cm]
                                     nop # already CLIENT_DATA_REGISTERED
                                 up_read(&clients_rwsem);
                                 up_read(&devices_rwsem);

> You are trying to say that the post-client fixup stuff will still see
> the DEVICE_REGISTERED before it reaches the clients_rwsem lock?
> 
> That probably just says the clients_rwsem should be obtained before
> changing the DEVICE_STATE too :\
> 


-- 
Thanks,
- Ding Hui