| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <ac8da9cf-9dec-a207-c80e-e9ee650b40fc@oracle.com>
Date: Wed, 16 Jun 2021 13:12:51 +0530
From: Anand Khoje <anand.a.khoje@...cle.com>
To: Leon Romanovsky <leon@...nel.org>
Cc: linux-rdma@...r.kernel.org, linux-kernel@...r.kernel.org,
dledford@...hat.com, jgg@...pe.ca, haakon.bugge@...cle.com
Subject: Re: [PATCH v4 for-next 3/3] IB/core: Obtain subnet_prefix from cache
in IB devices
On 6/16/2021 12:57 PM, Leon Romanovsky wrote:
> On Wed, Jun 16, 2021 at 12:22:13PM +0530, Anand Khoje wrote:
>> ib_query_port() calls device->ops.query_port() to get the port
>> attributes. The method of querying is device driver specific.
>> The same function calls device->ops.query_gid() to get the GID and
>> extract the subnet_prefix (gid_prefix).
>>
>> The GID and subnet_prefix are stored in a cache. But they do not get
>> read from the cache if the device is an Infiniband device. The
>> following change takes advantage of the cached subnet_prefix.
>> Testing with RDBMS has shown a significant improvement in performance
>> with this change.
>>
>> The function ib_cache_is_initialised() is introduced because
>> ib_query_port() gets called early in the stage when the cache is not
>> built while reading port immutable property.
>>
>> In that case, the default GID still gets read from HCA for IB link-
>> layer devices.
>>
>> In the situation of an event causing cache update, the subnet_prefix
>> will get retrieved from newly updated GID cache in ib_cache_update(),
>> so that we do not end up reading a stale value from cache via
>> ib_query_port().
>>
>> Fixes: fad61ad ("IB/core: Add subnet prefix to port info")
>> Suggested-by: Leon Romanovsky <leonro@...dia.com>
>> Suggested-by: Aru Kolappan <aru.kolappan@...cle.com>
>> Signed-off-by: Anand Khoje <anand.a.khoje@...cle.com>
>> Signed-off-by: Haakon Bugge <haakon.bugge@...cle.com>
>> ---
>>
>> v1 -> v2:
>> - Split the v1 patch in 3 patches as per Leon's suggestion.
>>
>> v2 -> v3:
>> - Added changes as per Mark Zhang's suggestion of clearing
>> flags in git_table_cleanup_one().
>> v3 -> v4:
>> - Removed the enum ib_port_data_flags and 8 byte flags from
>> struct ib_port_data, and the set_bit()/clear_bit() API
>> used to update this flag as that was not necessary.
>> Done to keep the code simple.
>> - Added code to read subnet_prefix from updated GID cache in the
>> event of cache update. Prior to this change, ib_cache_update
>> was reading the value for subnet_prefix via ib_query_port(),
>> due to this patch, we ended up reading a stale cached value of
>> subnet_prefix.
>>
>> ---
>> drivers/infiniband/core/cache.c | 18 +++++++++++++++---
>> drivers/infiniband/core/device.c | 9 +++++++++
>> include/rdma/ib_cache.h | 5 +++++
>> include/rdma/ib_verbs.h | 1 +
>> 4 files changed, 30 insertions(+), 3 deletions(-)
>>
>> diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
>> index 2325171..cd99c46 100644
>> --- a/drivers/infiniband/core/cache.c
>> +++ b/drivers/infiniband/core/cache.c
>> @@ -917,9 +917,11 @@ static void gid_table_cleanup_one(struct ib_device *ib_dev)
>> {
>> u32 p;
>>
>> - rdma_for_each_port (ib_dev, p)
>> + rdma_for_each_port (ib_dev, p) {
>> + ib_dev->port_data[p].cache_is_initialized = 0;
>
> I think that this line is not needed, we are removing device anyway and
> and query_port is not allowed at this stage.
>
We have kept this for code completeness purposes. Just as we did with
set_bit() and clear_bit() APIs.
>> cleanup_gid_table_port(ib_dev, p,
>> ib_dev->port_data[p].cache.gid);
>> + }
>> }
>>
>> static int gid_table_setup_one(struct ib_device *ib_dev)
>> @@ -1466,6 +1468,7 @@ static int config_non_roce_gid_cache(struct ib_device *device,
>> struct ib_port_attr *tprops = NULL;
>> struct ib_pkey_cache *pkey_cache = NULL;
>> struct ib_pkey_cache *old_pkey_cache = NULL;
>> + union ib_gid gid;
>> int i;
>> int ret;
>>
>> @@ -1523,13 +1526,21 @@ static int config_non_roce_gid_cache(struct ib_device *device,
>> device->port_data[port].cache.lmc = tprops->lmc;
>> device->port_data[port].cache.port_state = tprops->state;
>>
>> - device->port_data[port].cache.subnet_prefix = tprops->subnet_prefix;
>> + ret = rdma_query_gid(device, port, 0, &gid);
>> + if (ret) {
>> + write_unlock_irq(&device->cache.lock);
>> + goto err;
>> + }
>> +
>> + device->port_data[port].cache.subnet_prefix =
>> + be64_to_cpu(gid.global.subnet_prefix);
>> +
>> write_unlock_irq(&device->cache_lock);
>>
>> if (enforce_security)
>> ib_security_cache_change(device,
>> port,
>> - tprops->subnet_prefix);
>> + be64_to_cpu(gid.global.subnet_prefix));
>>
>> kfree(old_pkey_cache);
>> kfree(tprops);
>> @@ -1629,6 +1640,7 @@ int ib_cache_setup_one(struct ib_device *device)
>> err = ib_cache_update(device, p, true, true, true);
>> if (err)
>> return err;
>> + device->port_data[p].cache_is_initialized = 1;
>> }
>>
>> return 0;
>> diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
>> index 7a617e4..57b9039 100644
>> --- a/drivers/infiniband/core/device.c
>> +++ b/drivers/infiniband/core/device.c
>> @@ -2057,6 +2057,15 @@ static int __ib_query_port(struct ib_device *device,
>> IB_LINK_LAYER_INFINIBAND)
>> return 0;
>>
>> + if (!ib_cache_is_initialised(device, port_num))
>> + goto query_gid_from_device;
>
> IMHO, we don't need this new function and can access ib_port_data
> directly. In device.c, we have plenty of places that does it.
>
> Not critical.
>
Added this function to have a way to check validity of cache, such that
it could be used in future for the same check in areas to which
ib_port_data is opaque.
>> +
>> + ib_get_cached_subnet_prefix(device, port_num,
>> + &port_attr->subnet_prefix);
>> +
>> + return 0;
>> +
>> +query_gid_from_device:
>> err = device->ops.query_gid(device, port_num, 0, &gid);
>> if (err)
>> return err;
>> diff --git a/include/rdma/ib_cache.h b/include/rdma/ib_cache.h
>> index 226ae37..46b43a7 100644
>> --- a/include/rdma/ib_cache.h
>> +++ b/include/rdma/ib_cache.h
>> @@ -114,4 +114,9 @@ ssize_t rdma_query_gid_table(struct ib_device *device,
>> struct ib_uverbs_gid_entry *entries,
>> size_t max_entries);
>>
>> +static inline bool ib_cache_is_initialised(struct ib_device *device,
>> + u32 port_num)
>> +{
>> + return device->port_data[port_num].cache_is_initialized;
>> +}
>> #endif /* _IB_CACHE_H */
>> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
>> index c96d601..405f7da 100644
>> --- a/include/rdma/ib_verbs.h
>> +++ b/include/rdma/ib_verbs.h
>> @@ -2177,6 +2177,7 @@ struct ib_port_data {
>>
>> spinlock_t netdev_lock;
>>
>> + u8 cache_is_initialized:1;
>> struct list_head pkey_list;
>>
>> struct ib_port_cache cache;
>> --
>> 1.8.3.1
>>
Powered by blists - more mailing lists