| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <aebc6c2d-4711-95ba-daf2-1cd17fc6f0e7@huawei.com>
Date: Mon, 1 Feb 2021 16:47:50 +0800
From: Chao Leng <lengchao@...wei.com>
To: Hannes Reinecke <hare@...e.de>, Sagi Grimberg <sagi@...mberg.me>,
"Daniel Wagner" <dwagner@...e.de>
CC: <linux-nvme@...ts.infradead.org>, <linux-kernel@...r.kernel.org>,
"Jens Axboe" <axboe@...com>, Keith Busch <kbusch@...nel.org>,
Christoph Hellwig <hch@....de>
Subject: Re: [PATCH v2] nvme-multipath: Early exit if no path is available
On 2021/2/1 15:29, Hannes Reinecke wrote:
> On 2/1/21 3:16 AM, Chao Leng wrote:
>>
>>
>> On 2021/1/29 17:20, Hannes Reinecke wrote:
>>> On 1/29/21 9:46 AM, Chao Leng wrote:
>>>>
>>>>
>>>> On 2021/1/29 16:33, Hannes Reinecke wrote:
>>>>> On 1/29/21 8:45 AM, Chao Leng wrote:
>>>>>>
>>>>>>
>>>>>> On 2021/1/29 15:06, Hannes Reinecke wrote:
>>>>>>> On 1/29/21 4:07 AM, Chao Leng wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2021/1/29 9:42, Sagi Grimberg wrote:
>>>>>>>>>
>>>>>>>>>>> You can't see exactly where it dies but I followed the assembly to
>>>>>>>>>>> nvme_round_robin_path(). Maybe it's not the initial nvme_next_ns(head,
>>>>>>>>>>> old) which returns NULL but nvme_next_ns() is returning NULL eventually
>>>>>>>>>>> (list_next_or_null_rcu()).
>>>>>>>>>> So there is other bug cause nvme_next_ns abormal.
>>>>>>>>>> I review the code about head->list and head->current_path, I find 2 bugs
>>>>>>>>>> may cause the bug:
>>>>>>>>>> First, I already send the patch. see:
>>>>>>>>>> https://lore.kernel.org/linux-nvme/20210128033351.22116-1-lengchao@huawei.com/
>>>>>>>>>> Second, in nvme_ns_remove, list_del_rcu is before
>>>>>>>>>> nvme_mpath_clear_current_path. This may cause "old" is deleted from the
>>>>>>>>>> "head", but still use "old". I'm not sure there's any other
>>>>>>>>>> consideration here, I will check it and try to fix it.
>>>>>>>>>
>>>>>>>>> The reason why we first remove from head->list and only then clear
>>>>>>>>> current_path is because the other way around there is no way
>>>>>>>>> to guarantee that that the ns won't be assigned as current_path
>>>>>>>>> again (because it is in head->list).
>>>>>>>> ok, I see.
>>>>>>>>>
>>>>>>>>> nvme_ns_remove fences continue of deletion of the ns by synchronizing
>>>>>>>>> the srcu such that for sure the current_path clearance is visible.
>>>>>>>> The list will be like this:
>>>>>>>> head->next = ns1;
>>>>>>>> ns1->next = head;
>>>>>>>> old->next = ns1;
>>>>>>>
>>>>>>> Where does 'old' pointing to?
>>>>>>>
>>>>>>>> This may cause infinite loop in nvme_round_robin_path.
>>>>>>>> for (ns = nvme_next_ns(head, old);
>>>>>>>> ns != old;
>>>>>>>> ns = nvme_next_ns(head, ns))
>>>>>>>> The ns will always be ns1, and then infinite loop.
>>>>>>>
>>>>>>> No. nvme_next_ns() will return NULL.
>>>>>> If there is just one path(the "old") and the "old" is deleted,
>>>>>> nvme_next_ns() will return NULL.
>>>>>> The list like this:
>>>>>> head->next = head;
>>>>>> old->next = head;
>>>>>> If there is two or more path and the "old" is deleted,
>>>>>> "for" will be infinite loop. because nvme_next_ns() will return
>>>>>> the path which in the list except the "old", check condition will
>>>>>> be true for ever.
>>>>>
>>>>> But that will be caught by the statement above:
>>>>>
>>>>> if (list_is_singular(&head->list))
>>>>>
>>>>> no?
>>>> Two path just a sample example.
>>>> If there is just two path, will enter it, may cause no path but there is
>>>> actually one path. It is falsely assumed that the "old" must be not deleted.
>>>> If there is more than two path, will cause infinite loop.
>>> So you mean we'll need something like this?
>>>
>>> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
>>> index 71696819c228..8ffccaf9c19a 100644
>>> --- a/drivers/nvme/host/multipath.c
>>> +++ b/drivers/nvme/host/multipath.c
>>> @@ -202,10 +202,12 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node)
>>> static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head,
>>> struct nvme_ns *ns)
>>> {
>>> - ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns,
>>> - siblings);
>>> - if (ns)
>>> - return ns;
>>> + if (ns) {
>>> + ns = list_next_or_null_rcu(&head->list, &ns->siblings,
>>> + struct nvme_ns, siblings);
>>> + if (ns)
>>> + return ns;
>>> + }
>> No, in the scenario, ns should not be NULL.
>
> Why not? 'ns == NULL' is precisely the corner-case this is trying to fix...
>
>> May be we can do like this:
>>
>> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
>> index 282b7a4ea9a9..b895011a2cbd 100644
>> --- a/drivers/nvme/host/multipath.c
>> +++ b/drivers/nvme/host/multipath.c
>> @@ -199,30 +199,24 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node)
>> return found;
>> }
>>
>> -static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head,
>> - struct nvme_ns *ns)
>> -{
>> - ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns,
>> - siblings);
>> - if (ns)
>> - return ns;
>> - return list_first_or_null_rcu(&head->list, struct nvme_ns, siblings);
>> -}
>> +#define nvme_next_ns_condition(head, current, condition) \
>> +({ \
>> + struct nvme_ns *__ptr = list_next_or_null_rcu(&(head)->list, \
>> + &(current)->siblings, struct nvme_ns, siblings); \
>> + __ptr ? __ptr : (condition) ? (condition) = false, \
>> + list_first_or_null_rcu(&(head)->list, struct nvme_ns, \
>> + siblings) : NULL; \
>> +})
>>
> Urgh. Please, no. That is well impossible to debug.
> Can you please open-code it to demonstrate where the difference to the current (and my fixed) versions is?
> I'm still not clear where the problem is once we applied both patches.
For example assume the list has three path, and all path is not NVME_ANA_OPTIMIZED:
head->next = ns1;
ns1->next = ns2;
ns2->next = head;
old->next = ns2;
My patch work flow:
nvme_next_ns_condition(head, old, true) return ns2;
nvme_next_ns_condition(head, ns2, true) change the condition to false
and return ns1;
nvme_next_ns_condition(head, ns1, false) return ns2;
nvme_next_ns_condition(head, ns2, false) return NULL;
And then the loop end.
Your patch work flow:
nvme_next_ns(head, old) return ns2;
nvme_next_ns(head, ns2) return ns1;
nvme_next_ns(head, ns1) return ns2;
nvme_next_ns(head, ns2) return ns1;
nvme_next_ns(head, ns1) return ns2;
And then the loop is infinite.
>
> Cheers,
>
> Hannes
Powered by blists - more mailing lists