[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <6ebbb7ab-f053-d212-6d88-7eb6754f12a6@linux.alibaba.com>
Date: Fri, 17 Nov 2023 20:43:03 +0800
From: "D. Wythe" <alibuda@...ux.alibaba.com>
To: Wenjia Zhang <wenjia@...ux.ibm.com>, kgraul@...ux.ibm.com,
jaka@...ux.ibm.com, wintera@...ux.ibm.com, guwen@...ux.alibaba.com
Cc: kuba@...nel.org, davem@...emloft.net, netdev@...r.kernel.org,
linux-s390@...r.kernel.org, linux-rdma@...r.kernel.org,
tonylu@...ux.alibaba.com, pabeni@...hat.com, edumazet@...gle.com
Subject: Re: [PATCH net v2] net/smc: avoid data corruption caused by decline
On 11/17/23 8:35 PM, Wenjia Zhang wrote:
>
>
> On 17.11.23 05:59, D. Wythe wrote:
>> From: "D. Wythe" <alibuda@...ux.alibaba.com>
>>
>> We found a data corruption issue during testing of SMC-R on Redis
>> applications.
>>
>> The benchmark has a low probability of reporting a strange error as
>> shown below.
>>
>> "Error: Protocol error, got "\xe2" as reply type byte"
>>
>> Finally, we found that the retrieved error data was as follows:
>>
>> 0xE2 0xD4 0xC3 0xD9 0x04 0x00 0x2C 0x20 0xA6 0x56 0x00 0x16 0x3E 0x0C
>> 0xCB 0x04 0x02 0x01 0x00 0x00 0x20 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0xE2
>>
>> It is quite obvious that this is a SMC DECLINE message, which means that
>> the applications received SMC protocol message.
>> We found that this was caused by the following situations:
>>
>> client server
>> proposal
>> ------------->
>> accept
>> <-------------
>> confirm
>> ------------->
>> wait confirm
>>
>> failed llc confirm
>> x------
>> (after 2s)timeout
>> wait rsp
>>
>> wait decline
>>
>> (after 1s) timeout
>> (after 2s) timeout
>> decline
>> -------------->
>> decline
>> <--------------
>>
>> As a result, a decline message was sent in the implementation, and this
>> message was read from TCP by the already-fallback connection.
>>
>> This patch double the client timeout as 2x of the server value,
>> With this simple change, the Decline messages should never cross or
>> collide (during Confirm link timeout).
>>
>> This issue requires an immediate solution, since the protocol updates
>> involve a more long-term solution.
>>
>
> Hi D.Wythe,
>
> I think you understood me wrong. I mean we don't need sysctl. I like
> the first version more, where you just need to add some comments in
> the code.
>
😅, ok that. I understood it with wrong way. I will resend the first
version with comments.
> Thanks,
> Wenjia
Powered by blists - more mailing lists