linux-kernel - Re: [PATCH] fs/ceph/mds_client: ignore responses for waiting requests

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <cf545923-e782-76a7-dd94-f8586530502b@redhat.com>
Date:   Thu, 9 Mar 2023 13:30:53 +0800
From:   Xiubo Li <xiubli@...hat.com>
To:     Max Kellermann <max.kellermann@...os.com>
Cc:     idryomov@...il.com, jlayton@...nel.org, ceph-devel@...r.kernel.org,
        linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org,
        stable@...r.kernel.org
Subject: Re: [PATCH] fs/ceph/mds_client: ignore responses for waiting requests


On 08/03/2023 23:17, Max Kellermann wrote:
> On Wed, Mar 8, 2023 at 4:42 AM Xiubo Li <xiubli@...hat.com> wrote:
>> How could this happen ?
>>
>> Since the req hasn't been submitted yet, how could it receive a reply
>> normally ?
> I have no idea. We have frequent problems with MDS closing the
> connection (once or twice a week), and sometimes, this leads to the
> WARNING problem which leaves the server hanging. This seems to be some
> timing problem, but that MDS connection problem is a different
> problem.
> My patch just attempts to address the WARNING; not knowing much about
> Ceph internals, my idea was that even if the server sends bad reply
> packets, the client shouldn't panic.
>
>> It should be a corrupted reply and it lead us to get a incorrect req,
>> which hasn't been submitted yet.
>>
>> BTW, do you have the dump of the corrupted msg by 'ceph_msg_dump(msg)' ?
> Unfortunately not - we have already scrubbed the server that had this
> problem and rebooted it with a fresh image including my patch. It
> seems I don't have a full copy of the kernel log anymore.
>
> Coincidentally, the patch has prevented another kernel hang just a few
> minutes ago:
>
>   Mar 08 15:48:53 sweb1 kernel: ceph: mds0 caps stale
>   Mar 08 15:49:13 sweb1 kernel: ceph: mds0 caps stale
>   Mar 08 15:49:35 sweb1 kernel: ceph: mds0 caps went stale, renewing
>   Mar 08 15:49:35 sweb1 kernel: ceph: mds0 caps stale
>   Mar 08 15:49:35 sweb1 kernel: libceph: mds0 (1)10.41.2.11:6801 socket
> error on write
>   Mar 08 15:49:35 sweb1 kernel: libceph: mds0 (1)10.41.2.11:6801 session reset
>   Mar 08 15:49:35 sweb1 kernel: ceph: mds0 closed our session
>   Mar 08 15:49:35 sweb1 kernel: ceph: mds0 reconnect start
>   Mar 08 15:49:36 sweb1 kernel: ceph: mds0 reconnect success
>   Mar 08 15:49:36 sweb1 kernel: ceph:  dropping dirty+flushing Fx state
> for 0000000064778286 2199046848012
>   Mar 08 15:49:40 sweb1 kernel: ceph: mdsc_handle_reply on waiting
> request tid 1106187
>   Mar 08 15:49:53 sweb1 kernel: ceph: mds0 caps renewed
>
> Since my patch is already in place, the kernel hasn't checked the
> unexpected packet and thus hasn't dumped it....
>
> If you need more information and have a patch with more logging, I
> could easily boot those servers with your patch and post that data
> next time it happens.

Hi Max,

I figured out one possible case:

For example when mds closes our session the kclient will call 
'cleanup_session_requests()', which will drop the unsafe requests and 
then set 'req->r_attempts' to 0, that means for the requests if they 
were sent out without receiving unsafe reply they will be retried and 
then they will be possibly added to the wait list in '__do_request()'. 
If these requests will get a reply later just after being retried we 
could see this warning.

I attached one testing patch based yours, just added more debug logs, 
which won't be introduce perf issue since all the logs should be printed 
in corner cases.

Could you help test it ?

Thanks,

- Xiubo

> Max
>
View attachment "test.patch" of type "text/x-patch" (3926 bytes)