linux-kernel - Re: [PATCH] IBcore/CM: Issue DREQ when receiving REQ/REP for stale QP

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <8df2c90e-8581-4ce8-fd18-92ac57eb6160@grimberg.me>
Date:   Sun, 30 Oct 2016 23:06:33 +0200
From:   Sagi Grimberg <sagi@...mberg.me>
To:     Hans Westgaard Ry <hans.westgaard.ry@...cle.com>,
        Doug Ledford <dledford@...hat.com>,
        Sean Hefty <sean.hefty@...el.com>,
        Hal Rosenstock <hal.rosenstock@...il.com>,
        Matan Barak <matanb@...lanox.com>,
        Erez Shitrit <erezsh@...lanox.com>,
        Bart Van Assche <bart.vanassche@...disk.com>,
        Ira Weiny <ira.weiny@...el.com>,
        Or Gerlitz <ogerlitz@...lanox.com>,
        Hakon Bugge <haakon.bugge@...cle.com>,
        Yuval Shaia <yuval.shaia@...cle.com>,
        linux-rdma@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH] IBcore/CM: Issue DREQ when receiving REQ/REP for stale QP

> from "InfiBand Architecture Specifications Volume 1":
>
>   A QP is said to have a stale connection when only one side has
>   connection information. A stale connection may result if the remote CM
>   had dropped the connection and sent a DREQ but the DREQ was never
>   received by the local CM. Alternatively the remote CM may have lost
>   all record of past connections because its node crashed and rebooted,
>   while the local CM did not become aware of the remote node's reboot
>   and therefore did not clean up stale connections.
>
> and:
>
>    A local CM may receive a REQ/REP for a stale connection. It shall
>    abort the connection issuing REJ to the REQ/REP. It shall then issue
>    DREQ with "DREQ:remote QPN” set to the remote QPN from the REQ/REP.
>
> This patch solves a problem with reuse of QPN. Current codebase, that
> is IPoIB, relies on a REAP-mechanism to do cleanup of the structures
> in CM. A problem with this is the timeconstants governing this
> mechanism; they are up to 768 seconds and the interface may look
> inresponsive in that period.  Issuing a DREQ (and receiving a DREP)
> does the necessary cleanup and the interface comes up.

I like this fix, so,

Reviewed-by: Sagi Grimberg <sagi@...mberg.me>

But I think the CM layer still is buggy in this area.

In vol 1 the state transition table specifically states that DREP
timeouts should move the cm_id to timewait state but the CM doesn't
seem to maintain response timeouts on disconnect requests. If the
DREQ happened to fail (send error completion) things are fine, but
if the DREQ makes it to the peer but it doesn't reply then no one
will take care of it (i.e. we will never see a TIMEWAIT event from
this cm_id)...

I recall some debugging session with Hal on this area a ~year ago
with a new iser target (which didn't reply to DREQs on reboot
sequences). iser initiator waits for a DISCONNECTED/TIMEWAIT events
before destroying the cm_id (which never happened because of the
above). I think I ended up working around that in iser to just go
ahead and destroy the cm_id after issuing a DREQ (but now I realize
it was never included so I'll probably dig it up again soon).