linux-kernel - problem with bio handling on raid5 and pblk

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-Id: <66350920-EC5E-447F-B5DF-0F3C2CDEAA65@javigon.com>
Date:   Thu, 22 Mar 2018 15:34:37 +0100
From:   Javier González <javier@...igon.com>
To:     Jens Axboe <axboe@...nel.dk>, shli@...nel.org
Cc:     linux-raid@...r.kernel.org, linux-block@...r.kernel.org,
        LKML <linux-kernel@...r.kernel.org>,
        Huaicheng Li <huaicheng@...uchicago.edu>
Subject: problem with bio handling on raid5 and pblk

Hi,

I have been looking into a bug report when using pblk and raid5 on top
and I am having problems understanding if the problem is in pblk's bio
handling or on raid5's bio assumptions on the completion path.

The problem occurs on the read path. In pblk, we take a reference to
every read bio as it enters, and release it after completing the bio.

   generic_make_request()
   pblk_submit_read()
     bio_get()
     ...
     bio_endio()
     bio_put()

The problem seems to be that on raid5's bi_end_io completion path,
raid5_end_read_request(), bio_reset() is called. When put together
with pblk's bio handling:

   generic_make_request()
   pblk_submit_read()
     bio_get()
     ...
     bio_endio()
     raid5_end_read_request()
       bio_reset()
     bio_put()

it results in the newly reset bio being put immediately, thus freed.
When the bio is reused then, we have an invalid pointer. In the report
we received things crash at BUG_ON(bio->bi_next) at
generic_make_request().

As far as I understand, it is part of the bio normal operation for
drivers under generic_make_request() to be able to take references and
release them after bio completion. Thus, in this case, the assumption
made by raid5, that it can issue a bio_reset() is incorrect. But I might
be missing an implicit cross layer rule that we are violating in pblk.
Any ideas?

This said, after analyzing the problem from pblk's perspective, I see
not reason to use bio_get()/bio_put() in the read path as it is at the
pblk level that we are submitting bio_endio(), thus we cannot risk the
bio being freed underneath us. Is this reasoning correct? I remember I
introduced these at the time there was a bug on the aio path, which was
not cleaning up correctly and could trigger an early bio free, but
revisiting it now, it seems unnecessary.

Thanks for the help!

Javier

Download attachment "signature.asc" of type "application/pgp-signature" (834 bytes)