linux-kernel - Re: [V9fs-developer] 9pfs hangs since 4.7

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20170109203931.2315e1cd@duuni>
Date:   Mon, 9 Jan 2017 20:39:31 +0200
From:   Tuomas Tynkkynen <tuomas@...era.com>
To:     Al Viro <viro@...IV.linux.org.uk>
CC:     Greg Kurz <groug@...d.org>, <linux-fsdevel@...r.kernel.org>,
        <v9fs-developer@...ts.sourceforge.net>,
        <linux-kernel@...r.kernel.org>
Subject: Re: [V9fs-developer] 9pfs hangs since 4.7

On Sat, 7 Jan 2017 17:19:10 +0000
Al Viro <viro@...IV.linux.org.uk> wrote:

> On Sat, Jan 07, 2017 at 04:10:45PM +0100, Greg Kurz wrote:
> > > virtqueue_push(), but pdu freeing is delayed until v9fs_flush() gets woken
> > > up.  In the meanwhile, another request arrives into the slot of freed by
> > > that virtqueue_push() and we are out of pdus.
> > >   
> > 
> > Indeed. Even if this doesn't seem to be the problem here, I guess this should
> > be fixed.  
> 
> 	FWIW, there's something that looks like an off-by-one in
> v9fs_device_realize_common():
>     /* initialize pdu allocator */
>     QLIST_INIT(&s->free_list);
>     QLIST_INIT(&s->active_list);
>     for (i = 0; i < (MAX_REQ - 1); i++) {
>         QLIST_INSERT_HEAD(&s->free_list, &s->pdus[i], next);
>         s->pdus[i].s = s;
>         s->pdus[i].idx = i;
>     }
> 
> Had been there since the original merge of 9p support into qemu - that code
> had moved around a bit, but it had never inserted s->pdus[MAX_REQ - 1] into
> free list.  So your scenario with failing pdu_alloc() is still possible.
> In that log the total amount of pending requests has reached 128 for the
> first time right when the requests had stopped being handled and even
> though it had dropped below that shortly after, extra requests being put
> into queue had not been processed at all...
> 

Yes, this does seem to be related to this or otherwise MAX_REQ related!
- Bumping MAX_REQ up to 1024 makes the hang go away (on 4.7).
- Dropping it to 64 makes the same hang happen on kernels where it worked
  before (I tried 4.4.x).
- Doing s/(MAX_REQ - 1)/MAX_REQ/ makes the hang go away.

I tested QEMU 2.8.0 as well and the behaviour is the same there.

Here are the logs for 4.4 hanging with MAX_REQ == 64 without the loop
condition changed:

https://gist.githubusercontent.com/dezgeg/b5f0b7f8a0f3d8b6acb1566d7edcb2f0/raw/00241426890ea28d844986243c3b706881432fb4/9p-44.dmesg.log
https://gist.githubusercontent.com/dezgeg/2bffe6c0271c4c9c382ac6363ce1864b/raw/92329eaef38305f82090de5dde3c944561afa372/9p-44.qemu.log