linux-ext4 - Re: [PATCHv2 1/1] jbd2: Fix I/O hang in jbd2_journal_release_jbd

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Wed, 21 Jul 2010 21:06:50 +0200
From:	Jan Kara <jack@...e.cz>
To:	Brian King <brking@...ux.vnet.ibm.com>
Cc:	Josef Bacik <josef@...hat.com>, linux-ext4@...r.kernel.org,
	cmm@...ux.vnet.ibm.com, pmac@....ibm.com
Subject: Re: [PATCHv2 1/1] jbd2: Fix I/O hang in
	jbd2_journal_release_jbd_inode

> > 
> > I've been debugging a hang in jbd2_journal_release_jbd_inode
> > which is being seen on Power 6 systems quite a lot. When we get
> > in the hung state, all I/O to the disk in question gets blocked
> > where we stay indefinitely. Looking at the task list, I can see
> > we are stuck in jbd2_journal_release_jbd_inode waiting on a
> > wake up. I added some debug code to detect this scenario and
> > dump additional data if we were stuck in jbd2_journal_release_jbd_inode
> > for longer than 30 minutes. When it hit, I was able to see that
> > i_flags was 0, suggesting we missed the wake up.
> > 
> > This patch changes i_flags to be an unsigned long, uses bit operators
> > to access it, and adds barriers around the accesses. Prior to applying
> > this patch, we were regularly hitting this hang on numerous systems
> > in our test environment. After applying the patch, the hangs no longer
> > occur. Its still not clear to me why the j_list_lock doesn't protect us
> > in this path.
>   Thanks for debugging this! I was thinking hard about how it could happen that
> wake_up_bit doesn't wake up the waiter but I haven't found any explanation. All
> the waitqueue work seems to be properly wrapped inside the j_list_lock so
> even the waitqueue_active check in wake_up_bit should be fine.
>   I'd really like to understand what in my mind-model of spinlocks etc. is
> wrong. So could you maybe run a test with the attached debug patch and
> dump 'wait.seen' value in the hung task?
>   And one more question - if you remove 'waitqueue_active' check from
> kernel/wait.c:__wake_up_bit
>   is the problem still present? Thanks a lot in advance.
  Oops, the patch was missing 'seen' declaration. Here's an updated version.


-- 
Jan Kara <jack@...e.cz>
SuSE CR Labs

View attachment "0001-Debug-bit-wait-queues-in-JBD2.patch" of type "text/x-diff" (1783 bytes)