linux-ext4 - Re: [PATCH 1/1] jbd2: Fix I/O hang in jbd2_journal_release_jbd

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20100714174458.GA2378@localhost.localdomain>
Date:	Wed, 14 Jul 2010 13:44:58 -0400
From:	Josef Bacik <josef@...hat.com>
To:	Brian King <brking@...ux.vnet.ibm.com>
Cc:	linux-ext4@...r.kernel.org, cmm@...ux.vnet.ibm.com,
	pmac@....ibm.com
Subject: Re: [PATCH 1/1] jbd2: Fix I/O hang in
	jbd2_journal_release_jbd_inode

On Wed, Jul 14, 2010 at 09:56:15AM -0500, Brian King wrote:
> 
> I've been debugging a hang in jbd2_journal_release_jbd_inode
> which is being seen on Power 6 systems quite a lot. When we get
> in the hung state, all I/O to the disk in question gets blocked
> where we stay indefinitely. Looking at the task list, I can see
> we are stuck in jbd2_journal_release_jbd_inode waiting on a
> wake up. I added some debug code to detect this scenario and
> dump additional data if we were stuck in jbd2_journal_release_jbd_inode
> for longer than 30 minutes. When it hit, I was able to see that
> i_flags was 0, suggesting we missed the wake up.
> 
> This patch changes i_flags to be an unsigned long, uses bit operators
> to access it, and adds barriers around the accesses. Prior to applying
> this patch, we were regularly hitting this hang on numerous systems
> in our test environment. After applying the patch, the hangs no longer
> occur. Its still not clear to me why the j_list_lock doesn't protect us
> in this path. It also appears a hang very similar to this was seen
> in the past and then was no longer recreatable:
> 
> http://forum.soft32.com/linux/20090310-ext4-hangs-ftopict478916.html
> 
> Signed-off-by: Brian King <brking@...ux.vnet.ibm.com>
> ---
> 
>  fs/jbd2/commit.c     |   14 ++++++++++----
>  fs/jbd2/journal.c    |    5 ++++-
>  include/linux/jbd2.h |    2 +-
>  3 files changed, 15 insertions(+), 6 deletions(-)
> 
> diff -puN include/linux/jbd2.h~jbd2_ji_commit_barrier_patch include/linux/jbd2.h
> --- linux-2.6/include/linux/jbd2.h~jbd2_ji_commit_barrier_patch	2010-07-07 09:01:12.000000000 -0500
> +++ linux-2.6-bjking1/include/linux/jbd2.h	2010-07-07 09:01:12.000000000 -0500
> @@ -395,7 +395,7 @@ struct jbd2_inode {
>  	struct inode *i_vfs_inode;
>  
>  	/* Flags of inode [j_list_lock] */
> -	unsigned int i_flags;
> +	unsigned long i_flags;
>  };
>  
>  struct jbd2_revoke_table_s;
> diff -puN fs/jbd2/commit.c~jbd2_ji_commit_barrier_patch fs/jbd2/commit.c
> --- linux-2.6/fs/jbd2/commit.c~jbd2_ji_commit_barrier_patch	2010-07-07 09:01:12.000000000 -0500
> +++ linux-2.6-bjking1/fs/jbd2/commit.c	2010-07-07 09:01:12.000000000 -0500
> @@ -26,7 +26,9 @@
>  #include <linux/backing-dev.h>
>  #include <linux/bio.h>
>  #include <linux/blkdev.h>
> +#include <linux/bitops.h>
>  #include <trace/events/jbd2.h>
> +#include <asm/system.h>
>  
>  /*
>   * Default IO end handler for temporary BJ_IO buffer_heads.
> @@ -245,7 +247,7 @@ static int journal_submit_data_buffers(j
>  	spin_lock(&journal->j_list_lock);
>  	list_for_each_entry(jinode, &commit_transaction->t_inode_list, i_list) {
>  		mapping = jinode->i_vfs_inode->i_mapping;
> -		jinode->i_flags |= JI_COMMIT_RUNNING;
> +		set_bit(__JI_COMMIT_RUNNING, &jinode->i_flags);
>  		spin_unlock(&journal->j_list_lock);
>  		/*
>  		 * submit the inode data buffers. We use writepage
> @@ -260,7 +262,9 @@ static int journal_submit_data_buffers(j
>  		spin_lock(&journal->j_list_lock);
>  		J_ASSERT(jinode->i_transaction == commit_transaction);
>  		commit_transaction->t_flushed_data_blocks = 1;
> -		jinode->i_flags &= ~JI_COMMIT_RUNNING;
> +		smp_mb__before_clear_bit();
> +		clear_bit(__JI_COMMIT_RUNNING, &jinode->i_flags);
> +		smp_mb__before_clear_bit();
>  		wake_up_bit(&jinode->i_flags, __JI_COMMIT_RUNNING);
>  	}

This seems to be a bit overkill, you could probably just get away with

clear_bit()
smp_mb__after_clear_bit

That should be sufficient.  Other than that, it seems good.  Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html