linux-ext4 - Re: Bug#615998: linux-image-2.6.32-5-xen-amd64: Repeatable "kernel BUG at fs/jbd2/commit.c:534" from Postfix on ext4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <2F80BF45-28FA-46D3-9A28-CA9416DC5813@boeing.com>
Date:	Fri, 24 Jun 2011 11:03:52 -0500
From:	"Moffett, Kyle D" <Kyle.D.Moffett@...ing.com>
To:	Jan Kara <jack@...e.cz>
CC:	Sean Ryle <seanbo@...il.com>, "Ted Ts'o" <tytso@....edu>,
	"615998@...s.debian.org" <615998@...s.debian.org>,
	"linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>,
	Sachin Sant <sachinp@...ibm.com>,
	"Aneesh Kumar K.V" <aneesh.kumar@...ux.vnet.ibm.com>
Subject: Re: Bug#615998: linux-image-2.6.32-5-xen-amd64: Repeatable "kernel
 BUG at fs/jbd2/commit.c:534" from Postfix on ext4

On Jun 24, 2011, at 09:46, Jan Kara wrote:
> On Thu 23-06-11 16:19:08, Moffett, Kyle D wrote:
>> Besides which, line 534 in the Debian 2.6.32 kernel I am using is this
>> one:
>> 
>>  J_ASSERT(commit_transaction->t_nr_buffers <=
>>           commit_transaction->t_outstanding_credits);
> 
>  Hmm, OK, so we've used more metadata buffers than we told JBD2 to
> reserve. I suppose you are not using data=journal mode and the filesystem
> was created as ext4 (i.e. not converted from ext3), right? Are you using
> quotas?

The filesystem *is* using data=journal mode.  If I switch to data=ordered
or data=writeback, the problem goes away.

The filesystems were created as ext4 using the e2fstools in Debian squeeze:
1.41.12, and the kernel package is 2.6.32-5-xen-amd64 (2.6.32-34squeeze1).

The exact commands I used to create the Postfix filesystems were:
  lvcreate -L  5G -n postfix dbnew
  lvcreate -L 32M -n smtp    dbnew
  mke2fs -t ext4 -L db:postfix /dev/dbnew/postfix
  mke2fs -t ext4 -L db:smtp    /dev/dbnew/smtp
  tune2fs -i 0 -c 1 -e remount-ro -o acl,user_xattr,journal_data /dev/dbnew/postfix
  tune2fs -i 0 -c 1 -e remount-ro -o acl,user_xattr,journal_data /dev/dbnew/smtp

Then my fstab has:
  /dev/mapper/dbnew-postfix /var/spool/postfix ext4 noauto,noatime,nosuid,nodev 0 2
  /dev/mapper/dbnew-smtp    /var/lib/postfix   ext4 noauto,noatime,nosuid,nodev 0 2

I don't even think I have the quota tools installed on this system; there
are certainly none configured.

>> If somebody can tell me what information would help to debug this I'd be
>> more than happy to throw a whole bunch of debug printks under that error
>> condition and try to trigger the crash with that.
>> 
>> Alternatively I could remove that J_ASSERT() and instead add some debug
>> further down around the "commit_transaction->t_outstanding_credits--;"
>> to try to see exactly what IO it's handling when it runs out of credits.
> 
>  The trouble is that the problem is likely in some journal list shuffling
> code because if just some operation wrongly estimated the number of needed
> buffers, we'd fail the assertion in jbd2_journal_dirty_metadata():
> J_ASSERT_JH(jh, handle->h_buffer_credits > 0);

Hmm, ok...  I'm also going to turn that failing J_ASSERT() into a WARN_ON()
just to see how much further it gets.  I have an easy script to recreate this
data volume even if it gets totally hosed anyways, so...

> The patch below might catch the problem closer to the place where it
> happens...
> 
> Also possibly you can try current kernel whether the bug happens with it or
> not.

I'm definitely going to try this patch, but I'll also see what I can do about
trying a more recent kernel.

Thanks!

Cheers,
Kyle Moffett
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html