linux-ext4 - Re: JBD2: Spotted dirty metadata buffer....

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-Id: <A81188A3-AD1E-4472-8504-4FAA26C8C463@dilger.ca>
Date:   Tue, 22 Nov 2016 16:02:53 -0700
From:   Andreas Dilger <adilger@...ger.ca>
To:     Wolfgang Walter <linux@...m.de>
Cc:     Ext4 Developers List <linux-ext4@...r.kernel.org>,
        LKML <linux-kernel@...r.kernel.org>
Subject: Re: JBD2: Spotted dirty metadata buffer....

On Nov 22, 2016, at 6:56 AM, Wolfgang Walter <linux@...m.de> wrote:
> 
> Am Montag, 21. November 2016, 17:49:36 schrieben Sie:
>> On Nov 21, 2016, at 8:28 AM, Wolfgang Walter <linux@...m.de> wrote:
>>> Hello,
>>> 
>>> I'm testing EXT4 with an external journal (data=journal). When writing I
>>> rather often get>
>>> 	JBD2: Spotted dirty metadata buffer (dev = dm-22, blocknr = 1008028301).
>>> 	There's a risk of filesystem corruption in case of system crash.
>> Can you please determine what file this block belongs to and/or what kind
>> of metadata block it is.  You can check "dumpe2fs /dev/dm-22" to list all
>> of the metadata blocks, and if it isn't listed as filesystem metadata, you
>> can run "debugfs -c -R 'icheck 1008028301' /dev/dm-22" to find which file
>> this block belongs to, then "debugfs -c -R 'ncheck <inum>' /dev/dm-22" after
>> you have the inode number.
> 
> This think this is the metadata block which covers 1008028301? :
> 
> Group 30762: (Blocks 1008009216-1008041983) csum 0xb3c0 [INODE_UNINIT, ITABLE_ZEROED]
>  Block bitmap at 1007681546 (bg #30752 + 10), csum 0x94b3d052
>  Inode bitmap at 1007681562 (bg #30752 + 26), csum 0x00000000
>  Inode table at 1007684128-1007684383 (bg #30752 + 2592)
>  8340 free blocks, 4096 free inodes, 0 directories, 4096 unused inodes
>  Free blocks: 1008021210-1008021211, 1008021506-1008021511, 1008021520-1008021698, 1008021749-1008021887, 1008022528
> -1008022655, 1008024576-1008024768, 1008024839-1008024959, 1008026624-1008026879, 1008028524-1008035839
>  Free inodes: 126001153-126005248
> 
> I'm not sure if I read it correct but 1008028301 is part of the metadata?
> 
> I checked all of the other blocks logged so far and they are all alike,
> they are in between <x> and <y> of a "(Blocks <x>-<y>)" and they are above
> <b> of the "Inode table at <a>-<b>".

This means that block 1008028301 is part of this block group 30762, but
it is not listed in the range of "Block bitmap", "Inode bitmap", or "Inode
table" so it is just a regular block that was allocated to a file.  You
can use "debugfs -c -R 'icheck 1008028301' /dev/dm-22" to find out the
inode this block belongs to, and "debugfs -c -R 'stat <inum>' /dev/dm-22"
to see what the block is used for (e.g. data block, indirect block, etc).

Unfortunately, I don't think "debugfs icheck" reports the type of block
that this is, even though it probably could.

Cheers, Andreas

> Another thing I found was that no block has been mentioned more then twice.
> If they are mentioned twice this is within a second or so. Maybe this is
> simply because they only where touched when I did the rsync.
> 
> Here is an example:
> 
> [303709.144704] JBD2: Spotted dirty metadata buffer (dev = dm-22, blocknr = 1085342550). There's a risk of filesystem corruption in case of system crash.
> [303709.145561] JBD2: Spotted dirty metadata buffer (dev = dm-22, blocknr = 1085342538). There's a risk of filesystem corruption in case of system crash.
> 
>> 
>>> No other message is logged. If I unmount the filesystem an do an forced
>>> fsck, all seems fine.
>> This is likely a bug in the code, marking a block dirty without setting
>> up the journaling for it correctly.  A few bugs like this were fixed a
>> while ago by Eric, but those fixes should be in 4.8.8.
>> 
>>> The filesystem as it's journal are on a LV (which is again backed by
>>> DRBD). The journal, too.
>>> 
>>> I'm using stable kernel 4.8.8.
>>> 
>>> I created the journal with:
>>> 	mkfs.ext4 -O journal_dev -v -b 4096  -L jdyn /dev/export/jdyn
>>> 
>>> I created the filesystem with:
>>> 	mkfs.ext4 -J device=UUID=625d871f-c278-4acb-916d-774dc78dbd8a -v -b 4096
>>> 	-E
>>> 	stride=$((512/4)),stripe_width=$((512/4*3)),lazy_itable_init=0,nodiscard
>>> 	-O inline_data,mmp -L dyn /dev/export/dyn
>> It may be that inline_data is the culprit, since this is a relatively new
>> feature that isn't widely used yet.
>> 
>> Cheers, Andreas
>> 
>>> I tested it also with data=writeback. I didn't see these errors then.
>>> 
>>> Regards,
>>> --
>>> Wolfgang Walter
>>> Studentenwerk München
>>> Anstalt des öffentlichen Rechts
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
>>> the body of a message to majordomo@...r.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 
>> Cheers, Andreas
> 
> Regards,
> --
> Wolfgang Walter
> Studentenwerk München
> Anstalt des öffentlichen Rechts


Cheers, Andreas






Download attachment "signature.asc" of type "application/pgp-signature" (834 bytes)