linux-ext4 - Re: Reg: logical volume file system is getting corrupted after multiple reboots.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20180522142537.iynr7c26kvghioga@localhost.localdomain>
Date:   Tue, 22 May 2018 16:25:37 +0200
From:   Lukas Czerner <lczerner@...hat.com>
To:     RAJESH DASARI <raajeshdasari@...il.com>
Cc:     Andreas Dilger <adilger@...ger.ca>, Theodore Ts'o <tytso@....edu>,
        Eric Sandeen <sandeen@...hat.com>, linux-ext4@...r.kernel.org
Subject: Re: Reg: logical volume file system is getting corrupted after
 multiple reboots.

On Tue, May 22, 2018 at 03:57:41PM +0530, RAJESH DASARI wrote:
> Hi ,
> 
> Could someone please respond to my query. Issue here is i have
> upgraded e2fsprogs to 1.44.0 version from 1.43.9 and after that i am
> noticing the file system corruption mentioned in this mail chain. I
> have upgraded to 1.44.1 also but i still see the issue.
> 
> i have downgraded to 1.43.9 version , issue is disappeared.
> 
> Reason why i was upgrading because  there seems to be  some buffer
> overrun issues in the blkid library and in the fsck program of
> e2fsprogs. An attacker can use this to cause a denial of service and
> this issue is fixed from 1.44.0 onwards. For this i was trying to
> upgrade the e2fsprogs, if upgrade is not possible,i would like to back
> port the buffer over run fix by Ted to 1.43.9 version.
> 
> I checked the git commit log and noticed that the below commit by ted
> will fix the buffer over run issue.
> https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/commit/?id=d8e5da0a3b94f7445ab8cdd629bfc561986e7501
> 
> @Ted,
> 
> Could you please let me know the above commit is enough or do i have
> to take any other changes along with this commit to fix the buffer
> over run issues on 1.43.9 version ?
> 
> Thanks,
> Rajesh Dasari.

Hi,

I can't seem to find your original report so I have no idea at all what
the problem is.

However are you saying that when you run e2fsck v1.43.9 the file
systemis fine and when you run e2fsck v1.44.1 there is a problem ? If
so, pleasse show us the problem.

Also I recall that Andreas asked you to git-bisect the relevant code to
try to pin-point the problem, have you tried that ?

-Lukas

> 
> 
> On Fri, May 18, 2018 at 3:19 PM, RAJESH DASARI <raajeshdasari@...il.com> wrote:
> > Thanks,
> > Rajesh Dasari.
> >
> >
> > On Thu, May 3, 2018 at 1:40 AM, Andreas Dilger <adilger@...ger.ca> wrote:
> >> On May 2, 2018, at 10:26 AM, RAJESH DASARI <raajeshdasari@...il.com> wrote:
> >>> On Tue, May 1, 2018 at 6:15 PM, Eric Sandeen <esandeen@...hat.com> wrote:
> >>>> On 4/30/18 1:27 PM, RAJESH DASARI wrote:
> >>>>> Hi ,
> >>>>>
> >>>>> We are noticing an issue with logical volume file system is getting
> >>>>> corrupted after restarting the machine for multiple times.
> >>>>
> >>>> When you say restarting, are you talking about clean reboots, or
> >>>> power fails etc that may replay the log?
> >>>
> >>> It is clean reboot. no power failures.
> >>>
> >>>> (Also note that for a while at least on Fedora, systemd was preventing
> >>>> the root filesystem from unmounting cleanly on reboot.)
> >>>>
> >>>> So, were these log-replay-inducing machine restarts or "clean" reboots?
> >>>>
> >>>>> This issue we have started noticing after upgrading the kernel to 4.4.121.
> >>>>
> >>>> What was the previous kernel that did not seem to exhibit the problem?
> >>>
> >>> we have upgraded from 4.4.106 to 4.4.121 and e2fsprogs from 1.43.9 to
> >>> 1.44.0. After the upgrade this issue is noticed.
> >>>
> >>> Now I have downgraded the kernel to 4.4.106 and downgraded e2fsprogs
> >>> to 1.43.9 and issue is disappeared.
> >>
> >> If that is the case, please try the newer kernel and e2fsprogs independently to isolate which one introduced the problem.  Next, do a git-bisect on the relevant code to isolate it to a specific patch.
> >>
> > I tried it independently and noticed that it is the issue with the
> > e2fsprogs version 1.44.0 .  I downgraded to 1.43.9 and issue is
> > disappeared.  Is it any known issue in e2fsprogs? does the latest
> > version of
> > e2fsprogs contains any fixes for similar issues. Please provide your inputs.
> >
> >> Cheers, Andreas
> >>
> >>>> If this happens again, capturing the primary super in some way (i.e.
> >>>> e2image, or even simply using dd to copy it) might be interesting, to see
> >>>> exactly what the corruption is.
> >>>>
> >>> I tried capturing the primary super block using dd command to some
> >>> file, but still i get the same error when i do dumpe2fs on the file.
> >>>> -Eric
> >>>>
> >>>>> while running tune2fs -c 1 /dev/VG_NEW/state to set the
> >>>>> mmax_mounts_count we are noticing the error.
> >>>>>
> >>>>> tune2fs -c 1 /dev/VG_NEW/state
> >>>>> tune2fs 1.44.0 (7-Mar-2018)
> >>>>> tune2fs: The ext2 superblock is corrupt while trying to open /dev/VG_NEW/state
> >>>>> Couldn't find valid filesystem superblock.
> >>>>>
> >>>>> lvs command output is below (there are other logical volumes and
> >>>>> volume groups also along with state volume, I have not pasted them to
> >>>>> minimize this post).
> >>>>> Important thing to note here is always state volume only is getting
> >>>>> corrupted and no file system corruption seen on other logical volumes.
> >>>>
> >>
> >>
> >> Cheers, Andreas
> >>
> >>
> >>
> >>
> >>