linux-ext4 - Re: File system corruption after reboot, when rootfs in mounted over nfs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Mon, 25 Sep 2017 12:18:40 +0530
From:   RAJESH DASARI <raajeshdasari@...il.com>
To:     linux-ext4@...r.kernel.org, "Theodore Ts'o" <tytso@....edu>
Subject: Re: File system corruption after reboot, when rootfs in mounted over nfs

Added some more information.

Thanks,
Rajesh.

On Mon, Sep 25, 2017 at 11:05 AM, RAJESH DASARI <raajeshdasari@...il.com> wrote:
>  Thanks Ted for your reply , Please find my response in line and
> please let me know if any other logs are needed.
>
> Thanks,
> Rajesh Dasari.
>
>
> On Mon, Sep 25, 2017 at 10:53 AM, RAJESH DASARI <raajeshdasari@...il.com> wrote:
>> Thanks,
>> Rajesh Dasari.
>>
>>
>> On Tue, Sep 19, 2017 at 6:55 PM, Theodore Ts'o <tytso@....edu> wrote:
>>> On Mon, Sep 18, 2017 at 04:14:35PM +0530, RAJESH DASARI wrote:
>>>>
>>>> Could some one please help me with the below issue.
>>>>
>>>> I have  booted a mips based hardware with linux (4.4.36 kernel
>>>> )image(over tftp) and rootfs over nfs by passing nfsroot command line
>>>> option to the kernel.
>>>>
>>>> rootfs is mounted under / in my hardware environment.
>>>>
>>>> 192.168.113.254:/rootfs / type nfs
>>>> (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.113.254,mountvers=3,mountproto=tcp,local_lock=all,addr=192.168.113.254)
>>>>
>>>> I have a hard disk and i am mounting  it on /mnt , this /mnt directory
>>>> is part of nfsroot.
>>>>
>>>> e2fsck -f -y /dev/sda1 -> disk is clean with no errors.
>>>>
>>>> mount -t ext2 /dev/sda1 /mnt  (mount was successful ,mounting ext2 using ext4)
>>>> touch /mnt/test.log  -> this command is failing.
>>>> umount /mnt
>>>> reboot
>>>>
>>>> when i was executing the above commands in a loop i see that /dev/sda1
>>>> file system is getting corrupted.
>>>
>>> The above commands include running e2fsck?  Then it sounds like there
>>> is some kind of device driver bug.
>       yeah. e2fsck command also included and all the above commands
> were executed in a loop.
>>
>>>
>>> What if you include an e2fsck -f -y /dev/sda1 after the umount?  Can
>>> you capture the output from that e2fsck run?

 I tried running e2fsck after unmount also , i still see the issue.
Here is the output of e2fsck after
     unmount .

e2fsck -f -y /dev/sda1
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information /dev/sda1: 12/262144 files
(0.0% non-contiguous), 18510/1048576 blocks

e2fsck return code was 0. after executing e2fsck, i rebooted the node
and i ran e2fsck again, output of this command same as the previous
e2fsck o/p and e2fsck didn't return any errors ,  But
when i  mount the Hard disk it is failed with the below error.

[ 87.685184] EXT4-fs (sda1): mounting ext2 file system using the ext4
subsystem [ 87.694393] EXT4-fs error (device sda1): ext4_iget:4337:
inode #2: comm mount: bad extended attribute block 4278190080 [
87.707920] EXT4-fs (sda1): get root inode failed [ 87.712639] EXT4-fs
(sda1): mount failed mount: mounting /dev/sda1 on /mnt failed:
Structure needs cleaning

the below steps were executed in a loop for 100 times. I am seeing the
issue after executing for around 40 iterations.
    1) e2fsck -y -n /dev/sda1
     2) mount -t ext2 /dev/sda1 /mnt
     3) touch /mnt/test.log
     4) umount /mnt
     5) e2fsck -y -n /dev/sda1
     6)reboot
i tried to mount the hard disk as ext2 as my hard disk partition was
formatted as ext2 device.

I also formatted the hard disk as ext4 device and tried the above
steps (this time i mounted as ext4 only) , i still see the issue but
this time with a different error ,
mount command failed with the below error.
EXT4-fs (sda1): no journal found

the below ext4 kernel config options were used to build the kernel.

CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT2=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y

As i mentioned in my previous mail, when i boot the node from
HDD(kernel and rootfs both are booted from HDD) and  executed  steps 1
to 6 in a loop for 100 times ,  i am not noticing any issue. I am
seeing this issue continuously when i boot the node from nfs (kernel
is loaded from tftp and rootfs is over nfs).

>
>
>
>>>
>>>> I am able to reproduce
>>>> this issue always when i boot rootfs over nfs , if i boot from hard
>>>> disk , i am not noticing the issue .
>>>
>>> Is it exactly the same kernel in both cases?
    Kernel version and steps executed hardware environment everything
is same in both the cases.
>>>
>>> More detailed logs would certainly be helpful.  There's not enough
>>> detail in your description to do anything other than guess, since
>>> we're not mind readers....
>>>
>>>                                                 - Ted