linux-kernel - Re: [f2fs-dev] Writes stoped working on f2fs after the compression support was added

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20200225122706.d6pngz62iwyowhym@core.my.home>
Date:   Tue, 25 Feb 2020 13:27:06 +0100
From:   Ondřej Jirman <megi@....cz>
To:     Chao Yu <yuchao0@...wei.com>, Jaegeuk Kim <jaegeuk@...nel.org>,
        linux-kernel@...r.kernel.org,
        linux-f2fs-devel@...ts.sourceforge.net
Subject: Re: [f2fs-dev] Writes stoped working on f2fs after the compression
 support was added

On Tue, Feb 25, 2020 at 01:08:14PM +0100, megi xff wrote:
> Hello,
> 
> On Tue, Feb 25, 2020 at 07:24:12PM +0800, Chao Yu wrote:
> > On 2020/2/24 22:31, Ondřej Jirman wrote:
> > >> Just by a looks of it:
> > >>
> > >> root@...2[/proc/sys/kernel] # dmesg | grep up_write | wc -l
> > >> 324
> > >> root@...2[/proc/sys/kernel] # dmesg | grep down_write | wc -l
> > >> 347
> > >>
> > >> there seems to be a mismatch of lock/unlock counts.
> > >  
> > > Sorry, a wrong grep expression.
> > > 
> > > root@...2[~] # dmesg | grep inode_down_write | wc -l
> > > 357
> > > root@...2[~] # dmesg | grep inode_up_write | wc -l
> > > 357
> > > root@...2[~] # dmesg | grep inode_up_read | wc -l
> > > 16
> > > root@...2[~] # dmesg | grep inode_down_read | wc -l
> > > 16
> > 
> > I don't know why we have consistent down/up pair, but through disassembled
> > code, I doubt it's the f2fs_inode->i_sem.
> 
> Because we were counting attempts, and not a successful lock. ;)
> 
> > c0435d7c:       ebf54af8        bl      c0188964 <printk>
> > c0435d80:       e1a00006        mov     r0, r6
> > c0435d84:       eb138135        bl      c0916260 <down_write>
> > 
> > inode_down_write()
> > 
> > c0435d88:       e284ce1d        add     ip, r4, #464    ; 0x1d0
> > 
> > We are stuck here.
> > 
> > [  430.675754] [<c0916260>] (down_write) from [<c0435d88>] (f2fs_write_single_data_page+0x600/0x7d8)
> >                                                 ^^^^^^^^^
> > [  430.675764] [<c0435788>] (f2fs_write_single_data_page) from [<c0436214>] (f2fs_write_cache_pages+0x2b4/0x7c4)
> > 
> > 
> > c0435d8c:       e14b0ad4        ldrd    r0, [fp, #-164] ; 0xffffff5c
> > c0435d90:       e1cc20d0        ldrd    r2, [ip]
> > c0435d94:       e1520000        cmp     r2, r0
> > c0435d98:       e0d33001        sbcs    r3, r3, r1
> > c0435d9c:       b1cc00f0        strdlt  r0, [ip]
> > c0435da0:       e1a00006        mov     r0, r6
> > c0435da4:       ebf52227        bl      c017e648 <up_write>
> > c0435da8:       e51b2098        ldr     r2, [fp, #-152] ; 0xffffff68
> > c0435dac:       e30c0730        movw    r0, #50992      ; 0xc730
> > c0435db0:       e59f11a4        ldr     r1, [pc, #420]  ; c0435f5c <f2fs_write_single_data_page+0x7d4>
> > c0435db4:       e34c00b6        movt    r0, #49334      ; 0xc0b6
> > c0435db8:       ebf54ae9        bl      c0188964 <printk>
> > 
> > inode_up_write()
> 
> The patch you sent helped so far. I'll keep the tablet running for a while,
> but so far the issue did not reappear within a few minutes after boot as
> usual.

So this time it just took several times longer to appear (8-20mins to the hang):

https://megous.com/dl/tmp/dmesg1
https://megous.com/dl/tmp/dmesg2

thank you and regards,
	o.

> thank you and regards,
> 	o.
> 
> > Thanks,
> > 
> > > 
> > > So it's probably not inode locking.
> > > 
> > >> root@...2[/proc/sys/kernel] # dmesg | grep down_read | wc -l
> > >> 16
> > >> root@...2[/proc/sys/kernel] # dmesg | grep up_read | wc -l
> > >> 16
> > >>
> > >> regards,
> > >> 	o.
> > >>
> > >>> thank you,
> > >>> 	o.
> > >>>
> > >>>>> Thanks,
> > >>>>>
> > >>>>>> [  246.758190]  r5:eff213b0 r4:da283c60
> > >>>>>> [  246.758198] [<c0435578>] (f2fs_write_single_data_page) from [<c0435fd8>] (f2fs_write_cache_pages+0x2b4/0x7c4)
> > >>>>>> [  246.758204]  r10:da645c28 r9:da283d60 r8:da283c60 r7:0000000f r6:da645d80 r5:00000001
> > >>>>>> [  246.758206]  r4:eff213b0
> > >>>>>> [  246.758214] [<c0435d24>] (f2fs_write_cache_pages) from [<c043682c>] (f2fs_write_data_pages+0x344/0x35c)
> > >>>>>> [  246.758220]  r10:00000000 r9:d9ed002c r8:d9ed0000 r7:00000004 r6:da283d60 r5:da283c60
> > >>>>>> [  246.758223]  r4:da645d80
> > >>>>>> [  246.758238] [<c04364e8>] (f2fs_write_data_pages) from [<c0267ee8>] (do_writepages+0x3c/0xd4)
> > >>>>>> [  246.758244]  r10:0000000a r9:c0e03d00 r8:00000c00 r7:c0264ddc r6:da645d80 r5:da283d60
> > >>>>>> [  246.758246]  r4:da283c60
> > >>>>>> [  246.758254] [<c0267eac>] (do_writepages) from [<c0310cbc>] (__writeback_single_inode+0x44/0x454)
> > >>>>>> [  246.758259]  r7:da283d60 r6:da645eac r5:da645d80 r4:da283c60
> > >>>>>> [  246.758266] [<c0310c78>] (__writeback_single_inode) from [<c03112d0>] (writeback_sb_inodes+0x204/0x4b0)
> > >>>>>> [  246.758272]  r10:0000000a r9:c0e03d00 r8:da283cc8 r7:da283c60 r6:da645eac r5:da283d08
> > >>>>>> [  246.758274]  r4:d9dc9848
> > >>>>>> [  246.758281] [<c03110cc>] (writeback_sb_inodes) from [<c03115cc>] (__writeback_inodes_wb+0x50/0xe4)
> > >>>>>> [  246.758287]  r10:da3797a8 r9:c0e03d00 r8:d9dc985c r7:da645eac r6:00000000 r5:d9dc9848
> > >>>>>> [  246.758289]  r4:da5a8800
> > >>>>>> [  246.758296] [<c031157c>] (__writeback_inodes_wb) from [<c03118f4>] (wb_writeback+0x294/0x338)
> > >>>>>> [  246.758302]  r10:fffbf200 r9:da644000 r8:c0e04e64 r7:d9dc9848 r6:d9dc9874 r5:da645eac
> > >>>>>> [  246.758305]  r4:d9dc9848
> > >>>>>> [  246.758312] [<c0311660>] (wb_writeback) from [<c0312dac>] (wb_workfn+0x35c/0x54c)
> > >>>>>> [  246.758318]  r10:da5f2005 r9:d9dc984c r8:d9dc9948 r7:d9dc9848 r6:00000000 r5:d9dc9954
> > >>>>>> [  246.758321]  r4:000031e6
> > >>>>>> [  246.758334] [<c0312a50>] (wb_workfn) from [<c014f2b8>] (process_one_work+0x214/0x544)
> > >>>>>> [  246.758340]  r10:da5f2005 r9:00000200 r8:00000000 r7:da5f2000 r6:ef044400 r5:da5eb000
> > >>>>>> [  246.758343]  r4:d9dc9954
> > >>>>>> [  246.758350] [<c014f0a4>] (process_one_work) from [<c014f634>] (worker_thread+0x4c/0x574)
> > >>>>>> [  246.758357]  r10:ef044400 r9:c0e03d00 r8:ef044418 r7:00000088 r6:ef044400 r5:da5eb014
> > >>>>>> [  246.758359]  r4:da5eb000
> > >>>>>> [  246.758368] [<c014f5e8>] (worker_thread) from [<c01564fc>] (kthread+0x144/0x170)
> > >>>>>> [  246.758374]  r10:ec9e5e90 r9:dabf325c r8:da5eb000 r7:da644000 r6:00000000 r5:da5fe000
> > >>>>>> [  246.758377]  r4:dabf3240
> > >>>>>> [  246.758386] [<c01563b8>] (kthread) from [<c01010e8>] (ret_from_fork+0x14/0x2c)
> > >>>>>> [  246.758391] Exception stack(0xda645fb0 to 0xda645ff8)
> > >>>>>> [  246.758397] 5fa0:                                     00000000 00000000 00000000 00000000
> > >>>>>> [  246.758402] 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> > >>>>>> [  246.758407] 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000
> > >>>>>> [  246.758413]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c01563b8
> > >>>>>> [  246.758416]  r4:da5fe000
> > >>>>>> .
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> Linux-f2fs-devel mailing list
> > >>>>> Linux-f2fs-devel@...ts.sourceforge.net
> > >>>>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
> > >>>>>
> > > .
> > >