Message-ID: <ZNTxfQ0StiqKbvWj@bfoster>
Date: Thu, 10 Aug 2023 10:17:33 -0400
From: Brian Foster <bfoster@...hat.com>
To: Pengfei Xu <pengfei.xu@...el.com>
Cc: syzbot <syzbot+5050ad0fb47527b1808a@...kaller.appspotmail.com>,
	adilger.kernel@...ger.ca, linux-ext4@...r.kernel.org,
	linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org,
	netdev@...r.kernel.org, syzkaller-bugs@...glegroups.com,
	tytso@....edu, heng.su@...el.com
Subject: Re: [syzbot] [ext4?] WARNING in ext4_file_write_iter

On Mon, Aug 07, 2023 at 10:34:50AM +0800, Pengfei Xu wrote:
> Hi Brian,
> 
> On 2023-08-04 at 09:43:43 -0400, Brian Foster wrote:
> > On Thu, Aug 03, 2023 at 09:37:30AM +0800, Pengfei Xu wrote:
> > > On 2023-07-05 at 23:33:43 -0700, syzbot wrote:
> > > > syzbot has found a reproducer for the following issue on:
> > > > 
> > > > HEAD commit:    6843306689af Merge tag 'net-6.5-rc1' of git://git.kernel.o..
> > > > git tree:       net
> > > > console output: https://syzkaller.appspot.com/x/log.txt?x=114522aca80000
> > > > kernel config:  https://syzkaller.appspot.com/x/.config?x=7ad417033279f15a
> > > > dashboard link: https://syzkaller.appspot.com/bug?extid=5050ad0fb47527b1808a
> > > > compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
> > > > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=102cb190a80000
> > > > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=17c49d90a80000
> > > > 
> > > > Downloadable assets:
> > > > disk image: https://storage.googleapis.com/syzbot-assets/f6adc10dbd71/disk-68433066.raw.xz
> > > > vmlinux: https://storage.googleapis.com/syzbot-assets/5c3fa1329201/vmlinux-68433066.xz
> > > > kernel image: https://storage.googleapis.com/syzbot-assets/84db3452bac5/bzImage-68433066.xz
> > > > 
> > > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > > Reported-by: syzbot+5050ad0fb47527b1808a@...kaller.appspotmail.com
> > > > 
> > > > ------------[ cut here ]------------
> > > > WARNING: CPU: 1 PID: 5382 at fs/ext4/file.c:611 ext4_dio_write_iter fs/ext4/file.c:611 [inline]
> > > > WARNING: CPU: 1 PID: 5382 at fs/ext4/file.c:611 ext4_file_write_iter+0x1470/0x1880 fs/ext4/file.c:720
> > > > Modules linked in:
> > > > CPU: 1 PID: 5382 Comm: syz-executor288 Not tainted 6.4.0-syzkaller-11989-g6843306689af #0
> > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/27/2023
> > > > RIP: 0010:ext4_dio_write_iter fs/ext4/file.c:611 [inline]
> > > > RIP: 0010:ext4_file_write_iter+0x1470/0x1880 fs/ext4/file.c:720
> > > > Code: 84 03 00 00 48 8b 04 24 31 ff 8b 40 20 89 c3 89 44 24 10 83 e3 08 89 de e8 5d 5a 5b ff 85 db 0f 85 d5 fc ff ff e8 30 5e 5b ff <0f> 0b e9 c9 fc ff ff e8 24 5e 5b ff 48 8b 4c 24 40 4c 89 fa 4c 89
> > > > RSP: 0018:ffffc9000522fc30 EFLAGS: 00010293
> > > > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> > > > RDX: ffff8880277a3b80 RSI: ffffffff82298140 RDI: 0000000000000005
> > > > RBP: 0000000000000001 R08: 0000000000000005 R09: 0000000000000000
> > > > R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff8a832a60
> > > > R13: 0000000000000000 R14: 0000000000000000 R15: fffffffffffffff5
> > > > FS:  00007f154db95700(0000) GS:ffff8880b9900000(0000) knlGS:0000000000000000
> > > > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > CR2: 00007f154db74718 CR3: 000000006bcc7000 CR4: 00000000003506e0
> > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > > > Call Trace:
> > > >  <TASK>
> > > >  call_write_iter include/linux/fs.h:1871 [inline]
> > > >  new_sync_write fs/read_write.c:491 [inline]
> > > >  vfs_write+0x981/0xda0 fs/read_write.c:584
> > > >  ksys_write+0x122/0x250 fs/read_write.c:637
> > > >  do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> > > >  do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
> > > >  entry_SYSCALL_64_after_hwframe+0x63/0xcd
> > > > RIP: 0033:0x7f154dc094f9
> > > > Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 15 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
> > > > RSP: 002b:00007f154db952f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> > > > RAX: ffffffffffffffda RBX: 00007f154dc924f0 RCX: 00007f154dc094f9
> > > > RDX: 0000000000248800 RSI: 0000000020000000 RDI: 0000000000000006
> > > > RBP: 00007f154dc5f628 R08: 0000000000000000 R09: 0000000000000000
> > > > R10: 0000000000000000 R11: 0000000000000246 R12: 652e79726f6d656d
> > > > R13: 656c6c616b7a7973 R14: 6465646165726874 R15: 00007f154dc924f8
> > > >  </TASK>
> > > > 
> > > 
> > > Above issue in dmesg is:
> > > "WARNING: CPU: 1 PID: 5382 at fs/ext4/file.c:611 ext4_dio_write_iter fs/ext4/file.c:611 [inline]"
> > > 
> > > I found the similar behavior issue:
> > > "WARNING: CPU: 0 PID: 182134 at fs/ext4/file.c:611 ext4_dio_write_iter fs/ext4/file.c:611 [inline]"
> > > repro.report shows similar details.
> > > 
> > > Updated the bisect info for the above similar issue:
> > > Bisected and the problem commit was:
> > > "
> > > 310ee0902b8d9d0a13a5a13e94688a5863fa29c2: ext4: allow concurrent unaligned dio overwrites
> > > "
> > > After reverted the commit on top of v6.5-rc3, this issue was gone.
> > > 
> > 
> > Hi Pengfei,
> > 
> > Thanks for narrowing this down (and sorry for missing the earlier
> > report). Unfortunately I've not been able to reproduce this locally
> > using the generated reproducer. I tried both running the test program on
> > a local test vm as well as booting the generated disk image directly.
> > 
> > That said, I have received another report of this warning that happens
> > to be related to io_uring. The cause in that particular case is that
> > io_uring sets IOCB_HIPRI, which iomap dio turns into
> > REQ_POLLED|REQ_NOWAIT on the bio without necessarily having IOCB_NOWAIT
> > set on the request. This means we can expect -EAGAIN returns from the
> > storage layer without necessarily passing DIO_OVERWRITE_ONLY to iomap,
> > which in turn basically means that the warning added by this commit is
> > wrong.
> > 
> > I did submit the test patch at the link [1] referenced below to syzbot
> > to see if the OVERWRITE_ONLY flag is set and the results I got this
> > morning only showed the original !IOCB_NOWAIT warning. So while I still
> > do not know the source of the -EAGAIN in the syzbot test (and I would
> > like to), this shows that the overwrite flag is not involved and thus
> > the -EAGAIN is presumably unrelated to that logic.
> > 
> > So in summary I think the right fix is to just remove the overwrite flag
> > and warning from this ext4 codepath. It was always intended as an extra
> > precaution to support the warning, and the latter is clearly wrong. I'll
> > submit another test change in a separate mail just to see if syzbot
> > finds anything else and plan to send a proper patch to the list. In the
> > meantime, if you have any suggestions to help reproduce via the
> > generated program, I'm still interested in trying to grok where that
> > particular -EAGAIN comes from.. Thanks.
> 
> Thanks for your patch! I'm glad it's helpful.
> 
> It could not be reproduced immediately and it takes more than 200s to
> reproduce this issue, anyway here is my reproduced steps.
> As following link info for example, it's not my reproduced environment and it's
> alsmost the same for issue reproducing:
> https://lore.kernel.org/all/0000000000007faf0005fe4f14b9@google.com/T/#mf1678f17b7b7c4b3cc73abef436aac68787397ae
>   -> disk image: https://storage.googleapis.com/syzbot-assets/f6adc10dbd71/disk-68433066.raw.xz
>   -> kernel image: https://storage.googleapis.com/syzbot-assets/84db3452bac5/bzImage-68433066.xz
> 
> I used below simple script to boot up vm:
> "
> #!/bin/bash
> 
> bzimage=$1
> 
> [[ -z "$bzimage" ]] && bzimage="./bzImage-68433066"
> qemu-system-x86_64 \
>         -m 2G \
>         -smp 2 \
>         -kernel $bzimage \
>         -append "console=ttyS0 root=/dev/sda1 earlyprintk=serial net.ifnames=0 thunderbolt.dyndbg" \
>         -drive file=./disk-68433066.raw,format=raw \
>         -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10023-:22 \
>         -cpu host \
>         -net nic,model=e1000 \
>         -enable-kvm \
>         -nographic \
>         2>&1 | tee vmd.log
> "
> 
> gcc -pthread -o repro repro.c
> scp -P 10023 repro root@...alhost:/root/
> 
> And then access to above vm and execute the reproduced binary to reproduce.
> Sometimes it takes some time to reproduce the problem.
> Hope the above information helps you to reproduce the problem next time.
> 

Thanks! This was very useful as I've been able to reproduce fairly
reliably using this script. I suspect something was sufficiently
different in my guest env that made this difficult to reproduce for
whatever reason.

The source of the -EAGAIN in this particular case is the
mmap_read_lock_killable() call down in __get_user_pages_locked(). I
haven't fully characterized test behavior, but I suppose this is being
interrupted somehow or another. From there, this only occasionally
reproduces the ext4 warning because the iomap iteration code
(iomap_dio_bio_iter()) only returns the error when no progress has been
made to that point, otherwise it's a short write.

So this is indeed different from the io_uring variant I described
earlier, but is still another instance of an -EAGAIN return unrelated to
the overwrite logic or *_NOWAIT.

Brian

> I saw syzbot already verified your latest patch and it's passed.
> https://syzkaller.appspot.com/x/log.txt?x=17939bc1a80000
> 
> Best Regards,
> Thanks!
> 
> > 
> > Brian
> > 
> > [1] https://syzkaller.appspot.com/x/patch.diff?x=109a7c96a80000
> > 
> > > All information: https://github.com/xupengfe/syzkaller_logs/tree/main/230730_134501_ext4_file_write_iter
> > > Reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230730_134501_ext4_file_write_iter/repro.c
> > > repro.prog(syscall reproduced steps): https://github.com/xupengfe/syzkaller_logs/blob/main/230730_134501_ext4_file_write_iter/repro.prog
> > > repro.report: https://github.com/xupengfe/syzkaller_logs/blob/main/230730_134501_ext4_file_write_iter/repro.report
> > > Bisect log: https://github.com/xupengfe/syzkaller_logs/blob/main/230730_134501_ext4_file_write_iter/bisect_info.log
> > > Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230730_134501_ext4_file_write_iter/kconfig_origin
> > > Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230730_134501_ext4_file_write_iter/6eaae198076080886b9e7d57f4ae06fa782f90ef_dmesg.log
> > > 
> > > Best Regards,
> > > Thanks!
> > > 
> > > > 
> > > > ---
> > > > If you want syzbot to run the reproducer, reply with:
> > > > #syz test: git://repo/address.git branch-or-commit-hash
> > > > If you attach or paste a git patch, syzbot will apply it before testing.
> > > 
> > 
> 

Powered by blists - more mailing lists