Message-ID: <af401afc7e32d9c0eeb6b36da70d2488@3xo.fr>
Date: Thu, 24 Apr 2025 09:40:18 +0200
From: Nicolas Baranger <nicolas.baranger@....fr>
To: Paulo Alcantara <pc@...guebit.com>
Cc: Christoph Hellwig <hch@...radead.org>, hch@....de, David Howells
 <dhowells@...hat.com>, netfs@...ts.linux.dev, linux-cifs@...r.kernel.org,
 linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org, Steve French
 <smfrench@...il.com>, Jeff Layton <jlayton@...nel.org>, Christian Brauner
 <brauner@...nel.org>
Subject: Re: [netfs/cifs - Linux 6.14] loop on file cat + file copy when files
 are on CIFS share

Hi Paolo

Thanks again for help.

I'm sorry, I made a mistake in my answer yesterday:

> After a lot of testing, the mounts buffers values: rsize=65536, 
> wsize=65536, bsize=16777216,...

The actual values in /etc/fstab are:
rsize=4194304,wsize=4194304,bsize=16777216

But negociated values in /proc/mounts are:
rsize=65536,wsize=65536,bsize=16777216

And don't know if it's related but I have:
grep -i maxbuf /proc/fs/cifs/DebugData
CIFSMaxBufSize: 16384

I've just force a manual 'mount -o remount' and now I have in 
/proc/mounts the good values (SMB version is 3.1.1).
Where does this behavior comes from ?

After some search, it appears that when the CIFS share is mounted by 
systemd option x-systemd.automount (for example doing 'ls' in the mount 
point directory), negociated values are:
rsize=65536,wsize=65536,bsize=16777216
If I umount / remount manually, the negociated values are those defined 
in /etc/fstab !

Don't know if it's a normal behavior but it is a source of errors / 
mistake and makes troubleshooting performance issues harder

Kind regards
Nicolas



Le 2025-04-23 18:28, Nicolas Baranger a écrit :

> Hi Paolo
> 
> Thanks for answer, all explanations and help
> 
> I'm happy you found those 2 bugs and starting to patch them.
> Reading your answer, I want to remember that I already found a bug in 
> cifs DIO starting from Linux 6.10 (when cifs statring to use netfs to 
> do its IO) and it was fixed by David and Christoph
> full story here: 
> https://lore.kernel.org/all/14271ed82a5be7fcc5ceea5f68a10bbd@manguebit.com/T/
> 
>> I've noticed that you disabled caching with 'cache=none', is there any
>> particular reason for that?
> 
> Yes, it's related with the precedent use case describes in the other 
> bug:
> For backuping servers, I've got some KSMBD cifs share on which there 
> are some 4TB+ sparses files (back-files) which are LUKS + BTRFS 
> formatted.
> The cifs share is mounted on servers and each server mount its own 
> back-file as a block device and make its backup inside this crypted 
> disk file
> Due to performance issues, it is required that the disk files are using 
> 4KB block and are mounted in servers using losetup DIO option (+ 4K 
> block size options)
> When I use something else than 'cache=none', sometimes the BTRFS 
> filesystem on the back file get corrupted and I also need to mount the 
> BTRFS filesystem with 'space_cache=v2' to avoid filesystem corruption
> 
>> Have you also set rsize, wsize and bsize mount options?  If so, why?
> 
> After a lot of testing, the mounts buffers values: rsize=65536, 
> wsize=65536, bsize=16777216, are the one which provide the best 
> performances with no corruptions on the back-file filesystem and with 
> these options a ~2TB backup is possible in few hours during  timeframe 
> ~1 -> ~5 AM each night
> 
> For me it's important that kernel async DIO on netfs continue to work 
> as it's used by all my production backup system (transfer speed ratio 
> compared with and without DIO is between 10 to 25)
> 
> I will try the patch "[PATCH] netfs: Fix setting of transferred bytes 
> with short DIO reads", thanks
> 
> Let me know if you need further explanations,
> 
> Kind regards
> Nicolas Baranger
> 
> Le 2025-04-22 01:45, Paulo Alcantara a écrit :
> 
> Nicolas Baranger <nicolas.baranger@....fr> writes:
> 
> If you need more traces or details on (both?) issues :
> 
> - 1) infinite loop issue during 'cat' or 'copy' since Linux 6.14.0
> 
> - 2) (don't know if it's related) the very high number of several bytes
> TCP packets transmitted in SMB transaction (more than a hundred) for a 
> 5
> bytes file transfert under Linux 6.13.8
> According to your mount options and network traces, cat(1) is 
> attempting
> to read 16M from 'toto' file, in which case netfslib will create 256
> subrequests to handle 64K (rsize=65536) reads from 'toto' file.
> 
> The first 64K read at offset 0 succeeds and server returns 5 bytes, the
> client then sets NETFS_SREQ_HIT_EOF to indicate that this subrequest 
> hit
> the EOF.  The next subrequests will still be processed by netfslib and
> sent to the server, but they all fail with STATUS_END_OF_FILE.
> 
> So, the problem is with short DIO reads in netfslib that are not being
> handled correctly.  It is returning a fixed number of bytes read to
> every read(2) call in your cat command, 16711680 bytes which is the
> offset of last subrequest.  This will make cat(1) retry forever as
> netfslib is failing to return the correct number of bytes read,
> including EOF.
> 
> While testing a potential fix, I also found other problems with DIO in
> cifs.ko, so I'm working with Dave to get the proper fixes for both
> netfslib and cifs.ko.
> 
> I've noticed that you disabled caching with 'cache=none', is there any
> particular reason for that?
> 
> Have you also set rsize, wsize and bsize mount options?  If so, why?
> 
> If you want to keep 'cache=none', then a possible workaround for you
> would be making rsize and wsize always greater than bsize.  The default
> values (rsize=4194304,wsize=4194304,bsize=1048576) would do it.

Powered by blists - more mailing lists