linux-kernel - Re: [REGRESSION] Corruption on cifs / smb write on ARM, kernels 6.3-6.9

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CABaPp_iLCoCAW=2jHEvgM15UJiwWXq5BXh0rCXtu-80tk6Zuvw@mail.gmail.com>
Date: Sat, 28 Sep 2024 09:37:04 -0700
From: james young <pronoiac@...il.com>
To: Wang Yugui <wangyugui@...-tech.com>
Cc: pronoiac+kernel@...il.com, stable@...r.kernel.org, 
	regressions@...ts.linux.dev, linux-cifs@...r.kernel.org, 
	David Howells <dhowells@...hat.com>, linux-kernel@...r.kernel.org, 
	Steve French <sfrench@...ba.org>, smfrench@...il.com
Subject: Re: [REGRESSION] Corruption on cifs / smb write on ARM, kernels 6.3-6.9

I retraced my steps:
* looking for the breaking commit, between 6.2 and 6.3-rc1
* I switched to checksumming the stream and the written file; this can
save time, compared to decompression
* I checked for lzop, pigz, and pbzip2

So, breakage. I landed on different commits:
last working commit. ok: lzop, pigz, pbzip2.
16541195c6d9 cifs: Add a function to read into an iter from a socket

first broken commit. lzop failed.
d08089f649a0 cifs: Change the I/O paths to use an iterator rather than
a page list

That broken commit is right before my previous "last good" and "break".

I'm seeing some inconsistencies. I'd *thought* I was careful with dtb
files and .config; I might have dropped the ball occasionally, or
there's something else, I don't know what, that I'm stumbling over.

To check for marginal hardware, I tried another Raspberry Pi 4. I
verified baseline 6.6.52 didn't work there, and stopped there. It
doesn't have any cooling; it *almost certainly* would throttle for
thermal reasons, but I didn't want to push it.

-James


On Tue, Sep 24, 2024 at 9:35 PM james young <pronoiac@...il.com> wrote:
>
> On request:
> * adding another cc for Steven
> * I tested 6.6.52, without any extra commits: it was bad.
>
> -James
>
> On Mon, Sep 23, 2024 at 12:36 PM james young <pronoiac@...il.com> wrote:
> >
> > Hey there -
> >
> > On Sun, Sep 22, 2024 at 4:55 PM Wang Yugui <wangyugui@...-tech.com> wrote:
> > >
> > > Hi,
> > >
> > > > I was benchmarking some compressors, piping to and from a network share on a NAS, and some consistently wrote corrupted data.
> >
> > > > Important commits:
> > > > It looked like both the breakage and the fix came in during rc1 releases.
> > > >
> > > > Breakage, v6.3-rc1:
> > > > I manually bisected commits in fs/smb* and fs/cifs.
> > > >
> > > > 3d78fe73fa12 cifs: Build the RDMA SGE list directly from an iterator
> > > > > lzop and pigz worked. last working. test in progress: pbzip2
> >
> > This is a first for me: lzop was fine, but pbzip2 still had issues,
> > roughly a clock hour into compression. (When lzop has issues, it's
> > usually within a minute or two.)
> >
> >
> > > > 607aea3cc2a8 cifs: Remove unused code
> > > > > lzop didn't work. first broken
> > > >
> > > >
> > > > Fix, v6.10-rc1:
> > > > I manually bisected commits in fs/smb.
> > > >
> > > > 69c3c023af25 cifs: Implement netfslib hooks
> > > > > lzop didn't work. last broken one
> > > >
> > > > 3ee1a1fc3981 cifs: Cut over to using netfslib
> > > > > lzop, pigz, pbzip2, all worked. first fixed one
> >
> > > I checked 607aea3cc2a8, it just removed some code in #if 0 ... #endif.
> > > so this regression is not introduced in 607aea3cc2a8,  but the reproduce
> > > frequency is changed here.
> >
> > I agree. The pbzip2 results above, regarding the break bisection I
> > landed on: they mark when it became more of an issue, but not when it
> > started.
> >
> > I could re-run tests and dig into possible false negatives. It'll be
> > slower going, though.
> >
> >
> > > Another issue in 6.6.y maybe related
> > > https://lore.kernel.org/linux-fsdevel/9e8f8872-f51b-4a09-a92c-49218748dd62@meta.com/T/
> >
> > In comparison: I'm relieved that my issue is something that can be
> > tested within hours, on one device.
> >
> >
> > > Do this regression still happen after the following patches are applied?
> > >
> > > a60cc288a1a2 :Luis Chamberlain: test_xarray: add tests for advanced multi-index use
> > > a08c7193e4f1 :Sidhartha Kumar: mm/filemap: remove hugetlb special casing in filemap.c
> > > 6212eb4d7a63 :Hongbo Li: mm/filemap: avoid type conversion
> > >
> > > de60fd8ddeda :Kairui Song: mm/filemap: return early if failed to allocate memory for split
> > > b2ebcf9d3d5a :Kairui Song: mm/filemap: clean up hugetlb exclusion code
> > > a4864671ca0b :Kairui Song: lib/xarray: introduce a new helper xas_get_order
> > > 6758c1128ceb :Kairui Song: mm/filemap: optimize filemap folio adding
> >
> > No luck: I cherry-picked those commits into 6.6.52, and upon testing
> > lzop, the file didn't match the stream, and decompression failed.
> >
> > Thank you for investigating, and giving me something to try!
> >
> > -James