linux-kernel - Re: v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CA+G9fYtrsLHvMH=ofmdS3MMsMTEj3k0PD7=qsRsA4WkSqLkCzQ@mail.gmail.com>
Date: Thu, 3 Apr 2025 16:17:44 +0530
From: Naresh Kamboju <naresh.kamboju@...aro.org>
To: Nathan Chancellor <nathan@...nel.org>
Cc: open list <linux-kernel@...r.kernel.org>, 
	Linux ARM <linux-arm-kernel@...ts.infradead.org>, lkft-triage@...ts.linaro.org, 
	Linux Regressions <regressions@...ts.linux.dev>, clang-built-linux <llvm@...ts.linux.dev>, 
	Arnd Bergmann <arnd@...db.de>, Ard Biesheuvel <ardb@...nel.org>, 
	Dan Carpenter <dan.carpenter@...aro.org>, Anders Roxell <anders.roxell@...aro.org>
Subject: Re: v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b
 dragonboard-410c dragonboard-845c no console output

On Thu, 3 Apr 2025 at 06:48, Nathan Chancellor <nathan@...nel.org> wrote:
>
> Hi Naresh,
>
> On Wed, Apr 02, 2025 at 06:19:19PM +0530, Naresh Kamboju wrote:
> > Regressions on rk3399-rock-pi-4b, dragonboard-410c and dragonboard-845c
> > the lto-thing, hardening and lto-full config boot failed with toolchain
> > clang-nightly on the mainline master branch with no console output.
> >
> > First seen on the v6.14-12245-g91e5bfe317d8
> >  Good: v6.14-11270-g08733088b566
> >  Bad: v6.14-12245-g91e5bfe317d8
> >
> > Regressions found on rk3399-rock-pi-4b:
> >   - boot/clang-nightly-lkftconfig-kselftest
> >   - boot/clang-nightly-lkftconfig-lto-thing
> >   - boot/clang-nightly-lkftconfig-hardening
> >   - boot/clang-nightly-lkftconfig-lto-full
> >
> > Regressions found on dragonboard-410c:
> >   - boot/clang-nightly-lkftconfig-lto-thing
> >   - boot/clang-nightly-lkftconfig-lto-full
> >   - boot/clang-nightly-lkftconfig-hardening
> >
> > Regressions found on dragonboard-845c:
> >   - boot/clang-nightly-lkftconfig-hardening
> >   - boot/clang-nightly-lkftconfig-lto-thing
> >
> > Regression Analysis:
> >  - New regression? Yes
> >  - Reproducibility? Yes
> >
> > Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c
> > no console output
> >
> > Reported-by: Linux Kernel Functional Testing <lkft@...aro.org>
> >
> > ## Boot log
> > Starting kernel
> > ...
> > <No console output>
> >
> >
> > ## Source
> > * Kernel version: 6.14.0
> > * Git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> > * Git sha: 91e5bfe317d8f8471fbaa3e70cf66cae1314a516
> > * Git describe: v6.14-12245-g91e5bfe317d8
> > * Project details:
> > https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-12245-g91e5bfe317d8/
> > * Architectures: arm64 (rk3399-rock-pi-4b, dragonboard-410c, dragonboard-845c)
> > * Toolchains: clang-nightly (Debian clang version 21.0.0 )
>
> The version string for the toolchain seems to be slightly truncated but
> from the configuration it is:
>
>   Debian clang version 21.0.0 (++20250330105456+3b3d1a5c2614-1~exp1~20250330225508.1357)
>
> which can be parsed with our parse-debian-clang.py [1] to get more
> information:
>
>   $ scripts/parse-debian-clang.py -p -v 'Debian clang version 21.0.0 (++20250330105456+3b3d1a5c2614-1~exp1~20250330225508.1357)'
>   clang checkout date: 2025-03-30 10:54 UTC (3 days, 14:17:24.134272 ago)
>   clang revision: 3b3d1a5c2614
>   clang revision link: https://github.com/llvm/llvm-project/commit/3b3d1a5c2614
>
> Our CI is using a slightly newer version:
>
>   $ scripts/parse-debian-clang.py -p -v 'Debian clang version 21.0.0 (++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)'
>   clang checkout date: 2025-04-01 11:25 UTC (1 day, 13:49:39.394836 ago)
>   clang revision: 290d7b82cb5d
>   clang revision link: https://github.com/llvm/llvm-project/commit/290d7b82cb5d
>
> Can you see if it is reproducible with that revision?

I’ve re-run the tests to validate the boot behavior on the Rock Pi 4 board
with different Clang nightly versions and the latest mainline kernel.

The combination using clang-nightly:20250319 successfully booted the
Rock Pi 4 board.
However, the combination using clang-nightly:20250401 failed to boot
the same board.

"name": "clang",
"version": "21.0.0",
"version_full": "Debian clang version 21.0.0
(++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)"

Reference:
 - https://lkft.validation.linaro.org/scheduler/job/8196258
 - https://lkft.validation.linaro.org/scheduler/job/8196275

- Naresh



> For what it's
> worth, both of the arm64 boxes I have can boot a ThinLTO kernel compiled
> with a version of LLVM @ 749535ba2808e133682074f712ac6829335f8875, so it
> could be something that was broken for a little bit but Debian happened
> to sync before the fix was committed.
>
> [1]: https://github.com/ClangBuiltLinux/continuous-integration2/blob/fe4844afc1be91d469fc162c8a179f23fafb9384/scripts/parse-debian-clang.py
>
> Cheers,
> Nathan