Message-ID: <ca241957-389b-471d-a450-36b5e00ebe5a@gmx.de>
Date: Mon, 2 Dec 2024 17:35:05 +0100
From: Helge Deller <deller@....de>
To: matoro <matoro_mailinglist_kernel@...oro.tk>,
 John David Anglin <dave.anglin@...l.net>
Cc: Magnus Lindholm <linmag7@...il.com>,
 Linux Parisc <linux-parisc@...r.kernel.org>, deller@...nel.org,
 Sam James <sam@...too.org>,
 Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: [bisected] ext4 corruption on parisc since 6.12

Hi Matoro,

On 12/2/24 4:31 PM, matoro wrote:
> On 2024-12-02 09:54, John David Anglin wrote:
>> On 2024-12-02 1:30 a.m., Magnus Lindholm wrote:
>>> On Mon, Dec 2, 2024 at 5:55 AM matoro
>>> <matoro_mailinglist_kernel@...oro.tk> wrote:
>>>> Hmm, this is my config, also on an rp3440:
>>>>
>>>> #
>>>> # Timers subsystem
>>>> #
>>>> CONFIG_HZ_PERIODIC=y
>>>> # CONFIG_NO_HZ_IDLE is not set
>>>> # CONFIG_NO_HZ is not set
>>>> # CONFIG_HIGH_RES_TIMERS is not set
>>>> # end of Timers subsystem
>>>>
>>>> lindholm can confirm on their hardware/config.  Maybe you can try that and
>>>> see if you can reproduce?  I will try your config as well.
>>> Hi, I'm on a HPC8000 "parisc64 PA8800 (Mako) 9000/785/C8000". I can confirm
>>> that building a kernel CONFIG_SMP=n will mitigate this problem.
>>> I haven't messed around with the config in the Timer subsystem so in my case the
>>> parameters suggested are unset. (my config looks like matoros)
>> The clockevent driver was tested on both rp3440 and c8000, and some other SMP machines.
>> Helge knows details.  I have used it on rp3440 and c8000.
>>
>> I would try my settings.  The primary reason in switching to the clockevent drivers was to
>> improve clock resolution.  The best resolution with the old drivers was 1 ms at 1000 HZ.
>> This caused problems with various package tests.  If config is the issue, probably
>> CONFIG_HIGH_RES_TIMERS needs to be forced when clockevent drivers are used.
>>
>> Almost every other system uses the clockevent drivers.  So, there was a risk that parisc would
>> become unsupported.
>>
>> I wonder if this could be caused by dead RTC battery.  Did you check output of date command?
>> Maybe a dead RTC battery interacts badly with clockevent drivers.
>>
>> I run ntp on all my machines.
>>
>> What files have bad dates (i.e., is this really a ext4 file system issue) or is it just that system has
>> a bad clock?
>>
>> Dave
>
> The files that have bad dates seem to be the ones /init on this system touches at early boot.  See the output here:  https://paste.matoro.tk/8cq8omg
>
> When booted into the bad kernel, date(1) works and displays the correct time.  I'm using chrony for time syncing as well.
>
> After switching to the config specified above, boot hangs before even getting to userspace with the following output:
>
> [   12.473410] 0000:e0:01.1: ttyS2 at MMIO 0xfffffffff4050038 (irq = 73, base_baud = 115200) is a 16550A
> [   12.757386] sym0: <1010-66> rev 0x1 at pci 0000:20:01.0 irq 70
> [   12.761419] sym0: PA-RISC Firmware, ID 7, Fast-80, LVD, parity checking
> [   12.885367] sym0: SCSI BUS has been reset.
> [   12.889389] scsi host0: sym-2.2.3
> [   13.053380] sym1: <1010-66> rev 0x1 at pci 0000:20:01.1 irq 71
> [   13.055515] sym1: PA-RISC Firmware, ID 7, Fast-80, LVD, parity checking
> [   13.165367] sym1: SCSI BUS has been reset.
> [   13.169388] scsi host1: sym-2.2.3
> [   13.208927] rtc-generic rtc-generic: registered as rtc0
> [   13.281367] rtc-generic rtc-generic: setting system clock to 2024-12-02T07:17:02 UTC (1733123822)
> [   13.281367] NET: Registered PF_INET6 protocol family
> [   13.281367] Segment Routing with IPv6
> [   13.281367] In-situ OAM (IOAM) with IPv6
> [   13.281367] registered taskstats version 1

this message...:

> [   13.281367] Unstable clock detected, switching default tracing clock to "global"
> [   13.281367] If you want to keep using the local clock, then add:
> [   13.281367]   "trace_clock=local"
> [   13.281367] on the kernel command line

is very misleading and has nothing to do with the parisc system clock from my commit.
You will find this message in all older kernels you had booted as well.
It refers to the tracing clock, which isn't used here.

> > At the end there the clock seems to stop progressing forward, as
> there are several real-time seconds that elapse in between messages
> with the same timestamp.  So I'm completely unable to boot with this
> config at all.

I think the problems you see are manifold, as it's probably not just
triggered by the kernel clock patch itself.
I've seen various issues on debian in the past, as long as we hadn't built glibc
with 64-bit time64_t enabled.
One such issue is what you see now: your system fails to bring up userspace
because the file date stamps are a lot in the future (even if the dates
are wrong, it should be able to boot).

So, the kernel patch might trigger some time inconsistencies, but
having non-time64_t enabled glibc is probably the main reason why your boot fails.

Both Dave and myself did test this commit:
  commit b5ff52be891347f8847872c49d7a5c2fa29400a7
  parisc: Convert to generic clockevents

Nevertheless, maybe on your system one of the SMP CPUs returned an invalid
time (e.g. outside of 32-bit range) and thus your glibc seems to have gone crazy.
That's currently the only idea I have right now which somehow might explain your situation.

I think we need more testing.

Helge

Powered by blists - more mailing lists