Date:   Mon, 31 Oct 2022 10:05:58 +0000
From:   "John Thomson" <lists@...nthomson.fastmail.com.au>
To:     "Feng Tang" <feng.tang@...el.com>,
        "Vlastimil Babka" <vbabka@...e.cz>
Cc:     "Andrew Morton" <akpm@...ux-foundation.org>,
        "Christoph Lameter" <cl@...ux.com>,
        "Pekka Enberg" <penberg@...nel.org>,
        "David Rientjes" <rientjes@...gle.com>,
        "Joonsoo Kim" <iamjoonsoo.kim@....com>,
        "Roman Gushchin" <roman.gushchin@...ux.dev>,
        "Hyeonggon Yoo" <42.hyeyoo@...il.com>,
        "Dmitry Vyukov" <dvyukov@...gle.com>,
        "Jonathan Corbet" <corbet@....net>,
        "Andrey Konovalov" <andreyknvl@...il.com>,
        "Hansen, Dave" <dave.hansen@...el.com>,
        "linux-mm@...ck.org" <linux-mm@...ck.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "kasan-dev@...glegroups.com" <kasan-dev@...glegroups.com>,
        "Robin Murphy" <robin.murphy@....com>,
        "John Garry" <john.garry@...wei.com>,
        "Kefeng Wang" <wangkefeng.wang@...wei.com>
Subject: Re: [PATCH v6 1/4] mm/slub: enable debugging memory wasting of kmalloc

On Mon, 31 Oct 2022, at 02:36, Feng Tang wrote:
> Hi John,
>
> Thanks for the bisecting and reporting!
>
> On Mon, Oct 31, 2022 at 05:30:24AM +0800, Vlastimil Babka wrote:
>> On 10/30/22 20:23, John Thomson wrote:
>> > On Tue, 13 Sep 2022, at 06:54, Feng Tang wrote:
>> >> kmalloc's API family is critical for mm, with one nature that it will
>> >> round up the request size to a fixed one (mostly power of 2). Say
>> >> when user requests memory for '2^n + 1' bytes, actually 2^(n+1) bytes
>> >> could be allocated, so in worst case, there is around 50% memory
>> >> space waste.
>> > 
>> > 
>> > I have a ralink mt7621 router running Openwrt, using the mips ZBOOT kernel, and appear to have bisected
>> > a very-nearly-clean kernel v6.1rc-2 boot issue to this commit.
>> > I have 3 commits atop 6.1-rc2: fix a ZBOOT compile error, use the Openwrt LZMA options,
>> > and enable DEBUG_ZBOOT for my platform. I am compiling my kernel within the Openwrt build system.
>> > No guarantees this is not due to something I am doing wrong, but any insight would be greatly appreciated.
>> > 
>> > 
>> > On UART, No indication of the (once extracted) kernel booting:
>> > 
>> > transfer started ......................................... transfer ok, time=2.01s
>> > setting up elf image... OK
>> > jumping to kernel code
>> > zimage at:     80BA4100 810D4720
>> > Uncompressing Linux at load address 80001000
>> > Copy device tree to address  80B96EE0
>> > Now, booting the kernel...
>> 
>> It's weird that the commit would cause no output so early, SLUB code is 
>> run only later.
> 
> I noticed your cmdline has console setting, could you enable the
> earlyprintk in cmdline like "earlyprintk=ttyS0,115200" etc to see
> if there is more message printed out.

Still nothing from vmlinux with earlykprint on UART unless revert.

>
> Also I want to confirm this is a boot failure and not only a boot
> message missing.

Yes, boot failure.
Network comes up automatically on successful boot. Not happening when no kernel UART

>
>> > Nothing follows
>> > 
>> > 6edf2576a6cc  ("mm/slub: enable debugging memory wasting of kmalloc") reverted, normal boot:
>> > transfer started ......................................... transfer ok, time=2.01s
>> > setting up elf image... OK
>> > jumping to kernel code
>> > zimage at:     80BA4100 810D47A4
>> > Uncompressing Linux at load address 80001000
>> > Copy device tree to address  80B96EE0
>> > Now, booting the kernel...
>> > 
>> > [    0.000000] Linux version 6.1.0-rc2 (john@...n) (mipsel-openwrt-linux-musl-gcc (OpenWrt GCC 11.3.0 r19724+16-1521d5f453) 11.3.0, GNU ld (GNU Binutils) 2.37) #0 SMP Fri Oct 28 03:48:10 2022
>> > [    0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3
>> > [    0.000000] printk: bootconsole [early0] enabled
>> > [    0.000000] CPU0 revision is: 0001992f (MIPS 1004Kc)
>> > [    0.000000] MIPS: machine is MikroTik RouterBOARD 760iGS
>> > [    0.000000] Initrd not found or empty - disabling initrd
>> > [    0.000000] VPE topology {2,2} total 4
>> > [    0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
>> > [    0.000000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
>> > [    0.000000] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
>> > [    0.000000] Zone ranges:
>> > [    0.000000]   Normal   [mem 0x0000000000000000-0x000000000fffffff]
>> > [    0.000000]   HighMem  empty
>> > [    0.000000] Movable zone start for each node
>> > [    0.000000] Early memory node ranges
>> > [    0.000000]   node   0: [mem 0x0000000000000000-0x000000000fffffff]
>> > [    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000000fffffff]
>> > [    0.000000] percpu: Embedded 11 pages/cpu s16064 r8192 d20800 u45056
>> > [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 64960
>> > [    0.000000] Kernel command line: console=ttyS0,115200 rootfstype=squashfs,jffs2
>> > [    0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
>> > [    0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536 bytes, linear)
>> > [    0.000000] Writing ErrCtl register=00019146
>> > [    0.000000] Readback ErrCtl register=00019146
>> > [    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
>> > [    0.000000] Memory: 246220K/262144K available (7455K kernel code, 628K rwdata, 1308K rodata, 3524K init, 245K bss, 15924K reserved, 0K cma-reserved, 0K highmem)
>> > [    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
>> > [    0.000000] rcu: Hierarchical RCU implementation.
>> > 
>> > 
>> > boot continues as expected
>> > 
>> > 
>> > possibly relevant config options:
>> > grep -E '(SLUB|SLAB)' .config
>> > # SLAB allocator options
>> > # CONFIG_SLAB is not set
>> > CONFIG_SLUB=y
>> > CONFIG_SLAB_MERGE_DEFAULT=y
>> > # CONFIG_SLAB_FREELIST_RANDOM is not set
>> > # CONFIG_SLAB_FREELIST_HARDENED is not set
>> > # CONFIG_SLUB_STATS is not set
>> > CONFIG_SLUB_CPU_PARTIAL=y
>> > # end of SLAB allocator options
>> > # CONFIG_SLUB_DEBUG is not set
>> 
>> Also not having CONFIG_SLUB_DEBUG enabled means most of the code the 
>> patch/commit touches is not even active.
>> Could this be some miscompile or code layout change exposing some 
>> different bug, hmm.

Yes, it could be.

>> Is it any different if you do enable CONFIG_SLUB_DEBUG ?

No change

>> Or change to CONFIG_SLAB? (that would be really weird if not)

This boots fine

> I haven't found any clue from the code either, and I compiled
> kernel with the config above and tested booting on an Alder-lake
> desktop and a QEMU, which boot fine.
>
> Could you provide the full kernel config and demsg (in compressed
> format if you think it's too big), so we can check more?

Attached

> Thanks,
> Feng

vmlinux is bigger, and entry point is larger (0x8074081c vs 0x807407dc revert vs 0x8073fcbc),
so that may be it? Or not?
revert + SLUB_DEBUG + SLUB_DEBUG_ON is bigger still, but does successfully boot.
vmlinux entry point is 0x8074705c


transfer started ......................................... transfer ok, time=2.01s
setting up elf image... OK
jumping to kernel code
zimage at:     80BA4100 810D6FA0
Uncompressing Linux at load address 80001000
Copy device tree to address  80B9EEE0
Now, booting the kernel...
[    0.000000] Linux version 6.1.0-rc2 (john@...n) (mipsel-openwrt-linux-musl-gc
c (OpenWrt GCC 11.3.0 r19724+16-1521d5f453) 11.3.0, GNU ld (GNU Binutils) 2.37) 
#0 SMP Fri Oct 28 03:48:10 2022


 I will keep looking.

Thank you,
-- 
  John Thomson
Download attachment "mt7621-mm-waste-debug.tar.zstd" of type "application/octet-stream" (29758 bytes)

Powered by blists - more mailing lists