linux-kernel - Re: Widespread crashes in -next, bisected to 'mm: drop HASH

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Sat, 20 May 2017 12:38:08 -0400
From:   Pasha Tatashin <pasha.tatashin@...cle.com>
To:     Guenter Roeck <linux@...ck-us.net>,
        Michal Hocko <mhocko@...nel.org>
Cc:     linux-kernel@...r.kernel.org,
        Andrew Morton <akpm@...ux-foundation.org>,
        Stephen Rothwell <sfr@...b.auug.org.au>,
        Kevin Hilman <khilman@...libre.com>
Subject: Re: Widespread crashes in -next, bisected to 'mm: drop HASH_ADAPT'

The problem is due to 32-bit integer overflow in:

ADAPT_SCALE_BASE and adapt

In dcache_init_early() that is causing the problem. It was not enabled 
before 'mm: drop HASH_ADAPT' but is enabled now, and it should follow 
right after: "PID hash table entries: 1024 (order: 0, 4096 bytes)"

main()
   pidhash_init();
   vfs_caches_init_early();
     dcache_init_early()
       alloc_large_system_hash("Dentry cache", ...)

for (adapt = ADAPT_SCALE_NPAGES; adapt < numentries;
			     adapt <<= ADAPT_SCALE_SHIFT)

numentries  is very small, so it should be always smaller than adapt, 
and algorithm should not kick in, but 32-bit causes adapt to be smaller 
than numentries.

I will send out an updated "mm: Adaptive hash table scaling", with "mm: 
drop HASH_ADAPT" integrated.

Pasha

On 05/20/2017 10:21 AM, Guenter Roeck wrote:
> On 05/20/2017 12:26 AM, Michal Hocko wrote:
>> On Fri 19-05-17 09:46:23, Guenter Roeck wrote:
>>> Hi,
>>>
>>> my qemu tests of next-20170519 show the following results:
>>>     total: 122 pass: 30 fail: 92
>>>
>>> I won't bother listing all of the failures; they are available at
>>> http://kerneltests.org/builders. I bisected one (openrisc, because
>>> it gives me some console output before dying). It points to
>>> 'mm: drop HASH_ADAPT' as the culprit. Bisect log is attached.
>>>
>>> A quick glance suggests that 64 bit kernels pass and 32 bit kernels 
>>> fail.
>>> 32-bit x86 images fail and should provide an easy test case.
>>
>> Hmm, this is quite unexpected as the patch is not supposed to change
>> things much. It just removes the flag and perform the new hash scaling
> 
> It may well be that the problem is introduced with an earlier patch and 
> just
> enabled by this one.
> 
>> automatically for all requeusts which do not have any high limit.
>> Some of those didn't have HASH_ADAPT before but that shouldn't change
>> the picture much. The only thing that I can imagine is that what
>> formerly failed for early memblock allocations is now suceeding and that
>> depletes the early memory. Do you have any serial console from the boot?
>>
> 
> They are all the same. Either nothing or the following. Picking a couple:
> 
> metag:
> 
> Linux version 4.12.0-rc1-next-20170519 (groeck@...iter.roeck-us.net) 
> (gcc version 4.2.4 (IMG-1.4.0.300)) #1 Fri May 19 00:50:50 PDT 2017
> LNKGET/SET go through cache but CONFIG_METAG_LNKGET_AROUND_CACHE=y
> DA present
> console [ttyDA1] enabled
> OF: fdt: Machine model: toumaz,tz1090
> Machine name: Generic Meta
> Node 0: start_pfn = 0xb0000, low = 0xbfff7
> Zone ranges:
>    Normal   [mem 0x00000000b0000000-0x00000000bfff6fff]
> Movable zone start for each node
> Early memory node ranges
>    node   0: [mem 0x00000000b0000000-0x00000000bfff6fff]
> Initmem setup node 0 [mem 0x00000000b0000000-0x00000000bfff6fff]
> Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 65015
> Kernel command line: rdinit=/sbin/init doreboot
> PID hash table entries: 1024 (order: 0, 4096 bytes)
> 
> crisv32:
> 
> Linux version 4.12.0-rc1-next-20170519 (groeck@...ktop.roeck-us.net) 
> (gcc version 4.9.2 (Buildroot 2015.02-rc1-00005-gb13bd8e-dirty) ) #1 Fri 
> May 19 00:52:55 PDT 2017
> bootconsole [early0] enabled
> Setting up paging and the MMU.
> Linux/CRISv32 port on ETRAX FS (C) 2003, 2004 Axis Communications AB
> Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 4080
> Kernel command line: console=ttyS0,115200,N,8 rdinit=/sbin/init
> PID hash table entries: 128 (order: -4, 512 bytes)
> 
> powerpc:mpc8548cds:
> 
> Memory CAM mapping: 256 Mb, residual: 0Mb
> Linux version 4.12.0-rc1-next-20170519 (groeck@...iter.roeck-us.net) 
> (gcc version 4.8.1 (GCC) ) #1 Fri May 19 01:17:29 PDT 2017
> Found initrd at 0xc4000000:0xc4200c00
> Using MPC85xx CDS machine description
> bootconsole [udbg0] enabled
> -----------------------------------------------------
> phys_mem_size     = 0x10000000
> dcache_bsize      = 0x20
> icache_bsize      = 0x20
> cpu_features      = 0x0000000012100460
>    possible        = 0x0000000012100460
>    always          = 0x0000000000100000
> cpu_user_features = 0x84e08000 0x08000000
> mmu_features      = 0x00020010
> -----------------------------------------------------
> mpc85xx_cds_setup_arch()
> Could not find FPGA node.
> Zone ranges:
>    DMA      [mem 0x0000000000000000-0x000000000fffffff]
>    Normal   empty
> Movable zone start for each node
> Early memory node ranges
>    node   0: [mem 0x0000000000000000-0x000000000fffffff]
> Initmem setup node 0 [mem 0x0000000000000000-0x000000000fffffff]
> MMU: Allocated 1088 bytes of context maps for 255 contexts
> Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 65024
> Kernel command line: rdinit=/sbin/init console=ttyS0 console=tty doreboot
> PID hash table entries: 1024 (order: 0, 4096 bytes)
> 
> Guenter