linux-kernel - Re: [PATCH v2 1/2] mm, thp: check page mapping when truncating page cache

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <67906bf5-4de9-8433-3d70-cc8fc5cc2347@linux.alibaba.com>
Date:   Thu, 30 Sep 2021 09:54:04 +0800
From:   Rongwei Wang <rongwei.wang@...ux.alibaba.com>
To:     Song Liu <song@...nel.org>, Matthew Wilcox <willy@...radead.org>
Cc:     Andrew Morton <akpm@...ux-foundation.org>,
        Linux MM <linux-mm@...ck.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        William Kucharski <william.kucharski@...cle.com>,
        Hugh Dickins <hughd@...gle.com>
Subject: Re: [PATCH v2 1/2] mm, thp: check page mapping when truncating page
 cache



On 9/30/21 7:41 AM, Song Liu wrote:
> On Wed, Sep 29, 2021 at 10:56 AM Matthew Wilcox <willy@...radead.org> wrote:
>>
> [...]
>>> Now, I am able to crash the system on
>>>      find_lock_entries () {
>>>       ...
>>>         VM_BUG_ON_PAGE(page->index != xas.xa_index, page);
>>>      }
>>> I guess it is related. I will test more.
>>
>> That's a bogus VM_BUG_ON.  I have a patch in my tree to delete it.
>> Andrew has it too, but for some reason, he hasn't sent it on to Linus.
>>
>> +++ b/mm/filemap.c
>> @@ -2093,7 +2093,6 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
>>                  if (!xa_is_value(page)) {
>>                          if (page->index < start)
>>                                  goto put;
>> -                       VM_BUG_ON_PAGE(page->index != xas.xa_index, page);
>>                          if (page->index + thp_nr_pages(page) - 1 > end)
>>                                  goto put;
>>                          if (!trylock_page(page))
> 
> Yes, after removing this line, I am able to see the same bug.
> 
> Here is my finding so far:
> 
> The issue is NOT caused by concurrent khugepaged:collapse_file() and
> truncate_pagecache(inode, 0). With some printks, we can see a clear
> time gap (>2 second )  between collapse_file() finishes, and
> truncate_pagecache() (which crashes soon). Therefore, my earlier
> suggestion that adds deny_write_access() to collapse_file() does NOT
> work.
> 
> The crash is actually caused by concurrent truncate_pagecache(inode, 0).
> If I change the number of write thread in stress_madvise_dso.c to one,
> (IOW, one thread_read and one thread_write), I cannot reproduce the
> crash anymore.
Whether CONFIG_DEBUG_VM is enabled in your vm?

I think the second possibility mentioned above will been found if you 
enable CONFIG_DEBUG_VM:

1) multiple writers truncate the same page cache concurrently;
2) collapse_file rolls back when writer truncates the page cache;

The following log will be print after enable CONFIG_DEBUG_VM:

[22216.789904]  do_idle+0xb4/0x104
[22216.789906]  cpu_startup_entry+0x34/0x9c
[22216.790144] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 
0.0.0 02/06/2015
[22216.790553]  secondary_start_kernel+0x104/0x180
[22216.790778] Call trace:
[22216.791300] Code: d4210000 b0006161 910d4021 94013b45 (d4210000)
[22216.791662]  dump_backtrace+0x0/0x1ec
[22216.791664]  show_stack+0x24/0x30
[22216.791956] ---[ end trace dc769a61c1af087b ]---
[22216.792295]  dump_stack+0xd0/0x128
[22216.792299]  bad_page+0xe4/0x110
[22216.792579] Kernel panic - not syncing: Oops - BUG: Fatal exception 
in interrupt
[22216.792937]  check_free_page_bad+0x84/0x90
[22216.792940]  free_pcp_prepare+0x1fc/0x21c
[22216.793253] SMP: stopping secondary CPUs
[22216.793525]  free_unref_page+0x2c/0xec
[22216.805537]  __put_page+0x60/0x70
[22216.805931]  collapse_file+0xdc8/0x12f0
[22216.806385]  khugepaged_scan_file+0x2dc/0x37c
[22216.806900]  khugepaged_scan_mm_slot+0x2e0/0x380
[22216.807450]  khugepaged_do_scan+0x2dc/0x2fc
[22216.807946]  khugepaged+0x38/0x100
[22216.808342]  kthread+0x11c/0x120
[22216.808735] Kernel Offset: disabled
[22216.809153] CPU features: 0x0040002,62208238
[22216.809681] Memory Limit: none
[22216.813477] Starting crashdump kernel...

So I think the race also exists between collapse_file and 
truncate_pagecache.

> 
> I think this means we cannot fix this issue in collapse_file(), because it
> finishes long before the crash.
> 
> Thanks,
> Song
>