| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CA+wXwBR6S3StBwJJmo8Fu6KdPW5Q382N7FwnmfckBJo4e6ZD_A@mail.gmail.com>
Date: Mon, 24 Jul 2023 12:23:31 +0100
From: Daniel Dao <dqminh@...udflare.com>
To: linux-fsdevel@...r.kernel.org
Cc: Matthew Wilcox <willy@...radead.org>,
Dave Chinner <david@...morbit.com>,
kernel-team <kernel-team@...udflare.com>,
linux-kernel <linux-kernel@...r.kernel.org>, djwong@...nel.org
Subject: Re: Kernel NULL pointer deref and data corruptions with xfs on 6.1
Hi again,
We had another example of xarray corruption involving xfs and zsmalloc. We are
running zram as swap. We have 2 tasks deadlock waiting for page to be released
The following backtrace is from zsmalloc task
#0 context_switch (/cfsetup_build/build/linux/kernel/sched/core.c:5241:2)
#1 __schedule (/cfsetup_build/build/linux/kernel/sched/core.c:6554:8)
#2 schedule (/cfsetup_build/build/linux/kernel/sched/core.c:6630:3)
#3 io_schedule (/cfsetup_build/build/linux/kernel/sched/core.c:8774:2)
#4 folio_wait_bit_common (/cfsetup_build/build/linux/mm/filemap.c:1296:4)
#5 folio_wait_locked
(/cfsetup_build/build/linux/include/linux/pagemap.h:1022:3)
#6 wait_on_page_locked
(/cfsetup_build/build/linux/include/linux/pagemap.h:1034:2)
#7 lock_zspage (/cfsetup_build/build/linux/mm/zsmalloc.c:1736:3)
#8 async_free_zspage (/cfsetup_build/build/linux/mm/zsmalloc.c:1974:3)
#9 process_one_work (/cfsetup_build/build/linux/kernel/workqueue.c:2289:2)
#10 worker_thread (/cfsetup_build/build/linux/kernel/workqueue.c:2436:4)
#11 kthread (/cfsetup_build/build/linux/kernel/kthread.c:376:9)
#12 ret_from_fork+0x22/0x2d
(/cfsetup_build/build/linux/arch/x86/entry/entry_64.S:306)
The following backtrace is from a userspace task
#0 context_switch (/cfsetup_build/build/linux/kernel/sched/core.c:5241:2)
#1 __schedule (/cfsetup_build/build/linux/kernel/sched/core.c:6554:8)
#2 schedule (/cfsetup_build/build/linux/kernel/sched/core.c:6630:3)
#3 io_schedule (/cfsetup_build/build/linux/kernel/sched/core.c:8774:2)
#4 folio_wait_bit_common (/cfsetup_build/build/linux/mm/filemap.c:1296:4)
#5 folio_put_wait_locked (/cfsetup_build/build/linux/mm/filemap.c:1465:9)
#6 filemap_update_page (/cfsetup_build/build/linux/mm/filemap.c:2472:4)
#7 filemap_get_pages (/cfsetup_build/build/linux/mm/filemap.c:2606:9)
#8 filemap_read (/cfsetup_build/build/linux/mm/filemap.c:2676:11)
#9 xfs_file_buffered_read
(/cfsetup_build/build/linux/fs/xfs/xfs_file.c:277:8)
#10 xfs_file_read_iter (/cfsetup_build/build/linux/fs/xfs/xfs_file.c:302:9)
#11 call_read_iter (/cfsetup_build/build/linux/include/linux/fs.h:2199:9)
#12 new_sync_read (/cfsetup_build/build/linux/fs/read_write.c:389:8)
#13 vfs_read (/cfsetup_build/build/linux/fs/read_write.c:470:9)
#14 ksys_read (/cfsetup_build/build/linux/fs/read_write.c:613:9)
#15 do_syscall_x64 (/cfsetup_build/build/linux/arch/x86/entry/common.c:50:14)
#16 do_syscall_64 (/cfsetup_build/build/linux/arch/x86/entry/common.c:80:7)
#17 entry_SYSCALL_64+0x83/0x164
(/cfsetup_build/build/linux/arch/x86/entry/entry_64.S:120)
The folio in question has .mapping = (struct address_space
*)zsmalloc_mops+0x2 = 0xffffffffc1a9f332
and flag 'PG_locked|PG_waiters|PG_private|PG_slob_free'. In fact, the
file's i_pages
mapping has a node full of these pages. The following are entries we
get from mapping
in #6 at 0xffffffffa4e1c586 (filemap_get_pages+0x5d6/0x624) in
filemap_update_page at /cfsetup_build/build/linux/mm/filemap.c:2472:4
(inlined)
> for index, entry in xa_for_each(trace[6]['mapping'].i_pages.address_of_()):
print(index, entry, cast('struct folio *',
entry).page.mapping.address_of_())
2936 (void *)0xffffe53ab6454f00 *(struct address_space
**)0xffffe53ab6454f18 = 0xffff9ffc9ded16b0
2940 (void *)0xffffe53ab6454300 *(struct address_space
**)0xffffe53ab6454318 = 0xffff9ffc9ded16b0
2944 (void *)0xffffe53a02696000 *(struct address_space
**)0xffffe53a02696018 = zsmalloc_mops+0x2 = 0xffffffffc1a9f332 <==
index
2945 (void *)0xffffe53a02696000 *(struct address_space
**)0xffffe53a02696018 = zsmalloc_mops+0x2 = 0xffffffffc1a9f332
2946 (void *)0xffffe53a02696000 *(struct address_space
**)0xffffe53a02696018 = zsmalloc_mops+0x2 = 0xffffffffc1a9f332
...
2976 (void *)0xffffe53a02696000 *(struct address_space
**)0xffffe53a02696018 = zsmalloc_mops+0x2 = 0xffffffffc1a9f332 <==
last_index
...
3006 (void *)0xffffe53a02696000 *(struct address_space
**)0xffffe53a02696018 = zsmalloc_mops+0x2 = 0xffffffffc1a9f332
3007 (void *)0xffffe53ad71c37c0 *(struct address_space
**)0xffffe53ad71c37d8 = 0xffff9ffc9ded16b0
On Fri, Jul 21, 2023 at 11:49 AM Daniel Dao <dqminh@...udflare.com> wrote:
>
> Hi,
>
> In the past, we reported some corruptions on xfs/iomap/xarray combinations on
> kernel 6.1. This happened very rarely ( once a week for every 10000 hosts), and
> the host exhibited symptoms such as: rcu_preempt self-detected stalls,
> NULL pointer
> dereferences or deadlock when reading a particular file.
>
> We do not have a reproducer yet, but we now have more debugging data
> which hopefully
> should help narrow this down. Details as followed:
>
> 1. Kernel NULL pointer deferencences in __filemap_get_folio
>
> This happened on a few different hosts, with a few different repeated addresses.
> The addresses are 0000000000000036, 0000000000000076,
> 00000000000000f6. This looks
> like the xarray is corrupted and we were trying to do some work on a
> sibling entry.
>
> BUG: kernel NULL pointer dereference, address: 0000000000000036
> #PF: supervisor read access in kernel mode
> #PF: error_code(0x0000) - not-present page
> PGD 18806c5067 P4D 18806c5067 PUD 188ed48067 PMD 0
> Oops: 0000 [#1] PREEMPT SMP NOPTI
> CPU: 73 PID: 3579408 Comm: prometheus Tainted: G O
> 6.1.34-cloudflare-2023.6.7 #1
> Hardware name: GIGABYTE R162-Z12-CD1/MZ12-HD4-CD, BIOS M03 11/19/2021
> RIP: 0010:__filemap_get_folio (arch/x86/include/asm/atomic.h:29
> include/linux/atomic/atomic-arch-fallback.h:1242
> include/linux/atomic/atomic-arch-fallback.h:1267
> include/linux/atomic/atomic-instrumented.h:608
> include/linux/page_ref.h:238 include/linux/page_ref.h:247
> include/linux/page_ref.h:280 include/linux/page_ref.h:313
> mm/filemap.c:1863 mm/filemap.c:1915)
> Code: 10 e8 99 ac 84 00 48 3d 06 04 00 00 49 89 c4 74 e2 48 3d 02
> 04 00 00 74 da 48 85 c0 0f 84 2e 02 00 00 a8 01 0f 85 e3 00 00 00 <8b>
> 40 34 85 c0 74 c2 8d 50 01 4d 8d 7c 24 34 f0 41 0f b1 54 24 34
> All code
> ========
> 0: 10 e8 adc %ch,%al
> 2: 99 cltd
> 3: ac lods %ds:(%rsi),%al
> 4: 84 00 test %al,(%rax)
> 6: 48 3d 06 04 00 00 cmp $0x406,%rax
> c: 49 89 c4 mov %rax,%r12
> f: 74 e2 je 0xfffffffffffffff3
> 11: 48 3d 02 04 00 00 cmp $0x402,%rax
> 17: 74 da je 0xfffffffffffffff3
> 19: 48 85 c0 test %rax,%rax
> 1c: 0f 84 2e 02 00 00 je 0x250
> 22: a8 01 test $0x1,%al
> 24: 0f 85 e3 00 00 00 jne 0x10d
> 2a:* 8b 40 34 mov 0x34(%rax),%eax <-- trapping instruction
> 2d: 85 c0 test %eax,%eax
> 2f: 74 c2 je 0xfffffffffffffff3
> 31: 8d 50 01 lea 0x1(%rax),%edx
> 34: 4d 8d 7c 24 34 lea 0x34(%r12),%r15
> 39: f0 41 0f b1 54 24 34 lock cmpxchg %edx,0x34(%r12)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 8b 40 34 mov 0x34(%rax),%eax
> 3: 85 c0 test %eax,%eax
> 5: 74 c2 je 0xffffffffffffffc9
> 7: 8d 50 01 lea 0x1(%rax),%edx
> a: 4d 8d 7c 24 34 lea 0x34(%r12),%r15
> f: f0 41 0f b1 54 24 34 lock cmpxchg %edx,0x34(%r12)
> RSP: 0000:ffffaf5587cdfc60 EFLAGS: 00010246
> RAX: 0000000000000002 RBX: 0000000000000000 RCX: 0000000000000002
> RDX: 0000000000000008 RSI: ffffa45181fa8000 RDI: ffffaf5587cdfc70
> RBP: 0000000000000000 R08: 0000000000000402 R09: 000000000006e44f
> R10: 000000000006e450 R11: 000000000006e448 R12: 0000000000000002
> R13: ffffa3fff6fdfeb0 R14: 000000000006e44a R15: 00000000000000d1
> FS: 000000c9e385ac90(0000) GS:ffffa4153fc40000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000036 CR3: 000000296a1bc002 CR4: 0000000000770ee0
> PKRU: 55555554
> Call Trace:
> <TASK>
> ? __die_body.cold (arch/x86/kernel/dumpstack.c:478
> arch/x86/kernel/dumpstack.c:465 arch/x86/kernel/dumpstack.c:420)
> ? page_fault_oops (arch/x86/mm/fault.c:727)
> ? migrate_task_rq_fair (include/linux/sched.h:1921
> kernel/sched/fair.c:3932 kernel/sched/fair.c:7497)
> ? do_user_addr_fault (include/linux/kprobes.h:404
> include/linux/kprobes.h:597 arch/x86/mm/fault.c:1280)
> ? ttwu_queue_wakelist (kernel/sched/core.c:3880)
> ? exc_page_fault (arch/x86/include/asm/irqflags.h:40
> arch/x86/include/asm/irqflags.h:75 arch/x86/mm/fault.c:1527
> arch/x86/mm/fault.c:1575)
> ? asm_exc_page_fault (arch/x86/include/asm/idtentry.h:570)
> ? __filemap_get_folio (arch/x86/include/asm/atomic.h:29
> include/linux/atomic/atomic-arch-fallback.h:1242
> include/linux/atomic/atomic-arch-fallback.h:1267
> include/linux/atomic/atomic-instrumented.h:608
> include/linux/page_ref.h:238 include/linux/page_ref.h:247
> include/linux/page_ref.h:280 include/linux/page_ref.h:313
> mm/filemap.c:1863 mm/filemap.c:1915)
> filemap_fault (mm/filemap.c:3120)
> ? preempt_count_add (include/linux/ftrace.h:950
> kernel/sched/core.c:5685 kernel/sched/core.c:5682
> kernel/sched/core.c:5710)
> __do_fault (mm/memory.c:4234)
> do_fault (mm/memory.c:4564 mm/memory.c:4692)
> __handle_mm_fault (mm/memory.c:4964 mm/memory.c:5106)
> handle_mm_fault (mm/memory.c:5227)
> do_user_addr_fault (include/linux/sched/signal.h:433
> arch/x86/mm/fault.c:1430)
> exc_page_fault (arch/x86/include/asm/irqflags.h:40
> arch/x86/include/asm/irqflags.h:75 arch/x86/mm/fault.c:1527
> arch/x86/mm/fault.c:1575)
> asm_exc_page_fault (arch/x86/include/asm/idtentry.h:570)
> RIP: 0033:0x268b8b9
> Code: 70 48 89 4c 24 78 48 8b 94 24 b8 00 00 00 0f 1f 00 48 85 d2
> 74 3f 48 89 ce 48 29 d9 4c 8d 49 04 49 f7 d9 49 c1 f9 3f 49 21 f9 <46>
> 8b 0c 08 44 89 4c 24 34 90 90 48 89 d3 48 89 c1 41 b8 01 00 00
> All code
> ========
> 0: 70 48 jo 0x4a
> 2: 89 4c 24 78 mov %ecx,0x78(%rsp)
> 6: 48 8b 94 24 b8 00 00 mov 0xb8(%rsp),%rdx
> d: 00
> e: 0f 1f 00 nopl (%rax)
> 11: 48 85 d2 test %rdx,%rdx
> 14: 74 3f je 0x55
> 16: 48 89 ce mov %rcx,%rsi
> 19: 48 29 d9 sub %rbx,%rcx
> 1c: 4c 8d 49 04 lea 0x4(%rcx),%r9
> 20: 49 f7 d9 neg %r9
> 23: 49 c1 f9 3f sar $0x3f,%r9
> 27: 49 21 f9 and %rdi,%r9
> 2a:* 46 8b 0c 08 mov (%rax,%r9,1),%r9d <-- trapping
> instruction
> 2e: 44 89 4c 24 34 mov %r9d,0x34(%rsp)
> 33: 90 nop
> 34: 90 nop
> 35: 48 89 d3 mov %rdx,%rbx
> 38: 48 89 c1 mov %rax,%rcx
> 3b: 41 rex.B
> 3c: b8 .byte 0xb8
> 3d: 01 00 add %eax,(%rax)
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 46 8b 0c 08 mov (%rax,%r9,1),%r9d
> 4: 44 89 4c 24 34 mov %r9d,0x34(%rsp)
> 9: 90 nop
> a: 90 nop
> b: 48 89 d3 mov %rdx,%rbx
> e: 48 89 c1 mov %rax,%rcx
> 11: 41 rex.B
> 12: b8 .byte 0xb8
> 13: 01 00 add %eax,(%rax)
> ...
> RSP: 002b:000000cbc509f520 EFLAGS: 00010202
> RAX: 00007e81cf427e0c RBX: 00000000000222cc RCX: 00000000123817b2
> RDX: 000000c00001ac00 RSI: 00000000123a3a7e RDI: 00000000000222c8
> RBP: 000000cbc509f5b0 R08: 0000000003cb5910 R09: 00000000000222c8
> R10: 000000c4de3dea00 R11: 0000000000000123 R12: 0000000000000000
> R13: 0000000000000005 R14: 000000c83bad2340 R15: 0000010000000000
> </TASK>
> Modules linked in: xt_connlabel xt_MASQUERADE nf_conntrack_netlink
> xfrm_user xfrm_algo xt_addrtype br_netfilter bridge overlay zstd
> zstd_compress zram zsmalloc tun tcp_diag inet_diag raid0 md_mod essiv
> dm_crypt trusted asn1_encoder tee ip6table_filter ip6table_mangle
> ip6table_raw ip6table_security ip6table_nat ip6_tables xt_bpf
> xt_conntrack xt_multiport xt_set iptable_filter xt_NFLOG nfnetlink_log
> xt_connbytes xt_comment xt_connmark xt_statistic iptable_mangle xt_nat
> xt_tcpudp iptable_nat nf_nat xt_CT iptable_raw ip_set_hash_ip
> ip_set_hash_net ip_set nfnetlink sch_fq nf_conntrack nf_defrag_ipv6
> nf_defrag_ipv4 8021q garp mrp stp llc bonding nvme_fabrics amd64_edac
> kvm_amd ipmi_ssif kvm irqbypass crc32_pclmul crc32c_intel sha512_ssse3
> acpi_ipmi mlx5_core aesni_intel ipmi_si mlxfw rapl xhci_pci nvme tls
> ipmi_devintf tiny_power_button psample nvme_core xhci_hcd i2c_piix4
> ccp ipmi_msghandler button fuse dm_mod dax efivarfs ip_tables x_tables
> bcmcrypt(O)
> crypto_simd cryptd
> CR2: 0000000000000036
> ---[ end trace 0000000000000000 ]---
> RIP: 0010:__filemap_get_folio (arch/x86/include/asm/atomic.h:29
> include/linux/atomic/atomic-arch-fallback.h:1242
> include/linux/atomic/atomic-arch-fallback.h:1267
> include/linux/atomic/atomic-instrumented.h:608
> include/linux/page_ref.h:238 include/linux/page_ref.h:247
> include/linux/page_ref.h:280 include/linux/page_ref.h:313
> mm/filemap.c:1863 mm/filemap.c:1915)
> Code: 10 e8 99 ac 84 00 48 3d 06 04 00 00 49 89 c4 74 e2 48 3d 02
> 04 00 00 74 da 48 85 c0 0f 84 2e 02 00 00 a8 01 0f 85 e3 00 00 00 <8b>
> 40 34 85 c0 74 c2 8d 50 01 4d 8d 7c 24 34 f0 41 0f b1 54 24 34
> All code
> ========
> 0: 10 e8 adc %ch,%al
> 2: 99 cltd
> 3: ac lods %ds:(%rsi),%al
> 4: 84 00 test %al,(%rax)
> 6: 48 3d 06 04 00 00 cmp $0x406,%rax
> c: 49 89 c4 mov %rax,%r12
> f: 74 e2 je 0xfffffffffffffff3
> 11: 48 3d 02 04 00 00 cmp $0x402,%rax
> 17: 74 da je 0xfffffffffffffff3
> 19: 48 85 c0 test %rax,%rax
> 1c: 0f 84 2e 02 00 00 je 0x250
> 22: a8 01 test $0x1,%al
> 24: 0f 85 e3 00 00 00 jne 0x10d
> 2a:* 8b 40 34 mov 0x34(%rax),%eax <-- trapping instruction
> 2d: 85 c0 test %eax,%eax
> 2f: 74 c2 je 0xfffffffffffffff3
> 31: 8d 50 01 lea 0x1(%rax),%edx
> 34: 4d 8d 7c 24 34 lea 0x34(%r12),%r15
> 39: f0 41 0f b1 54 24 34 lock cmpxchg %edx,0x34(%r12)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 8b 40 34 mov 0x34(%rax),%eax
> 3: 85 c0 test %eax,%eax
> 5: 74 c2 je 0xffffffffffffffc9
> 7: 8d 50 01 lea 0x1(%rax),%edx
> a: 4d 8d 7c 24 34 lea 0x34(%r12),%r15
> f: f0 41 0f b1 54 24 34 lock cmpxchg %edx,0x34(%r12)
> RSP: 0000:ffffaf5587cdfc60 EFLAGS: 00010246
> RAX: 0000000000000002 RBX: 0000000000000000 RCX: 0000000000000002
> RDX: 0000000000000008 RSI: ffffa45181fa8000 RDI: ffffaf5587cdfc70
> RBP: 0000000000000000 R08: 0000000000000402 R09: 000000000006e44f
> R10: 000000000006e450 R11: 000000000006e448 R12: 0000000000000002
> R13: ffffa3fff6fdfeb0 R14: 000000000006e44a R15: 00000000000000d1
> FS: 000000c9e385ac90(0000) GS:ffffa4153fc40000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000036 CR3: 000000296a1bc002 CR4: 0000000000770ee0
> PKRU: 55555554
>
> BUG: kernel NULL pointer dereference, address: 0000000000000076
> #PF: supervisor read access in kernel mode
> #PF: error_code(0x0000) - not-present page
> PGD 7acd78067 P4D 7acd78067 PUD 7acd79067 PMD 0
> Oops: 0000 [#1] PREEMPT SMP NOPTI
> CPU: 93 PID: 3784417 Comm: prometheus Tainted: G O
> 6.1.20-cloudflare-2023.3.18 #1
> Hardware name: GIGABYTE R162-Z13-CD/MZ12-HD2-CD, BIOS R13 07/17/2020
> RIP: 0010:__filemap_get_folio (arch/x86/include/asm/atomic.h:29
> include/linux/atomic/atomic-arch-fallback.h:1242
> include/linux/atomic/atomic-arch-fallback.h:1267
> include/linux/atomic/atomic-instrumented.h:608
> include/linux/page_ref.h:238 include/linux/page_ref.h:247
> include/linux/page_ref.h:280 include/linux/page_ref.h:313
> mm/filemap.c:1863 mm/filemap.c:1915)
> Code: 10 e8 b9 a4 84 00 48 3d 06 04 00 00 49 89 c4 74 e2 48 3d 02
> 04 00 00 74 da 48 85 c0 0f 84 2e 02 00 00 a8 01 0f 85 e3 00 00 00 <8b>
> 40 34 85 c0 74 c2 8d 50 01 4d 8d 7c 24 34 f0 41 0f b1 54 24 34
> All code
> ========
> 0: 10 e8 adc %ch,%al
> 2: b9 a4 84 00 48 mov $0x480084a4,%ecx
> 7: 3d 06 04 00 00 cmp $0x406,%eax
> c: 49 89 c4 mov %rax,%r12
> f: 74 e2 je 0xfffffffffffffff3
> 11: 48 3d 02 04 00 00 cmp $0x402,%rax
> 17: 74 da je 0xfffffffffffffff3
> 19: 48 85 c0 test %rax,%rax
> 1c: 0f 84 2e 02 00 00 je 0x250
> 22: a8 01 test $0x1,%al
> 24: 0f 85 e3 00 00 00 jne 0x10d
> 2a:* 8b 40 34 mov 0x34(%rax),%eax <-- trapping instruction
> 2d: 85 c0 test %eax,%eax
> 2f: 74 c2 je 0xfffffffffffffff3
> 31: 8d 50 01 lea 0x1(%rax),%edx
> 34: 4d 8d 7c 24 34 lea 0x34(%r12),%r15
> 39: f0 41 0f b1 54 24 34 lock cmpxchg %edx,0x34(%r12)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 8b 40 34 mov 0x34(%rax),%eax
> 3: 85 c0 test %eax,%eax
> 5: 74 c2 je 0xffffffffffffffc9
> 7: 8d 50 01 lea 0x1(%rax),%edx
> a: 4d 8d 7c 24 34 lea 0x34(%r12),%r15
> f: f0 41 0f b1 54 24 34 lock cmpxchg %edx,0x34(%r12)
> RSP: 0000:ffffb15106683c60 EFLAGS: 00010246
> RAX: 0000000000000042 RBX: 0000000000000000 RCX: 0000000000000002
> RDX: 0000000000000018 RSI: ffff934b0029efc8 RDI: ffffb15106683c70
> RBP: 0000000000000000 R08: 0000000000000402 R09: 00000000000cbe5f
> R10: 00000000000cbe60 R11: 00000000000cbe5c R12: 0000000000000042
> R13: ffff93449c251eb0 R14: 00000000000cbe59 R15: 00000000000000d1
> FS: 000000c000300090(0000) GS:ffff937e6ed40000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000076 CR3: 0000000a6528e000 CR4: 0000000000350ee0
> Call Trace:
> <TASK>
> filemap_fault (mm/filemap.c:3120)
> ? preempt_count_add (include/linux/ftrace.h:950
> kernel/sched/core.c:5685 kernel/sched/core.c:5682
> kernel/sched/core.c:5710)
> __do_fault (mm/memory.c:4234)
> do_fault (mm/memory.c:4564 mm/memory.c:4692)
> __handle_mm_fault (mm/memory.c:4964 mm/memory.c:5106)
> handle_mm_fault (mm/memory.c:5227)
> do_user_addr_fault (include/linux/sched/signal.h:433
> arch/x86/mm/fault.c:1430)
> exc_page_fault (arch/x86/include/asm/irqflags.h:40
> arch/x86/include/asm/irqflags.h:75 arch/x86/mm/fault.c:1527
> arch/x86/mm/fault.c:1575)
> asm_exc_page_fault (arch/x86/include/asm/idtentry.h:570)
> RIP: 0033:0x268b8b9
> Code: 70 48 89 4c 24 78 48 8b 94 24 b8 00 00 00 0f 1f 00 48 85 d2
> 74 3f 48 89 ce 48 29 d9 4c 8d 49 04 49 f7 d9 49 c1 f9 3f 49 21 f9 <46>
> 8b 0c 08 44 89 4c 24 34 90 90 48 89 d3 48 89 c1 41 b8 01 00 00
> All code
> ========
> 0: 70 48 jo 0x4a
> 2: 89 4c 24 78 mov %ecx,0x78(%rsp)
> 6: 48 8b 94 24 b8 00 00 mov 0xb8(%rsp),%rdx
> d: 00
> e: 0f 1f 00 nopl (%rax)
> 11: 48 85 d2 test %rdx,%rdx
> 14: 74 3f je 0x55
> 16: 48 89 ce mov %rcx,%rsi
> 19: 48 29 d9 sub %rbx,%rcx
> 1c: 4c 8d 49 04 lea 0x4(%rcx),%r9
> 20: 49 f7 d9 neg %r9
> 23: 49 c1 f9 3f sar $0x3f,%r9
> 27: 49 21 f9 and %rdi,%r9
> 2a:* 46 8b 0c 08 mov (%rax,%r9,1),%r9d <-- trapping
> instruction
> 2e: 44 89 4c 24 34 mov %r9d,0x34(%rsp)
> 33: 90 nop
> 34: 90 nop
> 35: 48 89 d3 mov %rdx,%rbx
> 38: 48 89 c1 mov %rax,%rcx
> 3b: 41 rex.B
> 3c: b8 .byte 0xb8
> 3d: 01 00 add %eax,(%rax)
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 46 8b 0c 08 mov (%rax,%r9,1),%r9d
> 4: 44 89 4c 24 34 mov %r9d,0x34(%rsp)
> 9: 90 nop
> a: 90 nop
> b: 48 89 d3 mov %rdx,%rbx
> e: 48 89 c1 mov %rax,%rcx
> 11: 41 rex.B
> 12: b8 .byte 0xb8
> 13: 01 00 add %eax,(%rax)
> ...
> RSP: 002b:000000d735bb3558 EFLAGS: 00010206
> RAX: 00007c018402dad8 RBX: 000000000002c3d8 RCX: 0000000037f9be1c
> RDX: 000000c000222c00 RSI: 0000000037fc81f4 RDI: 000000000002c3d4
> RBP: 000000d735bb35e8 R08: 0000000003cb5910 R09: 000000000002c3d4
> R10: 000000c385d2a000 R11: 0000000000000021 R12: 0000000000000000
> R13: 000000000000000b R14: 000000d1bb70e340 R15: 0000000001000000
> </TASK>
> Modules linked in: veth xt_MASQUERADE nf_conntrack_netlink
> xfrm_user xfrm_algo xt_addrtype br_netfilter bridge overlay raid1
> md_mod essiv dm_crypt trusted tee asn1_encoder xt_hl ip6table_filter
> ip6table_mangle ip6table_raw ip6table_security ip6table_nat ip6_tables
> xt_tcpudp xt_conntrack xt_comment xt_multiport xt_set iptable_filter
> iptable_mangle iptable_nat nf_nat xt_CT iptable_raw ip_set_hash_ip
> ip_set_hash_net ip_set nfnetlink tcp_bbr sch_fq nf_conntrack
> nf_defrag_ipv6 nf_defrag_ipv4 8021q mrp garp stp llc bonding
> amd64_edac kvm_amd ipmi_ssif kvm irqbypass crc32_pclmul crc32c_intel
> mlx5_core sha512_ssse3 psample acpi_ipmi aesni_intel xhci_pci nvme
> ipmi_si rapl tls ipmi_devintf tiny_power_button nvme_core mlxfw
> xhci_hcd i2c_piix4 ccp ipmi_msghandler button fuse dm_mod dax efivarfs
> ip_tables x_tables bcmcrypt(O) crypto_simd cryptd
> CR2: 0000000000000076
> ---[ end trace 0000000000000000 ]---
> RIP: 0010:__filemap_get_folio (arch/x86/include/asm/atomic.h:29
> include/linux/atomic/atomic-arch-fallback.h:1242
> include/linux/atomic/atomic-arch-fallback.h:1267
> include/linux/atomic/atomic-instrumented.h:608
> include/linux/page_ref.h:238 include/linux/page_ref.h:247
> include/linux/page_ref.h:280 include/linux/page_ref.h:313
> mm/filemap.c:1863 mm/filemap.c:1915)
> Code: 10 e8 b9 a4 84 00 48 3d 06 04 00 00 49 89 c4 74 e2 48 3d 02
> 04 00 00 74 da 48 85 c0 0f 84 2e 02 00 00 a8 01 0f 85 e3 00 00 00 <8b>
> 40 34 85 c0 74 c2 8d 50 01 4d 8d 7c 24 34 f0 41 0f b1 54 24 34
> All code
> ========
> 0: 10 e8 adc %ch,%al
> 2: b9 a4 84 00 48 mov $0x480084a4,%ecx
> 7: 3d 06 04 00 00 cmp $0x406,%eax
> c: 49 89 c4 mov %rax,%r12
> f: 74 e2 je 0xfffffffffffffff3
> 11: 48 3d 02 04 00 00 cmp $0x402,%rax
> 17: 74 da je 0xfffffffffffffff3
> 19: 48 85 c0 test %rax,%rax
> 1c: 0f 84 2e 02 00 00 je 0x250
> 22: a8 01 test $0x1,%al
> 24: 0f 85 e3 00 00 00 jne 0x10d
> 2a:* 8b 40 34 mov 0x34(%rax),%eax <-- trapping instruction
> 2d: 85 c0 test %eax,%eax
> 2f: 74 c2 je 0xfffffffffffffff3
> 31: 8d 50 01 lea 0x1(%rax),%edx
> 34: 4d 8d 7c 24 34 lea 0x34(%r12),%r15
> 39: f0 41 0f b1 54 24 34 lock cmpxchg %edx,0x34(%r12)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 8b 40 34 mov 0x34(%rax),%eax
> 3: 85 c0 test %eax,%eax
> 5: 74 c2 je 0xffffffffffffffc9
> 7: 8d 50 01 lea 0x1(%rax),%edx
> a: 4d 8d 7c 24 34 lea 0x34(%r12),%r15
> f: f0 41 0f b1 54 24 34 lock cmpxchg %edx,0x34(%r12)
> RSP: 0000:ffffb15106683c60 EFLAGS: 00010246
> RAX: 0000000000000042 RBX: 0000000000000000 RCX: 0000000000000002
> RDX: 0000000000000018 RSI: ffff934b0029efc8 RDI: ffffb15106683c70
> RBP: 0000000000000000 R08: 0000000000000402 R09: 00000000000cbe5f
> R10: 00000000000cbe60 R11: 00000000000cbe5c R12: 0000000000000042
> R13: ffff93449c251eb0 R14: 00000000000cbe59 R15: 00000000000000d1
> FS: 000000c000300090(0000) GS:ffff937e6ed40000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000076 CR3: 0000000a6528e000 CR4: 0000000000350ee0
> note: prometheus[3784417] exited with irqs disabled
>
> 2. Kernel NULL pointer deferencences in xfs_read_iomap_begin
>
> BUG: unable to handle page fault for address: 0000000000034668
> #PF: supervisor read access in kernel mode
> #PF: error_code(0x0000) - not-present page
> PGD 11cfd37067 P4D 11cfd37067 PUD b88086067 PMD 0
> Oops: 0000 [#1] PREEMPT SMP NOPTI
> CPU: 124 PID: 3831226 Comm: rocksdb:low Kdump: loaded Tainted: G
> W O L 6.1.27-cloudflare-2023.5.0 #1
> Hardware name: HYVE EDGE-METAL-GEN11/HS1811D_Lite, BIOS V0.11-sig 12/23/2022
> RIP: 0010:xfs_read_iomap_begin (fs/xfs/xfs_iomap.c:1200)
> Code: 0f 1f 44 00 00 41 57 41 56 41 55 41 54 55 53 48 83 ec 50 48
> 89 14 24 4c 89 44 24 08 65 48 8b 04 25 28 00 00 00 48 89 44 24 48 <48>
> 8b 87 >
> All code
> ========
> 0: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
> 5: 41 57 push %r15
> 7: 41 56 push %r14
> 9: 41 55 push %r13
> b: 41 54 push %r12
> d: 55 push %rbp
> e: 53 push %rbx
> f: 48 83 ec 50 sub $0x50,%rsp
> 13: 48 89 14 24 mov %rdx,(%rsp)
> 17: 4c 89 44 24 08 mov %r8,0x8(%rsp)
> 1c: 65 48 8b 04 25 28 00 mov %gs:0x28,%rax
> 23: 00 00
> 25: 48 89 44 24 48 mov %rax,0x48(%rsp)
> 2a:* 48 rex.W <-- trapping instruction
> 2b: 8b .byte 0x8b
> 2c: 87 00 xchg %eax,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 48 rex.W
> 1: 8b .byte 0x8b
> 2: 87 00 xchg %eax,(%rax)
> RSP: 0018:ffffa63810733a70 EFLAGS: 00010282
> RAX: 78ac714f0997e100 RBX: ffffa63810733b40 RCX: 0000000000000000
> RDX: 0000000000004000 RSI: 0000000000000000 RDI: 00000000000347a0
> RBP: ffffffff8664d950 R08: ffffa63810733b68 R09: ffffa63810733bb0
> R10: 000000000001f627 R11: 0000000000000000 R12: ffffa63810733b68
> R13: ffffa63810733bb0 R14: 00000000000019c1 R15: 00000000fffffff5
> FS: 00007f48d8504700(0000) GS:ffffa2fe5ef00000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000034668 CR3: 00000013037ec001 CR4: 0000000000770ee0
> PKRU: 55555554
> Call Trace:
> <TASK>
> ? __mod_memcg_lruvec_state (mm/memcontrol.c:613 mm/memcontrol.c:799)
> iomap_iter (fs/iomap/iter.c:76)
> iomap_read_folio (fs/iomap/buffered-io.c:342)
> ? xfs_end_bio (fs/xfs/xfs_aops.c:542)
> filemap_read_folio (mm/filemap.c:2407)
> filemap_get_pages (mm/filemap.c:2492 mm/filemap.c:2606)
> filemap_read (mm/filemap.c:2677)
> xfs_file_buffered_read (fs/xfs/xfs_file.c:278)
> xfs_file_read_iter (fs/xfs/xfs_file.c:304)
> vfs_read (fs/read_write.c:390 fs/read_write.c:470)
> __x64_sys_pread64 (include/linux/file.h:44 fs/read_write.c:666
> fs/read_write.c:675 fs/read_write.c:672 fs/read_write.c:672)
> do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80)
> entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
> RIP: 0033:0x7f49061ca917
> Code: 08 89 3c 24 48 89 4c 24 18 e8 05 f4 ff ff 4c 8b 54 24 18 48
> 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 11 00 00 00 0f 05 <48>
> 3d 00 >
> All code
> ========
> 0: 08 89 3c 24 48 89 or %cl,-0x76b7dbc4(%rcx)
> 6: 4c 24 18 rex.WR and $0x18,%al
> 9: e8 05 f4 ff ff call 0xfffffffffffff413
> e: 4c 8b 54 24 18 mov 0x18(%rsp),%r10
> 13: 48 8b 54 24 10 mov 0x10(%rsp),%rdx
> 18: 41 89 c0 mov %eax,%r8d
> 1b: 48 8b 74 24 08 mov 0x8(%rsp),%rsi
> 20: 8b 3c 24 mov (%rsp),%edi
> 23: b8 11 00 00 00 mov $0x11,%eax
> 28: 0f 05 syscall
> 2a:* 48 rex.W <-- trapping instruction
> 2b: 3d .byte 0x3d
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 48 rex.W
> 1: 3d .byte 0x3d
> ...
> RSP: 002b:00007f48d84ffc70 EFLAGS: 00000293 ORIG_RAX: 0000000000000011
> RAX: ffffffffffffffda RBX: 00000000018a0c90 RCX: 00007f49061ca917
> RDX: 00000000000c294f RSI: 000000002265e000 RDI: 000000000000003c
> RBP: 00007f48d84ffda0 R08: 0000000000000000 R09: 00007f48d84ffe60
> R10: 000000000191dcd8 R11: 0000000000000293 R12: 0000000007c3c6c0
> R13: 00000000000c294f R14: 00000000000c294f R15: 000000000191dcd8
> </TASK>
> Modules linked in: xt_connlabel overlay nft_compat esp4
> xt_hashlimit ip_set_hash_netport xt_length nf_conntrack_netlink
> mpls_gso mpls_iptunnel >
> tcp_bbr sch_fq nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 8021q
> garp mrp stp llc ipmi_ssif amd64_edac kvm_amd kvm irqbypass
> crc32_pclmul crc32>
> CR2: 0000000000034668
> ---[ end trace 0000000000000000 ]---
>
> We also have a deadlock reading a very specific file on this host. We managed to
> do a kdump on this host and extracted out the state of the mapping.
>
>
> >>> trace
> #0 context_switch (/cfsetup_build/build/linux/kernel/sched/core.c:5241:2)
> #1 __schedule (/cfsetup_build/build/linux/kernel/sched/core.c:6554:8)
> #2 schedule (/cfsetup_build/build/linux/kernel/sched/core.c:6630:3)
> #3 io_schedule (/cfsetup_build/build/linux/kernel/sched/core.c:8774:2)
> #4 folio_wait_bit_common (/cfsetup_build/build/linux/mm/filemap.c:1296:4)
> #5 folio_put_wait_locked (/cfsetup_build/build/linux/mm/filemap.c:1465:9)
> #6 filemap_update_page (/cfsetup_build/build/linux/mm/filemap.c:2472:4)
> #7 filemap_get_pages (/cfsetup_build/build/linux/mm/filemap.c:2606:9)
> #8 filemap_read (/cfsetup_build/build/linux/mm/filemap.c:2676:11)
> #9 xfs_file_buffered_read
> (/cfsetup_build/build/linux/fs/xfs/xfs_file.c:277:8)
> #10 xfs_file_read_iter (/cfsetup_build/build/linux/fs/xfs/xfs_file.c:302:9)
> #11 call_read_iter (/cfsetup_build/build/linux/include/linux/fs.h:2199:9)
> #12 new_sync_read (/cfsetup_build/build/linux/fs/read_write.c:389:8)
> #13 vfs_read (/cfsetup_build/build/linux/fs/read_write.c:470:9)
> #14 ksys_read (/cfsetup_build/build/linux/fs/read_write.c:613:9)
> #15 do_syscall_x64
> (/cfsetup_build/build/linux/arch/x86/entry/common.c:50:14)
> #16 do_syscall_64 (/cfsetup_build/build/linux/arch/x86/entry/common.c:80:7)
> #17 entry_SYSCALL_64+0x83/0x164
> (/cfsetup_build/build/linux/arch/x86/entry/entry_64.S:120)
> #18 0x7f05f0b093ce
> >>> folio = trace[6]['folio']
> >>> decode_page_flags(folio)
> 'PG_locked|PG_waiters|PG_head'
> >>> folio
> *(struct folio *)0xffffd67406346000 = {
> .flags = (unsigned long)13510764522438785,
> .lru = (struct list_head){
> .next = (struct list_head *)0xdead000000000100,
> .prev = (struct list_head *)0xdead000000000122,
> },
> .__filler = (void *)0xdead000000000100,
> .mlock_count = (unsigned int)290,
> .mapping = (struct address_space *)0x0,
> .index = (unsigned long)18446641474676726016,
> .private = (void *)0x400000,
> ._mapcount = (atomic_t){
> .counter = (int)-1,
> },
> ._refcount = (atomic_t){
> .counter = (int)1,
> },
> .memcg_data = (unsigned long)0,
> .page = (struct page){
> .flags = (unsigned long)13510764522438785,
> .lru = (struct list_head){
> .next = (struct list_head *)0xdead000000000100,
> .prev = (struct list_head *)0xdead000000000122,
> },
> .__filler = (void *)0xdead000000000100,
> .mlock_count = (unsigned int)290,
> .buddy_list = (struct list_head){
> .next = (struct list_head *)0xdead000000000100,
> .prev = (struct list_head *)0xdead000000000122,
> },
> .pcp_list = (struct list_head){
> .next = (struct list_head *)0xdead000000000100,
> .prev = (struct list_head *)0xdead000000000122,
> },
> .mapping = (struct address_space *)0x0,
> .index = (unsigned long)18446641474676726016,
> .private = (unsigned long)4194304,
> .pp_magic = (unsigned long)16045481047390945536,
> .pp = (struct page_pool *)0xdead000000000122,
> ._pp_mapping_pad = (unsigned long)0,
> .dma_addr = (unsigned long)18446641474676726016,
> .dma_addr_upper = (unsigned long)4194304,
> .pp_frag_count = (atomic_long_t){
> .counter = (s64)4194304,
> },
> .compound_head = (unsigned long)16045481047390945536,
> .compound_dtor = (unsigned char)34,
> .compound_order = (unsigned char)1,
> .compound_mapcount = (atomic_t){
> .counter = (int)-559087616,
> },
> .compound_pincount = (atomic_t){
> .counter = (int)0,
> },
> .compound_nr = (unsigned int)0,
> ._compound_pad_1 = (unsigned long)16045481047390945536,
> ._compound_pad_2 = (unsigned long)16045481047390945570,
> .deferred_list = (struct list_head){
> .next = (struct list_head *)0x0,
> .prev = (struct list_head *)0xffffa2afcd181900,
> },
> ._pt_pad_1 = (unsigned long)16045481047390945536,
> .pmd_huge_pte = (pgtable_t)0xdead000000000122,
> ._pt_pad_2 = (unsigned long)0,
> .pt_mm = (struct mm_struct *)0xffffa2afcd181900,
> .pt_frag_refcount = (atomic_t){
> .counter = (int)-854058752,
> },
> .ptl = (spinlock_t){
> .rlock = (struct raw_spinlock){
> .raw_lock = (arch_spinlock_t){
> .val = (atomic_t){
> .counter = (int)4194304,
> },
> .locked = (u8)0,
> .pending = (u8)0,
> .locked_pending = (u16)0,
> .tail = (u16)64,
> },
> },
> },
> .pgmap = (struct dev_pagemap *)0xdead000000000100,
> .zone_device_data = (void *)0xdead000000000122,
> .callback_head = (struct callback_head){
> .next = (struct callback_head *)0xdead000000000100,
> .func = (void (*)(struct callback_head
> *))0xdead000000000122,
> },
> ._mapcount = (atomic_t){
> .counter = (int)-1,
> },
> .page_type = (unsigned int)4294967295,
> ._refcount = (atomic_t){
> .counter = (int)1,
> },
> .memcg_data = (unsigned long)0,
> },
> ._flags_1 = (unsigned long)13510764522373120,
> .__head = (unsigned long)18446698392541487105,
> ._folio_dtor = (unsigned char)1,
> ._folio_order = (unsigned char)2,
> ._total_mapcount = (atomic_t){
> .counter = (int)-1,
> },
> ._pincount = (atomic_t){
> .counter = (int)0,
> },
> ._folio_nr_pages = (unsigned int)4,
> }
> >>> for index, entry in
> xa_for_each(trace[7]['mapping'].i_pages.address_of_()):
> print(index, entry, cast('struct folio *',
> entry).page.mapping.address_of_())
> ....
> 6464 (void *)0xffffd674c130a000 *(struct address_space
> **)0xffffd674c130a018 = 0xffffa2b30e93b2b0
> 6528 (void *)0xffffd674beb22000 *(struct address_space
> **)0xffffd674beb22018 = 0xffffa2b30e93b2b0
> 6592 (void *)0xffffd67406346000 *(struct address_space
> **)0xffffd67406346018 = 0x0 <===== our folio
> 6624 (void *)0x7037e8d8000100d (struct address_space **)0x7037e8d80001025
> 6625 (void *)0x7037e047000100d (struct address_space **)0x7037e0470001025
> ....
>
> This looks like the xarray is corrupted, and for some reason we have a
> locked folio
> in the mapping with a page with no mapping.
>
> Any suggestions on narrowing this down to a hypothesis to try to reproduce this,
> or potential fixes are very much appreciated. We are also trying some
> different kernels
> configurations on different set of hosts to see if the problems go
> away for them, such as:
> - 6.1.36 without xfs: Support large folios
> 6795801366da0cd3d99e27c37f020a8f16714886
> - 6.1.36 without THP
> - 6.1.37 with the following series backported xfs, iomap: fix data
> corruption due to stale cached iomaps
> https://lore.kernel.org/linux-fsdevel/20221129001632.GX3600936@dread.disaster.area/
>
> Best,
> Daniel.
Powered by blists - more mailing lists