Message-ID: <2538f4c21dd2405aac2abb776047a0fa@hisilicon.com>
Date:   Thu, 11 Feb 2021 10:43:18 +0000
From:   "Song Bao Hua (Barry Song)" <song.bao.hua@...ilicon.com>
To:     Mikhail Gavrilov <mikhail.v.gavrilov@...il.com>,
        "sjenning@...ux.vnet.ibm.com" <sjenning@...ux.vnet.ibm.com>
CC:     Linux List Kernel Mailing <linux-kernel@...r.kernel.org>,
        "Linux Memory Management List" <linux-mm@...ck.org>
Subject: RE: kernel BUG at mm/zswap.c:1275! (rc6 - git 61556703b610)



> -----Original Message-----
> From: Mikhail Gavrilov [mailto:mikhail.v.gavrilov@...il.com]
> Sent: Thursday, February 11, 2021 9:58 PM
> To: sjenning@...ux.vnet.ibm.com; Song Bao Hua (Barry Song)
> <song.bao.hua@...ilicon.com>
> Cc: Linux List Kernel Mailing <linux-kernel@...r.kernel.org>; Linux Memory
> Management List <linux-mm@...ck.org>
> Subject: kernel BUG at mm/zswap.c:1275! (rc6 - git 61556703b610)
> 
> Hi folks.
> During the 5.11 test cycle I caught a rare but repeatable problem when
> after a day uptime happens "BUG at mm/zswap.c:1275!". I am still not
> having an idea how to reproduce it, but maybe the authors of this code
> could explain what happens here?

Are you using zsmalloc? There is a known bug on the combination
of zsmalloc and zswap, fixed by patches of tiantao:

mm: set the sleep_mapped to true for zbud and z3fold
mm/zswap: fix variable 'entry' is uninitialized when used
mm/zswap: fix potential memory leak
mm/zswap: add the flag can_sleep_mapped

at Linux-next:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/?qt=author&q=tiantao6%40hisilicon.com


> 
> $ grep "mm/zswap.c" dmesg*.txt
> dmesg101.txt:[127850.513201] kernel BUG at mm/zswap.c:1275!
> dmesg11.txt:[52211.962861] kernel BUG at mm/zswap.c:1275!
> dmesg8.txt:[46610.641843] kernel BUG at mm/zswap.c:1275!
> 
> [127850.513193] ------------[ cut here ]------------
> [127850.513201] kernel BUG at mm/zswap.c:1275!
> [127850.513210] invalid opcode: 0000 [#1] SMP NOPTI
> [127850.513214] CPU: 6 PID: 485132 Comm: brave Tainted: G        W
>    --------- ---  5.11.0-0.rc6.20210204git61556703b610.145.fc34.x86_64
> #1
> [127850.513218] Hardware name: System manufacturer System Product
> Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
> [127850.513221] RIP: 0010:zswap_frontswap_load+0x258/0x260
> [127850.513228] Code: ab 83 aa f0 2f 00 00 01 65 ff 0d c3 73 cd 54 eb
> 88 48 8d 7b 10 e8 78 b9 9f 00 c7 43 10 00 00 00 00 44 8b 63 70 e9 4a
> ff ff ff <0f> 0b 0f 0b 0f 0b 66 90 0f 1f 44 00 00 41 57 31 c0 b9 0c 00
> 00 00
> [127850.513231] RSP: 0000:ffffa92e866c7c48 EFLAGS: 00010282
> [127850.513235] RAX: 0000000000000006 RBX: ffffc92e7ca61830 RCX:
> 0000000000000001
> [127850.513238] RDX: 0000000000000000 RSI: ffffffffab3429fe RDI:
> ffff97f4d0393010
> [127850.513240] RBP: ffff97ee5544d1c0 R08: 0000000000000001 R09:
> 0000000000000000
> [127850.513242] R10: 0000000000000000 R11: 0000000000000000 R12:
> 00000000ffffffea
> [127850.513244] R13: ffff97ee016800c8 R14: ffff97ee016800c0 R15:
> ffffffffc0d54020
> [127850.513247] FS:  00007fcbe628de40(0000) GS:ffff97f507600000(0000)
> knlGS:0000000000000000
> [127850.513249] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [127850.513252] CR2: 0000381208c29250 CR3: 00000001c54ea000 CR4:
> 0000000000350ee0
> [127850.513254] Call Trace:
> [127850.513261]  __frontswap_load+0xc3/0x160
> [127850.513265]  swap_readpage+0x1ca/0x3a0
> [127850.513270]  swapin_readahead+0x2ee/0x4e0
> [127850.513276]  do_swap_page+0x4a4/0x900
> [127850.513279]  ? lock_release+0x1e9/0x400
> [127850.513283]  ? trace_hardirqs_on+0x1b/0xe0
> [127850.513288]  handle_mm_fault+0xe7d/0x19d0
> [127850.513294]  do_user_addr_fault+0x1c7/0x4c0
> [127850.513299]  exc_page_fault+0x67/0x2a0
> [127850.513304]  ? asm_exc_page_fault+0x8/0x30
> [127850.513307]  asm_exc_page_fault+0x1e/0x30
> [127850.513310] RIP: 0033:0x560297642f44
> [127850.513314] Code: 64 75 07 45 8b 76 03 4d 03 f5 45 8b 56 ff 4d 03
> d5 66 41 81 7a 07 83 00 0f 85 4f 01 00 00 8b 5f 13 49 03 dd 8b 5b 03
> 49 03 dd <8b> 4b ff 49 03 cd 66 81 79 07 a5 00 0f 85 0f 00 00 00 8b 4b
> 0f f6
> [127850.513317] RSP: 002b:00007ffc04cd4b30 EFLAGS: 00010202
> [127850.513320] RAX: 0000000000000000 RBX: 0000381208c29251 RCX:
> 0000560297642f00
> [127850.513322] RDX: 00003812080423b1 RSI: 0000381209b11231 RDI:
> 0000381209b1141d
> [127850.513323] RBP: 00007ffc04cd4b90 R08: 0000000000000043 R09:
> 0000000000000024
> [127850.513325] R10: 0000381208042a1d R11: 0000381209b1141f R12:
> 0000000009b1141d
> [127850.513327] R13: 0000381200000000 R14: 0000381208b368ed R15:
> 00003d2fb6b7da10
> [127850.513333] Modules linked in: tun snd_seq_dummy snd_hrtimer
> uinput rfcomm nft_objref nf_conntrack_netbios_ns
> nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib
> nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct
> nft_chain_nat ip6table_nat ip6table_mangle ip6table_raw
> ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6
> nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set
> nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter cmac
> bnep zstd sunrpc vfat fat hid_logitech_hidpp hid_logitech_dj
> snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio
> snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg soundwire_intel
> soundwire_generic_allocation mt76x2u mt76x2_common snd_soc_core
> mt76x02_usb snd_compress mt76_usb snd_pcm_dmaengine soundwire_cadence
> intel_rapl_msr intel_rapl_common iwlmvm snd_hda_codec uvcvideo
> mt76x02_lib snd_usb_audio videobuf2_vmalloc snd_hda_core mt76
> videobuf2_memops videobuf2_v4l2 videobuf2_common
> [127850.513395]  edac_mce_amd snd_usbmidi_lib ac97_bus snd_hwdep
> videodev mac80211 btusb snd_seq kvm_amd snd_rawmidi btrtl btbcm mc
> joydev btintel snd_seq_device kvm bluetooth snd_pcm iwlwifi eeepc_wmi
> snd_timer asus_wmi xpad irqbypass sparse_keymap ecdh_generic snd
> libarc4 ff_memless sp5100_tco rapl video ecc pcspkr wmi_bmof cfg80211
> soundcore i2c_piix4 k10temp rfkill acpi_cpufreq binfmt_misc ip_tables
> amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched crct10dif_pclmul
> crc32_pclmul crc32c_intel drm_kms_helper igb cec nvme dca
> ghash_clmulni_intel drm i2c_algo_bit ccp nvme_core wmi pinctrl_amd
> fuse
> [127850.513453] ---[ end trace 9d733ab8d8b8c1f2 ]---
> [127850.513456] RIP: 0010:zswap_frontswap_load+0x258/0x260
> [127850.513459] Code: ab 83 aa f0 2f 00 00 01 65 ff 0d c3 73 cd 54 eb
> 88 48 8d 7b 10 e8 78 b9 9f 00 c7 43 10 00 00 00 00 44 8b 63 70 e9 4a
> ff ff ff <0f> 0b 0f 0b 0f 0b 66 90 0f 1f 44 00 00 41 57 31 c0 b9 0c 00
> 00 00
> [127850.513462] RSP: 0000:ffffa92e866c7c48 EFLAGS: 00010282
> [127850.513465] RAX: 0000000000000006 RBX: ffffc92e7ca61830 RCX:
> 0000000000000001
> [127850.513467] RDX: 0000000000000000 RSI: ffffffffab3429fe RDI:
> ffff97f4d0393010
> [127850.513469] RBP: ffff97ee5544d1c0 R08: 0000000000000001 R09:
> 0000000000000000
> [127850.513471] R10: 0000000000000000 R11: 0000000000000000 R12:
> 00000000ffffffea
> [127850.513473] R13: ffff97ee016800c8 R14: ffff97ee016800c0 R15:
> ffffffffc0d54020
> [127850.513475] FS:  00007fcbe628de40(0000) GS:ffff97f507600000(0000)
> knlGS:0000000000000000
> [127850.513478] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [127850.513480] CR2: 0000381208c29250 CR3: 00000001c54ea000 CR4:
> 0000000000350ee0
> 
> $ /usr/src/kernels/`uname -r`/scripts/faddr2line
> /lib/debug/lib/modules/`uname -r`/vmlinux zswap_frontswap_load+0x258
> zswap_frontswap_load+0x258/0x260:
> zswap_frontswap_load at mm/zswap.c:1275 (discriminator 1)
> 
> $ git blame mm/zswap.c -L 1265,1285
> Blaming lines:   1% (21/1470), done.
> 1ec3b5fe6eec7 (Barry Song         2020-12-14 19:14:18 -0800 1265)
>  acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx);
> 1ec3b5fe6eec7 (Barry Song         2020-12-14 19:14:18 -0800 1266)
>  mutex_lock(acomp_ctx->mutex);
> 1ec3b5fe6eec7 (Barry Song         2020-12-14 19:14:18 -0800 1267)
>  sg_init_one(&input, src, entry->length);
> 1ec3b5fe6eec7 (Barry Song         2020-12-14 19:14:18 -0800 1268)
>  sg_init_table(&output, 1);
> 1ec3b5fe6eec7 (Barry Song         2020-12-14 19:14:18 -0800 1269)
>  sg_set_page(&output, page, PAGE_SIZE, 0);
> 1ec3b5fe6eec7 (Barry Song         2020-12-14 19:14:18 -0800 1270)
>  acomp_request_set_params(acomp_ctx->req, &input, &output,
> entry->length, dlen);
> 1ec3b5fe6eec7 (Barry Song         2020-12-14 19:14:18 -0800 1271)
>  ret = crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req),
> &acomp_ctx->wait);
> 1ec3b5fe6eec7 (Barry Song         2020-12-14 19:14:18 -0800 1272)
>  mutex_unlock(acomp_ctx->mutex);
> 1ec3b5fe6eec7 (Barry Song         2020-12-14 19:14:18 -0800 1273)
> f1c54846ee450 (Dan Streetman      2015-09-09 15:35:19 -0700 1274)
>  zpool_unmap_handle(entry->pool->zpool, entry->handle);
> 2b2811178e855 (Seth Jennings      2013-07-10 16:05:03 -0700 1275)
>  BUG_ON(ret);
> 2b2811178e855 (Seth Jennings      2013-07-10 16:05:03 -0700 1276)
> a85f878b443f8 (Srividya Desireddy 2018-01-31 16:15:59 -0800 1277) freeentry:
> 2b2811178e855 (Seth Jennings      2013-07-10 16:05:03 -0700 1278)
>  spin_lock(&tree->lock);
> 0ab0abcf51154 (Weijie Yang        2013-11-12 15:08:27 -0800 1279)
>  zswap_entry_put(tree, entry);
> 2b2811178e855 (Seth Jennings      2013-07-10 16:05:03 -0700 1280)
>  spin_unlock(&tree->lock);
> 2b2811178e855 (Seth Jennings      2013-07-10 16:05:03 -0700 1281)
> 2b2811178e855 (Seth Jennings      2013-07-10 16:05:03 -0700 1282)
>  return 0;
> 2b2811178e855 (Seth Jennings      2013-07-10 16:05:03 -0700 1283) }
> 2b2811178e855 (Seth Jennings      2013-07-10 16:05:03 -0700 1284)
> 2b2811178e855 (Seth Jennings      2013-07-10 16:05:03 -0700 1285) /*
> frees an entry in zswap */
> 
> Full kernel log is here: https://pastebin.com/Zqe0FYYD
> 
> --
> Best Regards,
> Mike Gavrilov.

Powered by blists - more mailing lists