linux-ext4 - Soft Lockup on running fsstress with next-20240327

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID:
 <DM4PR12MB508625859AF5FAC947815C02893A2@DM4PR12MB5086.namprd12.prod.outlook.com>
Date: Fri, 29 Mar 2024 03:31:23 +0000
From: "V, Narasimhan" <Narasimhan.V@....com>
To: "linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>,
	"linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>,
	"linux-mm@...ck.org" <linux-mm@...ck.org>, Linux Kernel Mailing List
	<linux-kernel@...r.kernel.org>
CC: "Petkov, Borislav" <Borislav.Petkov@....com>
Subject: Soft Lockup on running fsstress with next-20240327

[AMD Official Use Only - General]

Hi,

With next-20240327, we see soft lockup on running fstress on ext4 filesystem.
I have a loop device created and running fsstress from LTP.
The LTP fsstress runs on the loop device on a combination of raid or no raid, lv and without lv, ext4 fs.

(02/20) avocado-misc-tests/io/disk/ltp_fsstress.py:LtpFs.test_fsstress_run;run-fs-ext4-lv-lv-raid-no_raid-c2c0: STARTED
Mar 27 11:47:42 system_name kernel: [13803.399479] CPU: 73 PID: 341283 Comm: kworker/u514:2 Not tainted 6.9.0-rc1-next-20240327-1711524620260 #1
Mar 27 11:47:42 system_name kernel: [13803.399482] Hardware name: Dell Inc. PowerEdge R6515/07PXPY, BIOS 2.13.3 09/12/2023
Mar 27 11:47:42 system_name kernel: [13803.399484] Workqueue: loop6 loop_rootcg_workfn
Mar 27 11:47:42 system_name kernel: [13803.399491] RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x300
Mar 27 11:47:42 system_name kernel: [13803.399497] Code: 92 c2 41 8b 04 24 0f b6 d2 c1 e2 08 30 e4 09 d0 a9 00 01 ff ff 0f 85 f2 01 00 00 85 c0 74 14 41 0f b6 04 24 84 c0 74 0b f3 90 <41> 0f b6 04 24 84 c0 75 f5 b8 01 00 00 00 66 41 89 04 24 5b 41 5c
Mar 27 11:47:42 system_name kernel: [13803.399499] RSP: 0018:ffffb5eb037bf9b8 EFLAGS: 00000202
Mar 27 11:47:42 system_name kernel: [13803.399501] RAX: 0000000000000001 RBX: ffff90fc123ce068 RCX: 0000000000000000
Mar 27 11:47:42 system_name kernel: [13803.399503] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff90fc123ce0c0
Mar 27 11:47:42 system_name kernel: [13803.399504] RBP: ffffb5eb037bf9e0 R08: 0000000000000000 R09: ffffb5eae0704500
Mar 27 11:47:42 system_name kernel: [13803.399506] R10: 0000000000000000 R11: ffffb5eae0704500 R12: ffff90fc123ce0c0
Mar 27 11:47:42 system_name kernel: [13803.399507] R13: ffff90fc16a75140 R14: 000000010033682a R15: ffff90fc123ce0c0
Mar 27 11:47:42 system_name kernel: [13803.399509] FS:  0000000000000000(0000) GS:ffff910b5fa80000(0000) knlGS:0000000000000000
Mar 27 11:47:42 system_name kernel: [13803.399511] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 27 11:47:42 system_name kernel: [13803.399512] CR2: 00007fd6ada61ea0 CR3: 00000010b4724003 CR4: 0000000000770ef0
Mar 27 11:47:42 system_name kernel: [13803.399514] PKRU: 55555554
Mar 27 11:47:42 system_name kernel: [13803.399515] Call Trace:
Mar 27 11:47:42 system_name kernel: [13803.399517]  <IRQ>
Mar 27 11:47:42 system_name kernel: [13803.399521]  ? show_regs+0x6d/0x80
Mar 27 11:47:42 system_name kernel: [13803.399526]  ? watchdog_timer_fn+0x20e/0x290
Mar 27 11:47:42 system_name kernel: [13803.399530]  ? __pfx_watchdog_timer_fn+0x10/0x10
Mar 27 11:47:42 system_name kernel: [13803.399533]  ? __hrtimer_run_queues+0x111/0x290
Mar 27 11:47:42 system_name kernel: [13803.399536]  ? srso_alias_return_thunk+0x5/0xfbef5
Mar 27 11:47:42 system_name kernel: [13803.399541]  ? hrtimer_interrupt+0x108/0x240
Mar 27 11:47:42 system_name kernel: [13803.399546]  ? __sysvec_apic_timer_interrupt+0x5d/0x150
Mar 27 11:47:42 system_name kernel: [13803.399549]  ? sysvec_apic_timer_interrupt+0x80/0xb0
Mar 27 11:47:42 system_name kernel: [13803.399554]  </IRQ>
Mar 27 11:47:42 system_name kernel: [13803.399555]  <TASK>
Mar 27 11:47:42 system_name kernel: [13803.399556]  ? asm_sysvec_apic_timer_interrupt+0x1f/0x30
Mar 27 11:47:42 system_name kernel: [13803.399562]  ? native_queued_spin_lock_slowpath+0x86/0x300
Mar 27 11:47:42 system_name kernel: [13803.399566]  _raw_spin_lock+0x2d/0x40
Mar 27 11:47:42 system_name kernel: [13803.399569]  __wb_update_bandwidth+0x38/0x1e0
Mar 27 11:47:42 system_name kernel: [13803.399574]  do_writepages+0x256/0x280
Mar 27 11:47:42 system_name kernel: [13803.399579]  filemap_fdatawrite_wbc+0x79/0xb0
Mar 27 11:47:42 system_name kernel: [13803.399583]  ? inotify_free_event+0x15/0x20
Mar 27 11:47:42 system_name kernel: [13803.399586]  __filemap_fdatawrite_range+0x6d/0x90
Mar 27 11:47:42 system_name kernel: [13803.399595]  file_write_and_wait_range+0x5c/0xb0
Mar 27 11:47:42 system_name kernel: [13803.399598]  ext4_sync_file+0x8f/0x440
Mar 27 11:47:42 system_name kernel: [13803.399602]  ? srso_alias_return_thunk+0x5/0xfbef5
Mar 27 11:47:42 system_name kernel: [13803.399605]  vfs_fsync+0x56/0x90
Mar 27 11:47:42 system_name kernel: [13803.399608]  ? blk_mq_complete_request+0x58/0x80
Mar 27 11:47:42 system_name kernel: [13803.399613]  loop_process_work+0x3c4/0x8a0
Mar 27 11:47:42 system_name kernel: [13803.399616]  ? srso_alias_return_thunk+0x5/0xfbef5
Mar 27 11:47:42 system_name kernel: [13803.399622]  loop_rootcg_workfn+0x1f/0x30
Mar 27 11:47:42 system_name kernel: [13803.399624]  process_one_work+0x188/0x3d0
Mar 27 11:47:42 system_name kernel: [13803.399629]  ? __pfx_worker_thread+0x10/0x10
Mar 27 11:47:42 system_name kernel: [13803.399631]  worker_thread+0x2ce/0x3e0
Mar 27 11:47:42 system_name kernel: [13803.399634]  ? __pfx_worker_thread+0x10/0x10
Mar 27 11:47:42 system_name kernel: [13803.399637]  kthread+0xe8/0x120
Mar 27 11:47:42 system_name kernel: [13803.399639]  ? __pfx_kthread+0x10/0x10
Mar 27 11:47:42 system_name kernel: [13803.399642]  ret_from_fork+0x40/0x60
Mar 27 11:47:42 system_name kernel: [13803.399645]  ? __pfx_kthread+0x10/0x10
Mar 27 11:47:42 system_name kernel: [13803.399647]  ret_from_fork_asm+0x1a/0x30
Mar 27 11:47:42 system_name kernel: [13803.399654]  </TASK>

Happens with next-20240328 as well


Steps to reproduce:
1. Clone and install https://github.com/linux-test-project/ltp
2. Get a disk ready or create loop device to be treated as disk.
3. Run fsstress from testcases/kernel/fs/fsstress - ./fsstress -d disk_name -n 250 -p 250 -r -l 1

The disk can be raw / raid created on it / lv created on it / mounted as ext4 fs.


--
Regards
Narasimhan V