linux-kernel - Re: [bugreport 5.9-rc8] general protection fault in __bfq_deactivate

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <YKcf+xN/G4O75vZp@T590>
Date:   Fri, 21 May 2021 10:50:35 +0800
From:   Ming Lei <ming.lei@...hat.com>
To:     Hillf Danton <hdanton@...a.com>
Cc:     Paolo Valente <paolo.valente@...aro.org>,
        Ming Lei <tom.leiming@...il.com>,
        Dmitry Vyukov <dvyukov@...gle.com>,
        Mikhail Gavrilov <mikhail.v.gavrilov@...il.com>,
        linux-block <linux-block@...r.kernel.org>,
        Jens Axboe <axboe@...com>, LKML <linux-kernel@...r.kernel.org>
Subject: Re: [bugreport 5.9-rc8] general protection fault in
 __bfq_deactivate_entity

On Sat, Mar 06, 2021 at 07:15:24PM -0700, Hillf Danton wrote:
> On Fri, 5 Mar 2021 18:01:04 +0800  Ming Lei wrote:
> > On Fri, Mar 05, 2021 at 10:32:04AM +0100, Paolo Valente wrote:
> > > I'm thinking of a way to debug this too.  The symptom may hint at a
> > > use-after-free.  Could you enable KASAN in your tests?  (On the flip
> > > side, I know this might change timings, thereby making the fault
> > > disappear).
> > 
> > I have asked our QE to reproduce the issue with debug kernel, which may take a
> > while. And I can't trigger it in my box.
> > 
> > BTW, for the 2nd 'kernel NULL pointer dereference', the RIP points to:
> > 
> > (gdb) l *(__bfq_deactivate_entity+0x5b)
> > 0xffffffff814c31cb is in __bfq_deactivate_entity (block/bfq-wf2q.c:1181).
> > 1176		 * bfq_group_set_parent has already been invoked for the group
> > 1177		 * represented by entity. Therefore, the field
> > 1178		 * entity->sched_data has been set, and we can safely use it.
> > 1179		 */
> > 1180		st = bfq_entity_service_tree(entity);
> > 1181		is_in_service = entity == sd->in_service_entity;
> > 1182
> > 1183		bfq_calc_finish(entity, entity->service);
> > 1184
> > 1185		if (is_in_service)
> > 
> > Seems entity->sched_data points to NULL.
> 
> Hi Ming,
> 
> Thanks for your report.
> 
> Given the invalid pointer cannot explain line 1180, you are reporting
> a different issue from what Mike reported, and we can do nothing now
> for both without a reproducer.

BTW, we get this report 2 times on 5.12 kernel, following the kernel log, and this
time there is hard LOCKUP.


[  337.526984] systemd-shutdown[1]: Not all DM devices detached, 1 left.
[  337.526988] systemd-shutdown[1]: Cannot finalize remaining DM devices, continuing.
[  337.531043] systemd-shutdown[1]: Successfully changed into root pivot.
[  337.531046] systemd-shutdown[1]: Returning to initrd...
[  337.533136] watchdog: watchdog0: watchdog did not stop!
[  337.569177] dracut Warning: Killing all remaining processes
[  337.706605] XFS (dm-0): Unmounting Filesystem
[  351.593888] NMI watchdog: Watchdog detected hard LOCKUP on cpu 2
[  351.593890] Modules linked in: dm_multipath rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_ssc fscache rfkill sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm mgag200 dcdbas iTCO_wdt irqbypass i2c_algo_bit iTCO_vendor_support rapl drm_kms_helper intel_cstate syscopyarea sysfillrect sysimgblt fb_sys_fops cec intel_uncore pcspkr ipmi_ssif mei_me mei lpc_ich ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter drm fuse ip_tables xfs libcrc32c sd_mod qla2xxx ahci libahci nvme_fc crct10dif_pclmul crc32_pclmul crc32c_intel nvme_fabrics libata ghash_clmulni_intel tg3 nvme_core megaraid_sas t10_pi scsi_transport_fc wmi dm_mirror dm_region_hash dm_log dm_mod
[  351.593929] CPU: 2 PID: 95 Comm: kworker/2:1 Kdump: loaded Tainted: G               X --------- ---  5.12.0-1.el9.x86_64 #1
[  351.593930] Hardware name: Dell Inc. PowerEdge R430/0HFG24, BIOS 1.6.2 01/08/2016
[  351.593931] Workqueue: cgwb_release cgwb_release_workfn
[  351.593932] RIP: 0010:rb_prev+0x18/0x50
[  351.593933] Code: 31 c0 eb db 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 8b 17 48 39 d7 74 35 48 8b 47 10 48 85 c0 74 1c 49 89 c0 48 8b 40 08 <48> 85 c0 75 f4 4c 89 c0 c3 48 3b 78 10 75 f6 48 8b 10 48 89 c7 48
[  351.593934] RSP: 0018:ffffb7280048fd70 EFLAGS: 00000086
[  351.593935] RAX: ffff98bc30f448a0 RBX: ffff98bc10d1e150 RCX: 0000000000000014
[  351.593936] RDX: 0000000000000001 RSI: ffff98bc00b39098 RDI: ffff98bc00b39098
[  351.593937] RBP: ffff98bc00b39098 R08: ffff98bc30f448a0 R09: 0000000000000000
[  351.593938] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
[  351.593939] R13: 0000000000000001 R14: ffff98bc10d1e110 R15: 0000000000000000
[  351.593940] FS:  0000000000000000(0000) GS:ffff98c37fa80000(0000) knlGS:0000000000000000
[  351.593941] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  351.593941] CR2: 00007fffeeceaea0 CR3: 0000000105b40003 CR4: 00000000001706e0
[  351.593942] Call Trace:
[  351.593943]  bfq_idle_extract+0x98/0xb0
[  351.593943]  __bfq_deactivate_entity+0x224/0x240
[  351.593944]  bfq_pd_offline+0xaa/0xd0
[  351.593945]  blkg_destroy+0x52/0xf0
[  351.593945]  blkcg_destroy_blkgs+0x46/0xc0
[  351.593946]  cgwb_release_workfn+0xbe/0x150
[  351.593947]  process_one_work+0x1e6/0x380
[  351.593947]  worker_thread+0x53/0x3d0
[  351.593948]  ? process_one_work+0x380/0x380
[  351.593949]  kthread+0x11b/0x140
[  351.593949]  ? kthread_associate_blkcg+0xa0/0xa0
[  351.593950]  ret_from_fork+0x22/0x30
[  351.593950] Kernel panic - not syncing: Hard LOCKUP


Thanks,
Ming