linux-kernel - Re: kernel NULL pointer dereference: Workqueue: events_unbound nfsd_file_gc_worker, RIP: 0010:svc_wake

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <Z5YwgqH7t2BwYory@eldamar.lan>
Date: Sun, 26 Jan 2025 13:54:26 +0100
From: Salvatore Bonaccorso <carnil@...ian.org>
To: Jeff Layton <jlayton@...nel.org>
Cc: Chuck Lever <chuck.lever@...cle.com>, Neil Brown <neilb@...e.de>,
	Olga Kornievskaia <okorniev@...hat.com>,
	Dai Ngo <Dai.Ngo@...cle.com>, Tom Talpey <tom@...pey.com>,
	linux-nfs@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: kernel NULL pointer dereference: Workqueue: events_unbound
 nfsd_file_gc_worker, RIP: 0010:svc_wake_up+0x9/0x20

Hi Jeff,

On Sun, Jan 26, 2025 at 07:06:09AM -0500, Jeff Layton wrote:
> On Sun, 2025-01-26 at 08:57 +0100, Salvatore Bonaccorso wrote:
> > Hi Jeff,
> > 
> > On Sat, Jan 25, 2025 at 05:55:50PM -0500, Jeff Layton wrote:
> > > On Sat, 2025-01-25 at 21:44 +0100, Salvatore Bonaccorso wrote:
> > > > Hi Chuck, Jeff, NFSD maintainers,
> > > > 
> > > > In Debian we got a report from a user which triggered an issue during
> > > > package updates hwere nfs-kernel-server restart was involved, then
> > > > hanging and included a kernel trace of a NULL pointer dereference.
> > > > 
> > > > The full report is at:
> > > > https://bugs.debian.org/1093734
> > > > 
> > > > While I was not able to trigger the issue, the provided log is as
> > > > follows:
> > > > 
> > > > 2025-01-21T12:07:01.516291+01:00 $HOST kernel: device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
> > > > 2025-01-21T12:07:01.516310+01:00 $HOST kernel: device-mapper: uevent: version 1.0.3
> > > > 2025-01-21T12:07:01.516312+01:00 $HOST kernel: device-mapper: ioctl: 4.48.0-ioctl (2023-03-01) initialised: dm-devel@...ts.linux.dev
> > > > 2025-01-21T12:07:13.528044+01:00 $HOST kernel: NFSD: Using nfsdcld client tracking operations.
> > > > 2025-01-21T12:07:13.528061+01:00 $HOST kernel: NFSD: no clients to reclaim, skipping NFSv4 grace period (net f0000000)
> > > > 2025-01-21T12:07:17.558915+01:00 $HOST blkmapd[1148]: exit on signal(15)
> > > > 2025-01-21T12:07:17.574410+01:00 $HOST blkmapd[239859]: open pipe file /run/rpc_pipefs/nfs/blocklayout failed: No such file or directory
> > > > 2025-01-21T12:07:18.015541+01:00 $HOST kernel: BUG: kernel NULL pointer dereference, address: 0000000000000090
> > > 
> > > Thanks for the bug report. It's getting late here, so I can only take a
> > > quick look. svc_wake_up is pretty small:
> > > 
> > > void svc_wake_up(struct svc_serv *serv)
> > > {
> > >         struct svc_pool *pool = &serv->sv_pools[0];
> > > 
> > >         set_bit(SP_TASK_PENDING, &pool->sp_flags);
> > >         svc_pool_wake_idle_thread(pool);
> > > }
> > > 
> > > pahole on my machine says that struct svc_serv has this at offset 0x90:
> > > 
> > > 	struct svc_pool *          sv_pools;             /*  0x90   0x8 */
> > > 
> > > So it looks like the nn->nfsd_serv was a NULL pointer. That only
> > > happens when we shut down the server, so this looks like a race between
> > > filecache garbage collection with shutdown.
> > > 
> > > The filecache gets shut down in nfsd_shutdown_net, which gets called
> > > _after_ setting the nn->nfsd_serv pointer to NULL. We'll have to look
> > > at whether we can reorder the NULL pointer setting to later, or work
> > > around this some other way.
> > > 
> > > Could I trouble you to open a bug for this at bugzilla.kernel.org?
> > 
> > Thanks a lot for your quick response on it and the analysis.
> > 
> > Sure I can fill a bug in bugzilla.kernel.org, I see you submitted a
> > patch already, do you still want me to do it?
> >
> > If so I try to reference as well all followups so that the information
> > is not spread around threads.
> > 
> > Thanks a lot for your work!
> > 
> 
> I think you can skip the BZ for now.

Ok then I leave the bugzilla bug filling step off.

thanks again for your hard work on the NFS front!

Regards,
Salvatore