linux-kernel - 3.18.x: the return of the NFSv3 client BUG on reboot

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <871tmcarz9.fsf@spindle.srvr.nix>
Date:	Fri, 30 Jan 2015 21:17:14 +0000
From:	Nix <nix@...eri.org.uk>
To:	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	linux-nfs@...r.kernel.org,
	"J. Bruce Fields" <bfields@...i.umich.edu>
Subject: 3.18.x: the return of the NFSv3 client BUG on reboot

I'm seeing these BUGs, fairly reliably, on multiple NFSv3 clients (x86
and x86-64) as of 3.18. As with the bug a few months ago, it seems to be
a side effect of trying to create a new client while shutting everything
down:

[  567.092093] BUG: unable to handle kernel NULL pointer dereference at 00000004
[  567.100044] IP: [<c034fbc0>] rpc_new_client+0x13b/0x1f2
[  567.100044] *pde = 00000000
[  567.100044] Oops: 0000 [#1]
[  567.100044] CPU: 0 PID: 521 Comm: su Not tainted 3.18.5+ #1
[  567.100044] task: de28cfc0 ti: de2ba000 task.ti: de2ba000
[  567.100044] EIP: 0060:[<c034fbc0>] EFLAGS: 00010246 CPU: 0
[  567.100044] EIP is at rpc_new_client+0x13b/0x1f2
[  567.100044] EAX: 00000000 EBX: c000c800 ECX: 0000bae0 EDX: 00000005
[  567.100044] ESI: 00000000 EDI: de2ea800 EBP: 00000007 ESP: de2bbcac
[  567.100044]  DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[  567.100044] CR0: 8005003b CR2: 00000004 CR3: 1f63b000 CR4: 00000090
[  567.100044] Stack:
[  567.100044]  000000d0 de2ea800 de2bbd4c 00000000 de2bbd4c de2bbd4c de2bbd10 de2bbf8c
[  567.100044]  c03501ca de2bbd4c df57a940 de2bbd10 c0350334 00000010 c0451150 000280da
[  567.100044]  00000002 00000006 c044b178 00000000 de2bbddc 00000010 df57a940 00000000
[  567.100044] Call Trace:
[  567.100044]  [<c03501ca>] ? rpc_create_xprt+0xc/0x74
[  567.100044]  [<c0350334>] ? rpc_create+0x102/0x10f
[  567.100044]  [<c019ec6a>] ? __block_commit_write.isra.25+0x56/0x7f
[  567.100044]  [<c0359f38>] ? rpcb_create+0x6e/0x7c
[  567.100044]  [<c035a6ab>] ? rpcb_getport_async+0x124/0x25a
[  567.100044]  [<c0133bdf>] ? update_curr+0x81/0xb3
[  567.100044]  [<c0133e97>] ? check_preempt_wakeup+0xf0/0x134
[  567.100044]  [<c013110f>] ? check_preempt_curr+0x21/0x59
[  567.100044]  [<c03552c8>] ? rpcauth_lookupcred+0x3f/0x47
[  567.100044]  [<c017ebb6>] ? __kmalloc+0xa3/0xc4
[  567.100044]  [<c0354c3d>] ? rpc_malloc+0x39/0x48
[  567.100044]  [<c034ef39>] ? call_bind+0x2d/0x2e
[  567.100044]  [<c0354aa5>] ? __rpc_execute+0x5c/0x187
[  567.100044]  [<c03500f2>] ? rpc_run_task+0x55/0x5a
[  567.100044]  [<c0350160>] ? rpc_call_sync+0x69/0x81
[  567.100044]  [<c01e219e>] ? nsm_mon_unmon+0x8c/0xa0
[  567.100044]  [<c01e23c5>] ? nsm_unmonitor+0x5f/0xd3
[  567.100044]  [<c016b3cc>] ? bdi_unregister+0xf2/0x100
[  567.100044]  [<c01df4cc>] ? nlm_destroy_host_locked+0x4f/0x7c
[  567.100044]  [<c01df703>] ? nlmclnt_release_host+0xd8/0xe5
[  567.100044]  [<c01dddd5>] ? nlmclnt_done+0xc/0x14
[  567.100044]  [<c01ce621>] ? nfs_free_server+0x16/0x72
[  567.100044]  [<c01837a4>] ? deactivate_locked_super+0x26/0x37
[  567.100044]  [<c019493e>] ? cleanup_mnt+0x40/0x59
[  567.100044]  [<c012e185>] ? task_work_run+0x4f/0x5f
[  567.100044]  [<c0121373>] ? do_exit+0x264/0x670
[  567.100044]  [<c0182848>] ? vfs_write+0x98/0x151
[  567.100044]  [<c01829c4>] ? SyS_write+0x41/0x82
[  567.100044]  [<c01217cc>] ? do_group_exit+0x2b/0x5d
[  567.100044]  [<c012180d>] ? SyS_exit_group+0xf/0xf
[  567.100044]  [<c03679f6>] ? syscall_call+0x7/0x7
[  567.100044] Code: 89 43 40 8b 44 24 04 89 43 18 8d 43 78 8b 53 40 89 43 3c 8b 12 e8 32 ac 00 00 c7 03 01 00 00 00 a1 dc f0 42 c0 8b 80 00 03 00 00 <8b> 70 04 83 c6 45 89 f0 e8 77 d1 eb ff 83 f8 20 7f 05 89 43 44
[  567.100044] EIP: [<c034fbc0>] rpc_new_client+0x13b/0x1f2 SS:ESP 0068:de2bbcac
[  567.100044] CR2: 0000000000000004
[  567.100044] ---[ end trace 9d49b8a60cfa6a52 ]---
[  567.100044] Kernel panic - not syncing: Fatal exception
[  567.100044] Kernel Offset: 0x0 from 0xc0100000 (relocation range: 0xc0000000-0xe07fffff)
[  567.100044] Rebooting in 5 seconds..

Notably, these only seem to happen after at least one instance of the
dreaded, frequently-occurring but normally harmless

[  101.019879] xprt_adjust_timeout: rq_timeout = 0!
[  101.047744] lockd: server spindle.srvr.nix not responding, still trying

(i.e. if you manage to reboot before one of these happens -- as happened
recently when I was bisecting another problem -- you don't seem to get a
panic).

NFS-related .config options:

CONFIG_NFS_FS=y
# CONFIG_NFS_V2 is not set
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_SWAP is not set
# CONFIG_NFS_V4_1 is not set
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=y
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
# CONFIG_NFSD_V4 is not set
CONFIG_GRACE_PERIOD=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y

If there's any more info I can provide, just ask. I can reproduce this
one quite fast and I have a box I can test on easily (it's my firewall
so I'm knocked off the net every time I do it, but it's not doing many
other jobs and I can reboot it without losing any state to speak of).
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/