Message-ID: <20260129-geleckt-treuhand-4bb940acacd9@brauner>
Date: Thu, 29 Jan 2026 15:28:24 +0100
From: Christian Brauner <brauner@...nel.org>
To: "Zachary M. Raines" <zachary.raines@...onical.com>
Cc: Alexander Viro <viro@...iv.linux.org.uk>, 
	linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: PROBLEM: Duplicated entries in /proc/<pid>/mountinfo

On Wed, Jan 28, 2026 at 08:49:12AM -0600, Zachary M. Raines wrote:
> Greetings,
> 
> When mounting and unmounting many filesystems, /proc/<pid>/mountinfo sometimes
> contains entries which are duplicated many times.
> 
> Summary
> =======
> 
> Sometimes on a system that is mounting and unmounting filesystems frequently,
> for example running lots of docker containers, the size of /proc/1/mountinfo,
> can become very large -- 100s, to 1000s of entries or more -- with the vast
> majority being a single entry duplicated many times.
> 
> This causes other problems on the system, due to systemd parsing the mount table
> whenever it changes, and eating up a lot of memory, for example [1]. Waiting
> long enough there are rare events where the length of mountinfo can go into the
> millions of lines and lead to OOM and kernel panics.
> 
> Running the reproducers below, I pretty reliably see an Ubuntu virtual machine
> kernel panic due to lack of memory within about 24hrs.
> 
> Versions
> ========
> 
> Bisecting the kernel git history, I was able to track the issue back to
> '2eea9ce4310d8 mounts: keep list of mounts in an rbtree' [2].
> 
> I've tested on 6.19-rc7 in a virtual machine and the issue is still present
> there. /proc/version:
> 
> Linux version 6.19.0-rc7+ (ubuntu@...nel-builder) (gcc (Ubuntu 15.2.0-4ubuntu4)
> 15.2.0, GNU ld (GNU Binutils for Ubuntu) 2.45) #8 SMP PREEMPT_DYNAMIC Tue Jan 27
> 22:33:35 UTC 2026
> 
> running on Ubuntu 25.10
> 
> Reproducer
> ==========
> 
> The problem can be reproduced by mounting and then unmounting tmpfs in a loop
> and in a seperate process reading /proc/1/mountinfo and checking for duplicates.
> 
> I used the following scripts:
> 
> 1. Mounts and unmounts tmpfs
> 
> #!/bin/bash
> counter=0
> while true; do
>     unique_name="tmpfs_$$_$counter"
>    	mkdir -p "/tmp/$unique_name"
>    	sudo mount -t tmpfs "$unique_name" "/tmp/$unique_name"
>    	sudo umount "/tmp/$unique_name"
>    	rmdir "/tmp/$unique_name"
>    	((counter++))
>    	sleep 0.1
> done
> 
> 2. Reads `/prod/1/mountinfo` and checks for duplicates
> 
> #!/bin/bash
> THRESHOLD=75
> echo "Starting monitoring at $(date)"
> while true; do
>     # Get mountinfo entries and count total
>     mountinfo="$(cat /proc/1/mountinfo)"
>     mountinfo_count=$(echo "$mountinfo" | wc -l)
> 
>     if ((mountinfo_count > THRESHOLD)); then
>         echo "$(date): Mount count ($mountinfo_count) exceeds threshold ($THRESHOLD)"
> 
>         # Find and log duplicate mount points with their counts
>         duplicates=$(echo "$mountinfo" | sort | uniq -cd)
> 
>         if [[ -n "$duplicates" ]]; then
>             echo "Duplicate mounts :"
>             echo "$duplicates"
>         fi
>         echo "====="
>         echo "$mountinfo"
>         echo "---"
>     fi
> 
>     sleep 0.1
> done
> 
> Typically, within 5-10 minutes duplicates can be observed, often including
> hundreds or thousands of copies of the same mount point -- although the number
> can rarely spike to much higher values. Given a long enough uptime, I've
> observed up to 1.4 million duplicates at a time.
> 
> The duplication in mountinfo is very intermittent. `cat /proc/1/mountinfo` 100ms
> later shows no duplication.
> 
> Additional diagnostics
> ======================
> 
> While running the script (2.) above, I also ran the following bpftrace script
> 
> 3. Trace vfs_mounts as by `cat /proc/1/mountinfo`
> 
> #!/usr/bin/env bpftrace
> 
> fentry:show_mountinfo / comm == "cat"/ {
>     @mnts[args->mnt] = count();
> }
> 
> tracepoint:sched:sched_process_exit / comm == "cat"/ {
>     for ($mnt : @mnts) {
>         if ($mnt.1 > 1) {
>             printf("Duplicate mount %p\n", $mnt.0);
>             @dups[$mnt.0] = $mnt.1;
>         }
>     }
>     clear(@mnts);
> }
> 
> and observed that a single mount struct was reached multiple times -- perhaps
> unsurprisingly exactly the same number as there were duplicates detected by
> the above script.
> 
> Typical outputs of script (2.) and the bpftrace script above are
> 
> Starting monitoring at Tue Jan 27 20:48:13 UTC 2026
> Tue Jan 27 20:50:43 UTC 2026: Mount count (696) exceeds threshold (75)
> Duplicate mounts :
>   /proc/sys/fs/binfmt_misc: 2 occurrences
>   /tmp/tmpfs_856614_41491: 666 occurrences
> 
> and
> 
> @dups[0xffff88e5fb9f10a0]: 666

Thanks for the report. So it's a bit unfortunate that you're showing
duplication by source path. That's not as useful as that can
legitimately happen. So the better test would be to see whether you get
any duplicated unique mount ids, i.e., whether the same
mnt->mnt_id_unique appears multiple times. Because that's a bug for
sure.

I suspect the issue is real though. I'm appending a patch as a proposed
fix. Can you test that and report back, please? I'm traveling tomorrow
so might take a little.

View attachment "0001-namespace-fix-proc-mount-iteration.patch" of type "text/x-diff" (2101 bytes)

Powered by blists - more mailing lists