Kubernetes & exponential Bidirectional Mounts
This is Part 2 of a Kubernetes debugging report. This part is focussed on secondary issues caused by the crashLoop, and what actually causes it in the kernel.
Part 1 is available here.
WARNING
I am not a kernel engineer. I’m just an Ops guy who started with “why the hell are my mount tables exploding” and somehow ended up reading through the witchcraft and voodoo of the Linux Kernel. Some of this may be incorrect or ‘misguided’.
This post is going to get very long and very dense with information about Kubernetes mountPropagation; The Kernel’s sourcecode, functions and behaviors.
Engage the nerd-afterburners before you carry on reading.
THE PROBLEM
My pod configuration is (was) using the following mount config:
- name: host-volume
mountPath: /var/mnt/core-dump-handler
mountPropagation: Bidirectional
- name: sub-volume
mountPath: /var/mnt/core-dump-handler/cores
mountPropagation: BidirectionalWhenever the container restarted, I saw a fairly unpredictable increase (anywhere from one, to multiple thousands) in the number of mounts on the host. It took a while to figure out what was actually happening, but I managed to reduce this down to a simple replication method:
ps auxf | grep '[a]pp/core' && \
kill -9 $(ps auxf | grep '[a]pp/core' | awk '{print $2}') && \
echo "YASSSS";
sleep 10;
cat /proc/mounts | grep 'core' | wc -lPlonk that in a little script, and this showed me that I actually had exponential growth:
[root@host ~]# ./killScript.sh
15
[root@host ~]# ./killScript.sh
31
[root@host ~]# ./killScript.sh
63Given 12, 13, 14 iterations, I’m suddenly staring down the barrel of tens of thousands of mounts in the host’s mount table. This severely handicaps system performance, and you can see a dramatic increase in the sys CPU% on the host as it tries to process all of these mounts whilst performing it’s jobs.
Anything that touches the mount table (ContainerD or Kubelet, for example) become extremely slow, and struggle to even start/stop pods & containers on that host. Even logging into it to clean it up manually is nigh on impossible, and the instance often needs restarting or trashing for replacement. I wasn’t even able to drain these nodes in some cases.
What I thought was odd about this was that whilst I expected to see mounts leaking back to the host, what I didn’t expect to see was the number of mounts almost doubling with each iteration. That is to say, instead of:
$countMounts = $containerRestarts
What I actually see is:
$countMounst = (2 ^ $containerRestarts)-1
Which, at the time of discovering this, I thought was just… odd. And anyone that knows me knows that when I find something that looks ‘odd’, I cannot help myself but go and find out the cause of the odd thing.
The Solution
… to this specific application’s issues? Quite simply, it is to stop using bidirectional mounts. They are entirely unnecessary, and yet, they are the default mountPropagation mode for this chart, for reasons lost on me.
Anyway, that answer is really boring, and I know that’s not why you’re here, so lets get to the juicy bits…
kubernetes mount modes
Kubernetes has a handful of mountPropagation modes.
‘None’ = Private
mountPropagation mode of “None” (or just unset) asks the kernel to provide a "Private” (MS_PRIVATE) mount. This is the typical kind of mount in kubernetes, and provides the kind of isolation you want from a containered environment.
‘hosttocontainer’ = slave
mountPropagation mode of “hostToContainer” asks the kernel to proide a “Slave” (MS_SLAVE) mount. This means that it will receive mount events from the host, but it won’t be able to leak mounts back down onto said host.
‘bidirectional’ = shared
mountPropagation mode of “bidirectional” asks the kernel to proide a “Shared” (MS_SHARED) mount.
This means that mount events that take place inside the container’s mount namespace can propagate down to the host. This is helpful if you want to manage the mounts on the host from the container, but will very likely impact your security exposure somewhat drastically as a compromised container can, for all intents and purposes, trivially take control of the host.
There are very few usecases for bidirectional mounts, so if you find yourself thinking that you need this mount mode, you really need to consider your choices. And also, you should probably keep reading to make sure you understand the implications of this.
kernel mount modes
MS_PRIVATE
A private mount is completely isolated from mount propagation.
It doesn't belong to any peer group and has no master. Mount events happening elsewhere never reach it, and mount events happening inside it never propagate anywhere else. It's the default mode for mounts.
MS_SLAVE
A slave mount has a master.
Specifically, the master is from a shared peer group that the slave is subordinate to. When a mount event occurs in the master's group, the slave receives a copy of that event (mounts appear inside it automatically). The reverse never happens. if something is mounted inside the slave, that event stays local (similar to private) and is never propagated back to the master's group or anywhere else. It's a one-way relationship: master → slave.
MS_SHARED
A shared mount is a member of a peer group.
When a mount event occurs on any peer in the group, the kernel propagates it to every other peer. Each peer gets a copy of the mount. It's fully bidirectional: Events flow in and out equally, and the kernel "walks" the mounts to propagate the events (subject to some checks on whether or not they are needed for that particular mount).
Any mount created inside a shared mount is broadcast to all peers in the group, i.e any event from a peer is received by every other peer too.
MS_SHARED specifically
Now, we’re going to talk in detail about what happens with Shared mounts, and how they work. Understanding what happens with these is fundamental to this problem.
Event propagation
A shared mount group is a group of mounts that exist in a peer group. When a new mount succesfully joins the group, the kernel walks every other peer in the group and asks: "is this mount event relevant to you?". This relevance check is simple and essentially does:
Is this mount event not for the peer that instigated it?
Is the mountpoint a sub directory of the root path of a peer?
Is this not an orphaned peer?
Lets say you add the following mounts in a shared mount group in order:
- Root: /
- Mount1: /var/1 > /mnt/a
- Mount2: /var/2 > /mnt/a/sub_mountThis will create a mount table that looks like this:
/ / (The root mount)
/var/1 /mnt/a (Mount1)
/var/2 /mnt/a/sub_mount (Mount2 original. Parent: Mount1)
/var/2 /var/1/sub_mount (Mount2 copy. Parent: '/')As you can see, when you add the last mount, both peers in this ring receive the mount event, and mount it.
under the hood
The functions that matter here are the following:
propagate_mnt() - The function that copies mount events to peers.
need_secondary() - The relevancy check ran for each peer.
is_subdir() - “Is the desired mountpoint a subdirectory of this peer’s mount root?” check.
When the mount runs and succeeds:
the event gets passed to propagate_mnt(), which is the thing that walks through the peers in the group.
propagate_mnt() runs need_secondary() for each peer to determine if the event should be ran for that peer.
need_secondary() uses is_subdir() to check if event’s mountpoint dentry (i.e. the thing you want to mount) is a subdir of the the peer’s mount root… and this is where things start to get interesting…
is_subdir()
The function description for is_subdir() is:
Returns true if new_dentry is a subdirectory of the parent (at any depth). Returns false otherwise. […]
The first few lines of is_subdir() are as follows:
bool is_subdir(struct dentry *new_dentry, struct dentry *old_dentry)
{
if (new_dentry == old_dentry)
return true;
# and then lots of other stuff
# ...
}This means that if you run is_subdir() for two dentries that are the same dentry, you will get a response of true. For example (with subpaths instead of dentries):
is_subdir('/my/test/dir', '/my/test/dir') # returns trueHowever, this is obviously intentional and expected, as it’s heavily implied with the words “[…] at any depth” which includes a depth of zero, otherwise I don’t think you’d need to clarify that at all.
Which group to join?
The group is inherited from the source. Lets mount /var into /tmp/mnt:
68 1 259:1 / / ... shared:1 ...
39 68 0:32 / /tmp ... shared:17 ...
4831 39 259:1 /var /tmp/mnt ... shared:1 ...Whilst /tmp/mnt’s parent ID is 39 (tmp), it has inherited the shared:1 group because it’s source is shared:1.
tie it all together
Iteration One
When bidirectional (shared) mounts go onto our pod as:
/var/mnt/core-dump-handler/ (mount1)
/var/mnt/core-dump-handler/cores (mount2)The container starts and mount1 is mounted into the container’s process as shared:1, which you can see in /proc/$pid/mountinfo:
4649 4637 259:1 /var/mnt/core-dump-handler /var/mnt/core-dump-handler ... shared:1 ... Because mount2 is a subdir of mount1 it “passes” the relevancy checks and we mount it “on top” of mount1.
However, mount1is part of shared:1peer group, and it just did a mount event… so the event that applied the mount2 to mount1 also gets given to all other peers in that group. And one of those peers is… our host:
/ / ... shared:1 ...This means that when your container runs for the first time, the mount2 mount will also run on that peer, and join the ring on the actual host as another peer of the shared:1 group. This causes the shared:1 peer group to look like (sans the /var/mnt/core-dump-handler source):
/ /
/var/mnt/core-dump-handler/cores /var/mnt/core-dump-handler/coresiteration two
Container runs, mount1 mounts, mount2 mounts. mount1 passes the mount event to all of it’s peers, of which there are now two (see above).
However, one of those peers is /var/mnt/core-dump-handler/cores. According to the “relevancy” checks performed in need_secondary(), /var/mnt/core-dump-handler/cores is a subdirectory of /var/mnt/core-dump-handler/cores but with a “depth” of zero… meaning that it should mount this mount… directly on top of itself.
Thusly, as well as / running the mount event, so, too, does the /var/mnt/core-dump-handler/cores peer.
And so our peer group looks like this:
/ /
/var/mnt/core-dump-handler/cores /var/mnt/core-dump-handler/cores
/var/mnt/core-dump-handler/cores /var/mnt/core-dump-handler/cores
/var/mnt/core-dump-handler/cores /var/mnt/core-dump-handler/coresIteration three
container runs, blah blah blah… mount1 passes event to peers… needs_secondary() for each of the ‘…/cores/’ mounts passes, and our mount table looks like this:
/ /
/var/mnt/core-dump-handler/cores /var/mnt/core-dump-handler/cores
/var/mnt/core-dump-handler/cores /var/mnt/core-dump-handler/cores
/var/mnt/core-dump-handler/cores /var/mnt/core-dump-handler/cores
/var/mnt/core-dump-handler/cores /var/mnt/core-dump-handler/cores
/var/mnt/core-dump-handler/cores /var/mnt/core-dump-handler/cores
/var/mnt/core-dump-handler/cores /var/mnt/core-dump-handler/coresIteration X
You’ll notice, now, that the mounts are doubling with each iteration. And they do so at a rate of…
(2 ^ $iteration) - 1
replicating this yourself
You can run this small replication to see this in action yourself. If you run this 13, 14, 15 times, it’ll become fairly obvious when it’s… struggling. keep an eye on your sys CPU% at the same time in another terminal if you want.
# Make this use tmpfs as the base, so we don't pollute '/', # but ultimately it's the same outcome if you were to do mkdir -p /tmp/test/dir;
# First generation - 1 mount, and make it shared # to test the "shared" mode specifically
mount --bind --make-shared /tmp/test/dir /tmp/test/dir && \
cat /proc/mounts | grep 'test/dir' | wc -l;
# Second generation - 3 mounts
mount --bind /tmp/test/dir /tmp/test/dir && \
cat /proc/mounts | grep 'test/dir' | wc -l;
!! # Third generation - 7 entries !! # Fourth generation - 15 entries !! # Fifth generation - 31 entries # Repeat the above as many times as you want, # and this will continue at a rate of (2^x)-1 # ... 63, 127, 255, 511, 1023, 2047, ... # But, tidy up after yourself if your node isn't hosed umount /tmp/test/dir
the horses mouth
Realistically, I can barely read the source code as it is… I have no idea how these functions are used; how shared mounts are used for solving weird and wonderful problems; or if this kind of behavior for same dentry in MS_SHARED is intentional by the author. As such, I’ve requested some information from the Linux FS devs… the experts:
My hope is that these chaps can give some insight into this behavior. Specifically some explanation as to why this behaviour is necessary, or if it’s just a simple case of ‘do stupid things, get stupid results’.
my take
Again, I have zero experience with kernel development, and I have absolutely no idea how these processes, functions, and behaviors are used around the globe. That’s what the FS Dev chaps are for - they have extreme depth in this particular niche of linux, and frankly I am not even in the same galaxy, let alone on the same planet, in terms of expertise in this area.
But I’m going to give you my opinion anyway:
I think this is a bug; or ‘unintentional behavior’ at best
Though, I cannot truly express in writing just how much emphasis I am putting on the word ‘think’
My reasoning for this is that it seems as though need_secondary() is using is_subdir() exactly how I would have used it… “Is this a SUB-directory”, and not “is this a subdirectory OR the same directory”. This leads me to think the whilst behaviour here is expected (i.e. what the code actually does), it is not intended (i.e. not what the author wanted it to do).
On top of this; I have never, in my fairly long time working in ops and systems management, come across a case where a directory would need to mount to… itself? What case exists for mounting /var/bob to /var/bob? However, the world of Linux is a remarkable place, and I am sure that there is someone out there that is using this in some absolutely incomprehensible (at least to me) way to solve problems I don’t even know exists.
As such, I will impatiently refresh my emails every hour awaiting the reply of the clever chaps over at the Linux FS Dev mailing list, and update here accordingly if they reply. Though, at the time of writing, we are in the process of seeing 7.2-rc1 going through testing, so I’d imagine these chaps are, and will be for some time, rather busy stabilising that build ready for release.
addendum
tl;dr - I patched the kernel locally, and the patch… works? Maybe? It only breaks the pathalogical case, and the othe handful of cases I found scratching around the internet seem to continue to work just fine.
[ 04/07/26 ]
I’ve tested a few different things here and there does appear to be a legitimate case for self-binds but I’m not sure that the proposed fix will actually break those. I’m building a custom kernel (on 7.2.0-rc1) to test the change to see if it does or not.
—
I’ve ran the patch. Here’s the updated need_secondary():
static bool need_secondary(struct mount *m, struct mountpoint *dest_mp)
{
/* skip ones added by this propagate_mnt() */
if (IS_MNT_NEW(m))
return false;
/* skip if the mountpoint is the same as the mount root */
if (dest_mp->m_dentry == m->mnt.mnt_root)
return false;
/* skip if mountpoint isn't visible in m */
if (!is_subdir(dest_mp->m_dentry, m->mnt.mnt_root))
return false;
/* skip if m is in the anon_ns */
if (is_anon_ns(m->mnt_ns))
return false;
return true;
} It seems to work… fine? propagate_mnt() only runs after the mount has actually ran on the target, so self-referential binds still work just fine. You can mount /test onto /test, and instead of getting (2^x)-1 mounts, you get 2x-1. More info to follow after some more testing, but not going to lie, given that this is my first time tinkering around at this ‘depth’, I’m pleasantly surprised there’s no smoke eminating from my laptop.
—
I’ve ran some test commands for different mount configurations between 7.2.0-rc1 (current mainline HEAD, at time of writing) and 7.2.0-rc1-timspatch (my locally patched kernel), and I’ll show you output differences between the two versions. The only change to behavior in these tests is that of the pathalogical case. But again, I may be missing an obvious case — obvious to those ‘in the know’, at least — for where self-referential binds might need propagation between eachother.
I’m not sure that all the --make-shared is necessary for much of this as we’re using the shared:1 peer group to begin with, which will be inherited. These are mostly fudged from the /tmp/test examples, so forgive me if some of the commands seem… superfluous (it’s because they are). But for the sake of honesty, here it is raw.
# TEST ONE - THE PATHALOGICAL CASE (SELF-REFERENTIAL BINDS)
mount --bind --make-shared /test /test; cat /proc/1/mountinfo | grep test | wc -l; mount --bind /test /test && cat /proc/1/mountinfo | grep test | wc -l; mount --bind /test /test && cat /proc/1/mountinfo | grep test | wc -l; mount --bind /test /test && cat /proc/1/mountinfo | grep test | wc -l;# UNPATCHED OUTPUT 1 3 7 15 # PATCHED OUTPUT 1 3 5 7
# TEST TWO - BIND>SHARE>SLAVE>BIND>SHARE>SUB_BIND
mount --bind /test /test && mount --make-shared /test/ && mount --make-slave /test && mount --bind /test /test && mount --make-shared /test && mount --bind /test/1 /test/2 && cat /proc/1/mountinfo | grep -E '(/ / rw|test)';# UNPATCHED OUTPUT:
33 2 8:3 / / rw,relatime shared:1 - ext4 /dev/sda3 rw,errors=remount-ro
123 33 8:3 /test /test rw,relatime master:1 - ext4 /dev/sda3 rw,errors=remount-ro
127 123 8:3 /test /test rw,relatime shared:146 master:1 - ext4 /dev/sda3 rw,errors=remount-ro
128 127 8:3 /test/1 /test/2 rw,relatime shared:146 master:1 - ext4 /dev/sda3 rw,errors=remount-ro
# PATCHED OUTPUT
33 2 8:3 / / rw,relatime shared:1 - ext4 /dev/sda3 rw,errors=remount-ro
66 33 8:3 /test /test rw,relatime master:1 - ext4 /dev/sda3 rw,errors=remount-ro
70 66 8:3 /test /test rw,relatime shared:143 master:1 - ext4 /dev/sda3 rw,errors=remount-ro
76 70 8:3 /test/1 /test/2 rw,relatime shared:143 master:1 - ext4 /dev/sda3 rw,errors=remount-ro# TEST THREE - BIND>SLAVE>SHARE>SUB_BIND
mount --bind /test /test && mount --make-slave /test && mount --make-shared /test && mount --bind /test/1 /test/2 && cat /proc/1/mountinfo | grep -E '(/ / rw|test)';# UNPATCHED OUTPUT 33 2 8:3 / / rw,relatime shared:1 - ext4 /dev/sda3 rw,errors=remount-ro 123 33 8:3 /test /test rw,relatime shared:146 master:1 - ext4 /dev/sda3 rw,errors=remount-ro 127 123 8:3 /test/1 /test/2 rw,relatime shared:146 master:1 - ext4 /dev/sda3 rw,errors=remount-ro # PATCHED OUTPUT 33 2 8:3 / / rw,relatime shared:1 - ext4 /dev/sda3 rw,errors=remount-ro 66 33 8:3 /test /test rw,relatime shared:143 master:1 - ext4 /dev/sda3 rw,errors=remount-ro 70 66 8:3 /test/1 /test/2 rw,relatime shared:143 master:1 - ext4 /dev/sda3 rw,errors=remount-ro
# TEST FOUR - BIND>SHARE>BIND>SLAVE>SUB_BIND
mount --bind /test /test && mount --make-shared /test && mount --bind /test /test && mount --make-slave /test && mount --bind /test/1 /test/2 && cat /proc/1/mountinfo | grep -E '(/ / rw|test)';# UNPATCHED OUTPUT 33 2 8:3 / / rw,relatime shared:1 - ext4 /dev/sda3 rw,errors=remount-ro 123 128 8:3 /test /test rw,relatime shared:1 - ext4 /dev/sda3 rw,errors=remount-ro 127 123 8:3 /test /test rw,relatime master:1 - ext4 /dev/sda3 rw,errors=remount-ro 128 33 8:3 /test /test rw,relatime shared:1 - ext4 /dev/sda3 rw,errors=remount-ro 166 127 8:3 /test/1 /test/2 rw,relatime master:1 - ext4 /dev/sda3 rw,errors=remount-ro # PATCHED OUTPUT 33 2 8:3 / / rw,relatime shared:1 - ext4 /dev/sda3 rw,errors=remount-ro 66 76 8:3 /test /test rw,relatime shared:1 - ext4 /dev/sda3 rw,errors=remount-ro 70 66 8:3 /test /test rw,relatime master:1 - ext4 /dev/sda3 rw,errors=remount-ro 76 33 8:3 /test /test rw,relatime shared:1 - ext4 /dev/sda3 rw,errors=remount-ro 140 70 8:3 /test/1 /test/2 rw,relatime master:1 - ext4 /dev/sda3 rw,errors=remount-ro
