mount command not working as expected inside docker:cli and docker:dind images

[thaJeztah]

For some reason, the mount command doesn't output anything in the docker official images ... just posting this here, because I was banging my head against the wall "what???"

docker run --rm alpine:3.23 sh -c 'mount --help 2>&1 | head -n1'
BusyBox v1.37.0 (2025-12-16 14:19:28 UTC) multi-call binary.

docker run --rm alpine:3.23 sh -c 'mount 2>&1 | head -n1'
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/36/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/6/fs,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/37/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/37/work,nouserxattr)

docker run --rm docker:29.2-cli sh -c 'mount --help 2>&1 | head -n1'
BusyBox v1.37.0 (2025-12-16 14:19:28 UTC) multi-call binary.


docker run --rm docker:29.2-cli sh -c 'mount 2>&1 | head -n1'
# no output

Also tried to disable the entrypoint;

docker run --rm --entrypoint="" docker:29.2-cli sh -c 'ps aux'
PID   USER     TIME  COMMAND
    1 root      0:00 ps aux

docker run --rm --entrypoint="" alpine:3.23 sh -c 'ps aux'
PID   USER     TIME  COMMAND
    1 root      0:00 ps aux

Trying older versions of the image; it looks like ... something ... changed between 23 and 24;

docker run --rm docker:23-cli sh -c 'mount 2>&1 | head -n1'overlay on / type overlay (rw,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/82/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/66/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/65/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/64/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/63/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/62/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/61/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/60/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/59/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/58/fs,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/83/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/83/work,nouserxattr)

docker run --rm docker:24-cli sh -c 'mount 2>&1 | head -n1'
# no output

I started to try to see what makes things break, but found the most odd combinations; it seems some combination of filesystem changes (?) could it be a bug in BuildKit not applying changes correctly? See

• [DNM] debugging why mount doesn't work #571

[yosifkit]

I thought this sounded familiar, and it does seem similar to docker-library/busybox#230.

I did some testing locally in WSL2 (without docker desktop) and found that docker:24.0.6-cli worked but docker:24.0.7-cli gave no output. But when I tested on a Debian host, I didn't get any failure (probably because none of the mounts are too long).

The main difference between 24.0.6 and 24.0.7 is that they are different versions of Alpine, 3.18 and 3.19.

[yosifkit]

Maybe the answer is to install the mount binary (from util-linux) rather than relying on the one provided by busybox?

/ # mount
/ # apk add --no-cache mount
(1/4) Installing libeconf (0.8.3-r0)
(2/4) Installing libblkid (2.41.2-r0)
(3/4) Installing libmount (2.41.2-r0)
(4/4) Installing mount (2.41.2-r0)
Executing busybox-1.37.0-r30.trigger
OK: 25.5 MiB in 40 packages
/ # mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12120/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12026/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12025/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12024/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12023/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12022/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12021/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12020/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12019/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12018/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/11221/fs,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12121/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12121/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup type cgroup2 (ro,nosuid,nodev,noexec,relatime,nsdelegate)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
/dev/sdd on /etc/resolv.conf type ext4 (rw,relatime,discard,errors=remount-ro,data=ordered)
/dev/sdd on /etc/hostname type ext4 (rw,relatime,discard,errors=remount-ro,data=ordered)
/dev/sdd on /etc/hosts type ext4 (rw,relatime,discard,errors=remount-ro,data=ordered)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime)
tmpfs on /proc/interrupts type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/latency_stats type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/scsi type tmpfs (ro,relatime)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /sys/firmware type tmpfs (ro,relatime)
/ #

[thaJeztah]

Yeah, so the odd bit was that a vanilla alpine image (same busybox version) did work; that brought me down the rabbit hole 😂

[thaJeztah]

THANKS btw for looking ❤️

[thaJeztah]

Ah, right; reading up on the other ticket, it's related to the size of /proc/mounts, which ... could .. explain some of that odd behavior I saw when trying to narrow it down in the Dockerfile (although it could still be some BuildKit cache randomness)

Reading up on the upstream issue in busybox; https://lists.busybox.net/pipermail/busybox/2025-December/091859.html I see that Natanael provided some patches, but that thread ends here;
https://lists.busybox.net/pipermail/busybox/2025-December/091864.html I'm horrible at those mailing lists, so not sure if the patch is accepted anywhere or if the thread just died down and stalled.

[tianon]

I have been re-checking that mailing list thread for a month now hoping that it'd have some upstream chatter, but the fix that was applied in Alpine in https://gitlab.alpinelinux.org/alpine/aports/-/blob/54df8010645d3dfc64719895e8b32c121ecb3066/main/busybox/0001-mount-fix-parsing-proc-mounts-with-long-lines.patch only doubles the buffer size, so if you end up with more layers and your line is say, 1500 characters, then a 1000 character buffer still isn't big enough and the musl bug returns.

docker-library/busybox#230 (comment) includes my conclusion about this from September:

I think this is a bug/quirk/"feature" of musl -- I can't see a way, given the apparent behavior of musl's getmntent_r function, for BusyBox to work around this except perhaps providing an even bigger buffer, but then how big do they go? It's really hard to choose a "reasonable" upper limit, and I imagine upstream's opinion is going to instead be that you shouldn't use musl with BusyBox

Doesn't Docker itself directly invoke the mount system call? (so wouldn't be affected by this?)

[thaJeztah]

Yeah, it was just for debugging, and I thought it was specific to the DIND image, but .. it wasn't. Let's close