[SOLVED] Docker: no space left on device, but Ubuntu has over 900GB free

Issue

I have a dedicated server with about 900gb of free space left.

I have about 5-6 images on docker running atm, and I'm trying to spin up another instance for mongo.

 docker run --name mongo-database -d mongo

However, for some reason I get the error

docker: write /var/lib/docker/tmp/GetImageBlob907911044: no space left on device.
See 'docker run --help'.

Here are the results of my df -l

Filesystem      1K-blocks      Used Available Use% Mounted on
udev              4015064         0   4015064   0% /dev
tmpfs              807140     90900    716240  12% /run
/dev/sda2        20026236  20009852         0 100% /
tmpfs             4035692         0   4035692   0% /dev/shm
tmpfs                5120         0      5120   0% /run/lock
tmpfs             4035692         0   4035692   0% /sys/fs/cgroup
/dev/sda3      1902052484 840138092 965272704  47% /home
shm                 65536         0     65536   0% /var/lib/docker/containers/2e7393b7423c1b6fea4281cb62d36c19a675361647e90a626447f2819834c11f/shm
shm                 65536         8     65528   1% /var/lib/docker/containers/31230c5f2084670d51b8ce209195d77e75eb50e2458a0f3d2814df70738af824/shm
shm                 65536         0     65536   0% /var/lib/docker/containers/4926d95aff554033316ca77be5c9af1c1f9f498b5302eb7595c4c73255e9d561/shm
shm                 65536         0     65536   0% /var/lib/docker/containers/fc8d440f5cc0af51753d9c8d8f7b0f3acecef0c24dd04252ad7e7adbad108064/shm
shm                 65536         0     65536   0% /var/lib/docker/containers/2aea985f6b820534f1b21f297f5ab35fed98757781e94b6db217f7b8b90da243/shm
shm                 65536         0     65536   0% /var/lib/docker/containers/73d69f72ee727b1528185d38e6eb8c5281a3be53aaa32f28f2790800f9da6104/shm
tmpfs              807136         0    807136   0% /run/user/0
shm                 65536         0     65536   0% /var/lib/docker/containers/fbf96c6191adcb2d8eeb6b7fc71f2a2af4d9f167b0e3865a79b17d7c882ea227/shm
shm                 65536         0     65536   0% /var/lib/docker/containers/42ac3d9fb6611e652ff5a6fb49183e3d8ef645734ccb3e74ade31194cff4204e/shm
shm                 65536         0     65536   0% /var/lib/docker/containers/a48f0a74716f9c320e746b9d99f0fd53cf1e7b3e5fc8e0021203d45ab2800842/shm
shm                 65536         4     65532   1% /var/lib/docker/containers/cb2358b204f1ca68eefdb3b0950e6418c36c524a62957714f5b05c97ca1c3b40/shm
shm                 65536         4     65532   1% /var/lib/docker/containers/b388580576c4f72b8729815e6a5d86796d1f4527e5a98606f7d6875b7c95ca28/shm

Thanks!

Solution

Your problem is that /dev/sda2 is full. To moved your data to /dev/sda3, and keep as much of your current setup as possible, follow these steps:

1. Stop docker. You're not going to be able to do this without downtime. sudo systemctl stop docker
2. Move /var/lib/docker to somewhere within /home. Your choice, I'll use /home/docker. sudo mv /var/lib/docker /home/docker.
3. Link it back to /var/lib/docker, with a bind mount, which should be more stable. sudo mount --bind /home/docker /var/lib/docker
4. Start docker again. sudo systemctl start docker

Answered By - tcnj

Answer Checked By - Dawn Plyler (WPSolving Volunteer)