Stop /var/lib/docker/overlay2 from filling up your disk with docker system prune

The problem

I noticed that the analytics for this blog had stopped working, so I logged onto the server that hosts the analytics and found that the disk was completely full. After some investigation, I discovered that the /var/lib/docker/overlay2 directory was taking up essentially the entire disk.

What is /var/lib/docker/overlay2 and why can it take up a lot of space?

This folder is where Docker stores the filesystem layers that make up your images and the writable layers used by containers. When you pull an image or run a container, Docker does not treat it as a single monolithic block of data. Instead, it builds everything from layers, and the overlay2 storage driver keeps those layers on disk. In simple terms, this is Docker’s storage area for image layers, container data, and the changes that happen while containers are running.

It can grow quite large because Docker keeps old layers around even after you no longer need them. If you have built images repeatedly, stopped containers, or pulled lots of different images, those unused layers can accumulate over time. In practice, the folder can also retain data for containers that are no longer running until Docker cleans it up.

How to diagnose the issue

If you want to check how much space docker is using, you can use the docker system df command:

me@my-server:~$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          6         6         5.745GB   0B (0%)
Containers      7         7         43.49MB   0B (0%)
Local Volumes   5         5         2.121GB   0B (0%)
Build Cache     0         0         0B        0B

How to fix the issue temporarily

To clean up Docker’s unused data, including the data stored under /var/lib/docker/overlay2, use the following command: docker system prune. By default, this removes stopped containers, dangling images, and unused networks. If you also want to remove unused volumes and build cache, you can add the --volumes flag.

How to fix the issue permanently

Running the command once will get you back on the road, but eventually (sometimes quite quickly!) the issue can return.

For a more permanent fix, you can set this command to run on a schedule using the crontab.

Run crontab -e and add the following (check that the path to the docker binary is the same on your system with whereis docker first):

0 0 * * * /usr/bin/docker system prune -f

This will run the command at midnight each day and remove unused resources without prompting for confirmation.