Burhan Deniz Abdi

Intro

Ok so I've been working with Docker in production for a while now. If like me your sitting in the same boat, then I'm sure you can appreciate the pain that comes along with Docker and disk space in production environments. In this post I'm going to list some tasks you may find yourself having to do to get yourself out of a pickle.

Tasks

So what are these tasks I mentioned. Well, ever asked yourself any of these questions?;

• Why am I getting log messages about a read only file system?
• I have a cluster of containers running, how can I find out which ones are read only?
• How can reclaim space back from my docker-pool LV?
• My docker-pool LV is full, how can I resize it?
• My docker-pool LV is corrupted how can I recreated it?

You will notice I've structured these questions in a logical way starting with the issue you might see and ending with the worst case scenario. We'll take a look at what tasks we can do to answer these questions.

Prerequisites

So going forwards, I'm expecting you to be;

• Familiar with Docker and comfortable with Docker terminology and concepts.
• Proficient Linux administration skills and understand storage terminologies in Linux systems.

Read Only

So your sitting back, drinking your ice tea. watching your container cluster at work..... Then you start seeing errors getting flagged up, lets say you see them coming into Logstash. Lets say you see an error "filesystem is readonly". Of course this error message is going to be unique to your environment but the key point is that you've deduce you have a disk related issue.
So what causes this? Could be a range of things, however there may be good chances that your docker-pool LV has run out of space. Do the below on your system;

# You should see output something like below when running lvs
$ lvs
LogVol00        VolGroup00 -wi-ao---- 37.97g
LogVol01        VolGroup00 -wi-ao----  1.50g
docker-pool     VolGroup00 -wi-a----- 40.00g          100%

The docker-pool lvs is where all your docker images, containers and ultimately data is stored in. So if you see 100% data utilisation against the docker-pool then you've just confirmed why you are seeing read only error messages.

Determine which containers are read only

From what I've seen in production, when this occurs don't assume all containers on your host become read only. As I've seen only a handful of containers turning read only in my cluster when hitting this issue. Having said that, DO expect to see all sorts of issues occurring in your cluster.
So at this point you will want to do some analysis to determine where all the space has gone. I'll always recommend you do this before jumping straight in and recreating/resizing your docker-pool. Blowing away your docker-pool and recreating it will resolve your problem in the short term, but if you haven't identified what has used up all your space, you are likely to run into this issue again.

Ok so before we can proceed, first we need to get the PID's of all the running containers. We can do this my running the following command on your host:

docker ps -q | xargs docker inspect --format '{{.ID}}, {{.State.Pid}}' | awk {'print $2'}

Now that you have a list of all the PIDs, go into your /Proc directory. Here you will find directories for each of the PIDs you go back. If you have a look in there you will find a mounts file. This file countains the mount points of your running container. It essentially displays the out from a 'mount -l'. Run the below command and you can see if that running container is readonly.

cat mounts | awk '/docker/'

I've created the below little script you can run to automate this process for all your running containers.

#! /bin/bash

echo "ContainerID  ContainerPID  Mounts"

for i in $(/bin/docker ps -q); do
  container_pid=$(/bin/docker inspect -f '{{ .State.Pid }}' ${i})
  /bin/cat /proc/${container_pid}/mounts | /bin/echo ${i} $container_pid $(/bin/awk '/docker/ {print "Container=" $4}')
done

Now that you have identified your RO containers, you can start debugging further. In production I've seen containers not freeing up space because a file is still in use even though it has been deleted, or heavy log output taking up a lot of disk space. These are just some of the things that can be the cause of your disk space issue. Its up to you to identify this and resolve the issue.

Reclaim some diskspace

Ok so the commands you will find here can be run against a running environment however I recommend bringing your services down before doing this, as if only a little bit of space is reclaimed, you could run the risk of instantly filling that back up again. Essentially what we are going to do here is run a fstrim on the filesystem for your containers. Now usually you would run your fstrim on the mountpoint in question. However for containers its a bit more tricky as you can't just do a df to find the mount point as you would on a normal linux host. To do this we need to employ a similar tactic as before. Use the command we saw earlier to get the PIDs of your running containers.

docker ps -q | xargs docker inspect --format '{{.ID}}, {{.State.Pid}}' | awk {'print $2'}

Then cd into the proc locations i.e.;

cd /proc/17756

In here you will see there is a root symlink. This is actually a symlink to your running container fs. Now that you found this you can do an fstrim on it like so;

fstrim -v /proc/17756/root

You can apply this logical to all of your running containers to trim down any empty blocks. As before I've provided a script that will automate doing this for all your running containers.

#! /bin/bash

for i in ‘docker ps -q’; do
  container_pid=$(/bin/docker inspect -f '{{ .State.Pid }}' ${i})
  fstrim -v /proc/${container_pid}/root
done

After running this you should have reclaimed so disk space back. Check your docker-pool and see if it back to reasonable levels with;

lvs

Resize it?

So trimming down has not helped and you need to resize. Ok no problem, if you have free space left in the volume group your docker-pool is sitting on you can do something like;

lvextend -l 100%FREE /dev/VolGroup00/docker-pool

This will take all available space left in your volume group and assign it to your docker-pool. Of course if you no longer have space left in your volume group you will first have to assign more space to it before performing the above.

Recreate it?

You might not care for the data in your docker-pool, or there is some sort of corruption that has gotten too messy to unpick. Well there is a last resort.....Nuke it.
It should go without saying that blowing away the docker-pool will wipe and data Docker will be holding on your cluster. This will be all container resources, metadata and container images..... proceed at your own peril...

So to do this first stop your cluster then stop docker;

systemctl stop docker

Once docker has stopped remove the below files;

rm -rf /var/lib/docker/*

Once these steps are done you will can now blow away the docker pool LV;

lvremove VolGroup00/docker-pool

Now that is done we can create the pool;

lvcreate -T --name docker-pool -l 90%FREE VolGroup00

Now your done, do an lvs and you should see your pool is clean and back down to zero. (note the 90%, leaving space just incase...paranoia is a good thing here. ^_^ )

Fin

Hope this post has helped. Till next time!