We have a server that runs a kind of multisite Discourse environment that I discussed a number of years ago in this post. It is an Ubuntu server with Docker installed, and each of the Discourse instances on that server are spun up in Docker containers. It’s a very small, experimental part of what we do. In fact, we discontinued offering Discourse and Ghost in this kind of environment a while back, and are far more interested in options like Cloudron, which makes hosting Ghost a breeze. That said, we have a couple of Discourse instances we still host and today the biggest one went down, which is always a bit of a scare for me given it is a unique environment. So, this post is simply going to retrace my steps in terminal to fix this because I always forget given it is not something I do often enough.
When I learned the server was down I figured I would try stopping and restarting the Container to see if that works. To do that I needed to go to var/discourse:
From there, I tried to stop the container (to find the container name I looked in the /var/discourse/containers/ directory which has all the YAML files for each install, and the container names are everything before the .yml extension.
./launcher stop containername
That will stop the container and the following will restart it:
./launcher start containername
But when I went to stop the container I got the a storage full error, and when I ran a
on the server it was confirmed, the disk was full. I then proceeded to run the trusty NCDU command to get a sense of what was taking up all the space, and I have a suspicion it might be related to this overlay2 storage space issue others have complained about with Docker, but I took the easy route and deleted 10 GBs of old backups for the site and it was immediately back up and running. In the end a restart was not necessary, and I was able to solve a fairly random issue fairly quickly.