Stopping Docker Containers Gracefully

Last update: May 28, 2020

This is a post from my old blog, originally written in 2015. The old blog is gone, and I decided to repost it here to redirect old links. The content is just slightly adjusted.

I started to work with Docker containers seven years ago. I made my first Docker playground with a bunch of different images. As I began to work on enterprise-level applications deployment, I found out that there were a lot of things I was doing wrong. One of them was how I started applications inside a container.

Almost all my Dockerfiles have some bash script at the end to make some minor changes before starting the application. I usually add a bash script to CMD instruction in the Dockerfile. I thought there isn't anything wrong with that, except that docker stop didn't work as it should.

Docker Containers and PID 1

When you run a bash script in a container, it will get PID 1, and the application will be a child process as PPID 1. This is a problem because Bash will not forward the termination signal SIGTERM to the app on container stop instruction. Instead, Docker kills the container after 10 seconds. You can adjust the stop timeout, of course, but the main reason it exists is if the application needs more time to stop gracefully.

There is an easy way to handle this with the exec command inside a bash script. Exec will replace the shell without creating a new process, and the application will get PID 1 instead.

Let’s test both scenarios first. For testing purposes, you can use this simple Redis Dockerfile:

FROM ubuntu:trusty
ENV DEBIAN_FRONTEND noninteractive

RUN \
  apt-get update && \
  apt-get -y install \
          software-properties-common && \
  add-apt-repository -y ppa:chris-lea/redis-server && \
  apt-get update && \
  apt-get -y install \
          redis-server && \
  rm -rf /var/lib/apt/lists/*

COPY start.sh start.sh
RUN chmod +x start.sh

EXPOSE 6379

RUN rm /usr/sbin/policy-rc.d
CMD ["/start.sh"]

And here is the start.sh script which will change recommended kernel settings and start Redis server:

#!/usr/bin/env bash

# Disable THP Support in kernel
echo never > /sys/kernel/mm/transparent_hugepage/enabled
# TCP backlog setting (defaults to 128)
sysctl -w net.core.somaxconn=16384
#---------------------------------------------------------------
/usr/bin/redis-server

Now let’s build and run this container (privileged option must be set to true when starting a container because of kernel changes):

docker build -t my/redis .
docker run -d --privileged --name test my/redis

Then check for the processes running inside the container:

docker exec test ps -ef
  UID        PID  PPID  C STIME TTY          TIME CMD
  root         1     0  0 13:20 ?        00:00:00 bash /start.sh
  root         6     1  0 13:20 ?        00:00:00 /usr/bin/redis-server *:6379

As you can see, Redis is running as PID 6, which is why graceful stop doesn't work. Let’s try to stop this container with a docker stop test. Docker will kill the container after 10 seconds. If you check container logs docker logs test, the last message will be Ready to accept connections, meaning Redis didn't receive termination signal.

The simplest way of gracefully stopping Redis container is to change the last line in start.sh script to exec /usr/bin/redis-server:

#!/usr/bin/env bash

# Disable THP Support in kernel
echo never > /sys/kernel/mm/transparent_hugepage/enabled
# TCP backlog setting (defaults to 128)
sysctl -w net.core.somaxconn=16384
#---------------------------------------------------------------
exec /usr/bin/redis-server

Rebuild the image, start the container, and again check for running processes:

docker exec test ps -ef
  UID        PID  PPID  C STIME TTY          TIME CMD
  root         1     0  1 13:24 ?        00:00:00 /usr/bin/redis-server *:6379

As you can see now, Redis is running as PID 1, and docker stop will work just fine. Let's try it first and recheck the Docker logs. You should see this message in the Redis log Received SIGTERM scheduling shutdown...

In the above cases, Redis is running as root user, which a bad practice. Here are a few more examples of how I’m using exec with Postgres and Tomcat containers where processes are not running with root user:

exec sudo -E -u tomcat7 ${CATALINA_HOME}/bin/catalina.sh run
exec su postgres -c "${POSTGRES_BIN} -D ${PGDATA} -c config_file=${CONF}"

I will not go into the details for the above commands, but in this case, processes will not be running as PID 1 because of sudo and su commands. However, Docker stop works correctly in both cases. Those commands will forward the SIGTERM signal to child processes, unlike Bash.