Understanding Docker Volume: Persisting a Grafana Configuration File

By Iliana Maifeld-Carucci and Martial Michel

When using Docker Compose, it is relevant to understand Docker volumes and how they are used to persist configurations and data. Persistence is important, because it allows data to continue to exist after the machine or service that created them is terminated. It also allows that information to be transferred to another host computer intact. We will use Docker volumes to create a pseudo-persistence of configuration and possibly data (this depends on what you want to push). It is important to note that for backing up stateful services such as a database, this is not the proper method. For our example, we will be persisting a Grafana configuration file that will visualize data from an InfluxDB database. 

Background on Docker Compose and Volumes

Docker Compose is a powerful tool to use when dealing with deployments of Docker applications with multiple containers. It gives users the ability to explicitly define the specifications needed for each service in their application, and concisely start all the services together with a single command. Docker volumes are the mechanism by which Docker persists data. They are easy to back up, migrate, and share among different containers. Additionally, they do not increase the size of the container using them, but instead exist separate to the run state of the container. That means that if a container is stopped or deleted, a volume will still be running with the data it has been populated with. Volumes are directories that live outside Docker’s Union File System on the host machine but within the Docker area of the host. 

Pre-create the Volumes

The volumes we will need are grafana-storage and influxdb-storage. These will be marked as external for the docker-compose.yml (i.e. the compose file will refuse to start if these volumes do not exist).

First, confirm neither of the required volumes already exist using docker volume ls. Because we are using named volumes, this should be easy to confirm. If they do and you want to start from scratch, you can delete an already existing volume using docker volume rm volname (adapt volname). If you see volumes that you are not sure are needed (by a running container, etc), you can always run docker volume prune, but you should read the message to understand what you are doing.

Create the needed volumes:

docker volume create grafana-storage
docker volume create influxdb-storage

Docker-compose

Below is the docker-compose.yml file we propose to use:

version: "2"

services:
  influxdb:
    image: influxdb:1.7
    restart: always
    ports:
     - "8086:8086"
    volumes:
      - influxdb-storage:/var/lib/influxdb
    environment:
      INFLUXDB_DB: actualdb
  grafana:
    image: grafana/grafana
    restart: always
    ports:
      - "8080:3000"
    depends_on:
      - influxdb
    volumes:
      - grafana-storage:/var/lib/grafana
volumes:
  grafana-storage:
    external: true
  influxdb-storage:
    external: true

Initial Boot

We will use a minimal docker-compose.yml file that starts influxdb (accessible on port 8086) with an actualdb database, and starts grafana (accessible on port 8080).

To use the file, enter the command:

docker-compose up

We are leaving the docker-compose attached to the terminal so we can Ctrl+C it later.

In a web browser, go to http://127.0.0.1:8080/

• Change the admin/admin login to admin/secureadmin or other generic user/password combination.

• In add source, select influxdb, then select HTTP -> Access -> Browser. Modify the default URL to use http://localhost:8086 (this has to do with host resolution within the running container). Go to InfluxDB Details -> Database and use actualdb, and click Save & Test. You should get Data source is working as a pop-up message.

We have done the minimal configuration needed, since the Grafana UI will automatically generate configuration files, and you can add a Dashboard when you push data to it. Grafana will also keep an eye on these configuration files and automatically load changes.

Ctrl+C your docker-compose; next we will test data persistence.

Grab the Grafana Configuration

Let's use a double mount (-v) to access the grafana-storage as /grafana and the local directory as /dmc, and create an archive of the configuration files in that latter directory (named grafana-config.tar.bz2) without the leading /grafana mount (-C). Note that if a nonexistent local directory is mounted to an existing directory in the volume or container, the local directory will be populated with its contents. For example:

docker run --rm -it -v grafana-storage:/grafana -v `pwd`:/dmc ubuntu:18.04 tar cvfj /dmc/grafana-config.tar.bz2 -C /grafana .

Delete the Grafana Config

We could now delete both grafana-storage and influxdb-storage if we so chose. For now, let's just delete the Grafana one using the command: docker volume rm grafana-storage. You will notice that Docker will tell you that this is not possible, because your Exited container still uses it. You can run docker ps -a (the proper way is docker container ps -a nowadays) to see that the two containers we started using docker-compose up are still here. To get rid of them, we can docker rm each one of them in turn. Use the following command to do so using a few piped commands:

docker ps -a | grep 'Exited (' | tr -s " " | cut -d " " -f 1  | xargs docker rm

Once this is done, we can try to delete our volume again: docker volume rm grafana-storage should now succeed.

Start Grafana Again

First, let's confirm docker-compose up will refuse to start since we deleted its volume. This error message seems to confirm it:

ERROR: Volume grafana-storage declared as external, but could not be found [...]

Create a new Grafana volume using

docker volume create grafana-storage

Restart docker-compose up and go to http://localhost:8080. We will see the login page again (the service was restarted, after all), but we cannot login using the modified admin/secureadmin, because we deleted the configuration file and as such the user/password is back to the default admin/admin. You can confirm that the system is not configured if you so desire.

Ctrl+C the docker-compose again.

Let's now populate grafana-storage using the previously saved configuration file and extracting it in the volume using a similar command as earlier, replacing the archive creation by its extraction:

docker run --rm -it -v grafana-storage:/grafana -v `pwd`:/dmc ubuntu:18.04 tar xvfj /dmc/grafana-config.tar.bz2 -C /grafana/

Now, docker-compose up and once again go to http://localhost:8080. This time, we can login with admin/secureadmin, and we are back to a preconfigured data source. You can confirm it is functional by going to Configuration -> Data Sources -> InfluxDB. Then, click the Save & Test button at the bottom. All is still well, as can be seen by the Data source is working message.

Now that we have finished, you can Ctrl+C that docker-compose again.