I use ZFS as a storage driver for docker engine running on my machine. Today after my machine rebooted from a crash, yes Linux system crashes too, I notice that all my docker images and containers disappeared.

~ ❯❯❯ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
~ ❯❯❯ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

First thing came to my mind was “did I accidentally destroy docker zfs data set last night?”

# zfs list
NAME                                                                                USED  AVAIL  REFER  MOUNTPOINT
tank                                                                               1.52T   239G   120K  /tank
tank/docker                                                                         999M   239G  73.9M  /var/lib/docker

It’s still there. At that point I suspected that Docker might no longer use ZFS as its data storage.

~ ❯❯❯ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-253:0-2753561-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 11.8 MB
 Data Space Total: 107.4 GB
 Data Space Available: 31.79 GB
 Metadata Space Used: 581.6 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
...

The output information from docker info command confirmed my suspicion. But how do we change the storage back to ZFS instead of devicemapper? According to ZFS storage in practice the only prerequisite to have ZFS as the data storage is having /var/lib/docker as a ZFS dataset.

I was under the impression that tank/docker was mounted to /var/lib/docker. Actually /var/lib/docker directory lived on my local LVM file system (hence devicemapper was the storage driver).

To fix this, I stopped docker service, cleared out /var/lib/docker, and re-mounted the tank/docker dataset.

# systemctl stop docker
# rm -rf /var/lib/docker/*
# mount tank/docker
# systemctl start docker

Let’s see if it’s working again.

docker info
Containers: 6
 Running: 0
 Paused: 0
 Stopped: 6
Images: 18
Server Version: 1.12.1
Storage Driver: zfs
 Zpool: tank
 Zpool Health: ONLINE
 Parent Dataset: tank/docker
 Space Used By Parent: 1047826432
 Space Available: 263167705088
 Parent Quota: no
 Compression: off

# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
abiosoft/caddy      latest              af55a59400be        2 days ago          40.69 MB
...

Everything seemed to back to normal. I’m still not sure why tank/docker wasn’t mounted on boot, but I’ll leave it for another day. As for now, I’m quite happy.