I use ZFS as a storage driver for docker engine running on my machine. Today after my machine rebooted from a crash, yes Linux system crashes too, I notice that all my docker images and containers disappeared.
~ ❯❯❯ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ~ ❯❯❯ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
First thing came to my mind was “did I accidentally destroy docker zfs data set last night?”
# zfs list NAME USED AVAIL REFER MOUNTPOINT tank 1.52T 239G 120K /tank tank/docker 999M 239G 73.9M /var/lib/docker
It’s still there. At that point I suspected that Docker might no longer use ZFS as its data storage.
~ ❯❯❯ docker info Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 1.12.1 Storage Driver: devicemapper Pool Name: docker-253:0-2753561-pool Pool Blocksize: 65.54 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 11.8 MB Data Space Total: 107.4 GB Data Space Available: 31.79 GB Metadata Space Used: 581.6 kB Metadata Space Total: 2.147 GB Metadata Space Available: 2.147 GB ...
The output information from docker info command confirmed my suspicion. But how do we change the storage back to ZFS instead of devicemapper? According to ZFS storage in practice the only prerequisite to have ZFS as the data storage is having
/var/lib/docker as a ZFS dataset.
I was under the impression that
tank/docker was mounted to
/var/lib/docker directory lived on my local LVM file system (hence devicemapper was the storage driver).
To fix this, I stopped docker service, cleared out
/var/lib/docker, and re-mounted the
# systemctl stop docker # rm -rf /var/lib/docker/* # mount tank/docker # systemctl start docker
Let’s see if it’s working again.
docker info Containers: 6 Running: 0 Paused: 0 Stopped: 6 Images: 18 Server Version: 1.12.1 Storage Driver: zfs Zpool: tank Zpool Health: ONLINE Parent Dataset: tank/docker Space Used By Parent: 1047826432 Space Available: 263167705088 Parent Quota: no Compression: off # docker images REPOSITORY TAG IMAGE ID CREATED SIZE abiosoft/caddy latest af55a59400be 2 days ago 40.69 MB ...
Everything seemed to back to normal. I’m still not sure why
tank/docker wasn’t mounted on boot, but I’ll leave it for another day. As for now, I’m quite happy.