Run nested Docker in Incus / LXD containers with BTRFS

Last updated on Sunday, 15 September 2024 3 min read

I like Docker, but I don’t like how it interferes with firewall rules. Moreover, I like to keep the base OS of every server as basic and clean as possible and use unprivileged Incus / LXD containers to run services. My servers usually use BTRFS, as this enables efficient backups through creating snapshots with Incus’ BTRFS storage driver. Docker can also use BTRFS, but requires some additional configuration, explained in this post.

Create and configure the container and storage volume on the Incus host

Create a new Incus container. In this instance, I name this container docker (use any name you like).

incus launch images:debian/12 docker

Create a loop-backed pool named docker (use any name you like).

incus storage create <pool_name> <driver>

E.g.:

incus storage create docker btrfs

Create a custom storage volume within this new pool (use any name you like).

incus storage volume create <pool_name> <filesystem_volume_name>

E.g.:

incus storage volume create docker docker

Add the new storage volume docker, located in storage pool docker using the following syntax

incus config device add <instance_name> <device_name> disk pool=<disk_pool> source=<file_path_on_host> path=<file_path_on_container>

E.g.:

incus config device add docker docker disk pool=docker source=docker path=/var/lib/docker

Configure the previously created Incus docker container to allow nested containerisation using security.nesting. Allow mknod syscalls which may be called to create block or character devices. Allow setxattr calls, which can be used to set extended attributes on files. After changing these settings, restart the docker container.

incus config set docker security.nesting=true security.syscalls.intercept.mknod=true security.syscalls.intercept.setxattr=true
incus restart docker

Configure the container

Enter into the container. If you haven’t set up SSH yet, you can use:

incus exec docker bash

Otherwise, if you have setup SSH and added an entry to your .ssh/config file, use:

ssh docker

Remove possibly remaining old Docker packages.

for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; do sudo apt-get remove $pkg; done

Update the system and add the official Docker package repositories

sudo apt update
sudo apt install ca-certificates curl gnupg
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "\n# Docker \ndeb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee -a /etc/apt/sources.list > /dev/null

sudo apt update

Install Docker

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Optionally, add your admin user to the Docker group, so you don’t need to use sudo to talk to the Docker daemon.

sudo usermod -aG docker $USER

Enable the BTRFS storage driver in Docker

To make use of BTRFS on Docker, configuring the Docker BTRFS driver is required. Therefore, edit or create /etc/docker/daemon.json, before running any containers.

sudo nano /etc/docker/daemon.json

{
  "storage-driver": "btrfs"
}

Reboot the container afterwards.

sudo reboot

Docker can now be used like on any other system.

References

Pieter Hollander
Pieter Hollander