Supporting runtimes that support Docker-in-Docker

[blitss]

Creating this issue/discussion here for visibility that maybe someone had overcome this problem or maybe maintainers of Talos have any insights into this issue.

Docker in Docker (dind) can be used to safely run Docker inside containers, which is used by remote development envinroments like Coder.

The most obvious choice for doing that is sysbox which is currently not supported by Talos so I was researching other solutions.

The closest you can get to that currently is by using kata runtimes via extension that was recently aded. It works, but it's not perfect and comes at a price:

kataShared mount types are powered by virtio-fs, a marked improvement over virtio-9p, thanks to kata-containers/runtime#1016. While virtio-fs is normally an excellent choice, in the case of DinD workloads virtio-fs causes an issue -- it cannot be used as a "upper layer" of overlayfs without a custom patch.

Kata maintainers suggest using loop mounted disks which is something that doesn't work in Talos, specifically mount part because it's unable to set up loop device:

mount: can't setup loop device: No space left on device

spec:
  containers:
    - name: docker
      image: docker:20.10-dind
      command: ["sh", "-c"]
      args:
      - if [[ $(df -PT /var/lib/docker | awk 'NR==2 {print $2}') == virtiofs ]]; then
          apk add e2fsprogs &&
          truncate -s 20G /tmp/disk.img &&
          mkfs.ext4 /tmp/disk.img &&
          mount /tmp/disk.img /var/lib/docker; fi &&
        dockerd-entrypoint.sh;
      securityContext:
        privileged: true

So docker in kata runtime works but it's slow because it's using slow storage driver. And to overcome that you have to create a loop mounted disk which currently doesn't work in Talos.

Relevant issues: #5803 #4385 #3922

[smira]

We run dind on Talos ourselves a lot, and the only thing you need is to make sure /var/lib/docker is a volume mount in a pod (e.g. emptyDir work, or any other volume option in Kubernetes). As container's rootfs is overlayfs, you can't run Docker on top of that, as nested overlayfs mounts don't work.

[flashpixx]

can you show a configuration example to run DinD?

[shanduur]

@flashpixx we have that in our tests, see: https://github.com/siderolabs/talos/blob/80ab7a06/internal/integration/k8s/tink.go

[blitss]

also added my example!

[blitss]

btw that's how i finally ran dind with proper isolation using kata (i was using in coder). Didn't figure out how to make in-cluster-DNS work inside of container, but I didn't want to so I disabled it and opted to external DNS resolution.

Goes without saying that you need to enable kata system extension.
I was using piraeus & linstor for storage

apiVersion: v1
kind: Pod
metadata:
  name: kata-dind-check
  labels:
    app: kata-dind-check
spec:
  runtimeClassName: kata
  nodeSelector:
    kubernetes.io/hostname: node5 # hard pin to node5
  securityContext:
    seccompProfile: { type: RuntimeDefault }
  dnsPolicy: None
  dnsConfig:
    nameservers: [1.1.1.1, 9.9.9.9]
    searches: []
  containers:
    - name: dev
      resources:
        requests:
          memory: 16Gi
          cpu: 4000m
      image: mcr.microsoft.com/devcontainers/universal:linux # has dockerd + docker CLI
      securityContext:
        privileged: true # required for dockerd net plumbing & mount inside guest
        allowPrivilegeEscalation: true
      env:
        - name: DOCKER_HOST
          value: unix:///var/run/docker.sock
      volumeMounts:
        - name: docker-run
          mountPath: /var/run
      volumeDevices:
        - name: docker-data-block
          devicePath: /dev/vdb # Kata typically exposes first block PVC here
      readinessProbe:
        exec: { command: ["sh", "-lc", "docker info >/dev/null 2>&1"] }
        initialDelaySeconds: 5
        periodSeconds: 3
      command: ["sh", "-lc"]
      args:
        - |
          # If the block device is blank, format it; then mount it
          blkid /dev/vdb || mkfs.ext4 -F /dev/vdb
          mkdir -p /var/lib/docker
          mount -t ext4 -o noatime /dev/vdb /var/lib/docker

          # Start dockerd with sane flags for k8s
          dockerd --host=unix:///var/run/docker.sock --storage-driver=overlay2 &

          # Stay alive so you can exec in and play
          tail -f /dev/null
  volumes:
    - name: docker-run
      emptyDir: {}
    - name: docker-data-block
      persistentVolumeClaim:
        claimName: docker-data-block
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: docker-data-block
spec:
  accessModes: [ ReadWriteOnce ]
  storageClassName: piraeus-storage-kata
  resources:
    requests:
      storage: 50Gi
  volumeMode: Block