Proposal: hooks for `RUN` instructions (use cases: reproducible builds, cross-compilation, malware detection, ...)

[AkihiroSuda]

I'd like to propose a hooking mechanism for RUN instructions of Dockerfile.

e.g.,

buildctl build \
  --frontend dockerfile.v0 \
  --opt hook="$(cat hook.json)"

with hook.json as follows:

{
  "RUN": {
    "entrypoint": ["/dev/.dfhook/entrypoint"],
    "mounts": [
       {"from": "example.com/hook", "target": "/dev/.dfhook"},
       {"type": "secret", "source": "something", "target": "/etc/something"}
    ]
  }
}

This will let the frontend treat RUN foo as:

RUN \
  --mount=from=example.com/hook,target=/dev/.dfhook \
  --mount=type=secret,source=something,target=/etc/something \
  /dev/.dfhook/entrypoint foo

docker history will still show this as RUN foo.

Note

The proposed json schema may change.
See the PR for the latest status:

• dockerfile: implement hooks for RUN instructions #4669

Use cases

Reproducible builds

A hook can be used for wrapping apt-get command to use snapshot.debian.org for reproducing package versions without modifying the Dockerfile.

The /dev/.dfhook/entrypoint script can be like this:

#!/bin/bash
set -eu -o pipefail

: "${SOURCE_DATE_EPOCH:=$(stat --format=%Y /etc/apt/sources.list.d/debian.sources)}"
snapshot="$(printf "%(%Y%m%dT%H%M%SZ)T\n" "${SOURCE_DATE_EPOCH}")"
. /etc/os-release

# Rewrite /etc/apt to use snapshot.debian.org
cp -a /etc/apt /etc/apt.bak
rm -f /etc/apt/sources.list.d/debian.sources
cat <<EOF >>/etc/apt/sources.list
deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/${snapshot} ${VERSION_CODENAME} main
deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/${snapshot} ${VERSION_CODENAME}-security main
deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/${snapshot} ${VERSION_CODENAME}-updates main
EOF

# Run the command
set +e
"$@"
status=$?
set -e

# Restore /etc/apt
rm -rf /etc/apt
mv /etc/apt.bak /etc/apt

exit $status

A hook may also push/pull dpkg blobs to an OCI registry (or whatever) for efficient caching.

Cross-compilation

xx-apt, etc. (https://github.com/tonistiigi/xx) can be reimplemented as a hook.

Malware detection

A hook may use seccomp, etc. to hook the syscalls and detect malicious actions, etc.

Enterprise networking

Enterprise networks often require installing a MITM proxy cert.
This can be easily automated with a hook.

FAQs

• Q. Why not just modify Dockerfile?
  • A. Because it affects the history object in OCI Image Config and decreases reproducibility

[AkihiroSuda]

@tonistiigi @thaJeztah SGTY?

[thaJeztah]

Would these hooks run inside the container that's run as part of the RUN, or is this all running on the host? If it's inside the container, would this be similar to a custom SHELL (to set a custom entry point for RUN steps)?

[AkihiroSuda]

PR:

• dockerfile: implement hooks for RUN instructions #4669

[AkihiroSuda]

Would these hooks run inside the container that's run as part of the RUN, or is this all running on the host?

Inside.

If it's inside the container, would this be similar to a custom SHELL (to set a custom entry point for RUN steps)?

Yes, somewhat similar.

(I noticed I accidentally edited @thaJeztah's original comment 😅 . Now reverted. )

[tonistiigi]

My initial impression is that this is not where we want to take Dockerfiles. Dockerfiles should be self-contained and guaranteed to work. This makes the outcome of a build undefined and likely broken if the author repository and hook come from a different codebase. If the hook is useful then why doesn't Dockerfile author integrate it into their build process or make it so that it can be turned on by some config variable and tested.

Even if used for some hand-written script, it will make inefficient builds because the "hooks" leak to everywhere. The foundation of a good Dockerfile is to define minimal dependencies for every point of your build. That gives speed, efficient caching and small images. From that point, if the problem is that you wish to build a Dockerfile from some external source that you can't modify but add some modifications, I would even consider some option to load a patch file over it before it is processed. At least in that case the patch can choose exactly what modifications are needed. Atm it is limited to what can be modified but once you add a modification it applies everywhere.

I'd like to understand more what is the actual problem with reproducible builds and how is this solving it. We want to make it easy for image authors to make their builds reproducible, but I'm not sure how this is helping (unless we ask image author to now call make instead of docker build that would attach some hook to it, but that doesn't improve docker build and increases fragmentation and untested branches).

If the fundamental problem is to improve code reuse and reduce need to do copy-paste to write good Dockerfiles then I'm very interested in solving that problem.

xx-apt, etc. (https://github.com/tonistiigi/xx) can be reimplemented as a hook.

I think it is the opposite. xx shows the limitation of this approach. You can't just mask over apt and have magic cross-compilation. User needs to choose what packages make sense for what architecture, and for what stage. Main stage needs to be $BUILDPLATFORM, other args need to be exposed, dependencies need to be split up, compilation result needs to be verified etc. Additionally user is left with a broken behavior as some articles will write that they can "just add a hook", but the repo author has never tested it and it will almost never work. If it works, it likely produces something inefficient or broken in runtime. I do agree though that including xx in your project may be considered a bit hacky atm. and maybe ideally you should be able to include xx directly from source (eg. like it is possible for the helper targets in that repo via bake) or maybe patch it locally. I'd be interested in finding better solutions for these cases if you agree.

Because it affects the history object in OCI Image Config and decreases reproducibility

I don't get the reproducibility part, but history should give a representation of how the image was built. If it does not, then it is incorrect. Latest versions also create provenance attestations, that supercedes history - it is always precise and can't be modified by frontend.