From c76d3607799ed2b89cb798e3810f10d11ae70158 Mon Sep 17 00:00:00 2001 From: Jack Luo Date: Thu, 23 Jul 2026 11:26:28 -0400 Subject: [PATCH 1/2] feat(docker): Add ca-trust library for propagating host CA trust into containerized builds. Ported from clp-plugin-presto-connector's tools/build-packages/internal/ca-trust with README example paths updated to the consumer-facing submodule path. Co-Authored-By: Claude Fable 5 --- exports/docker/ca-trust/README.md | 58 +++++++ exports/docker/ca-trust/container.sh | 76 +++++++++ .../ca-trust/generators/java-pkcs12/README.md | 44 ++++++ .../generators/java-pkcs12/generate.sh | 118 ++++++++++++++ exports/docker/ca-trust/host.sh | 145 ++++++++++++++++++ 5 files changed, 441 insertions(+) create mode 100644 exports/docker/ca-trust/README.md create mode 100644 exports/docker/ca-trust/container.sh create mode 100644 exports/docker/ca-trust/generators/java-pkcs12/README.md create mode 100755 exports/docker/ca-trust/generators/java-pkcs12/generate.sh create mode 100644 exports/docker/ca-trust/host.sh diff --git a/exports/docker/ca-trust/README.md b/exports/docker/ca-trust/README.md new file mode 100644 index 0000000..b0c85f8 --- /dev/null +++ b/exports/docker/ca-trust/README.md @@ -0,0 +1,58 @@ +# CA trust + +A reusable library for propagating the host's trusted certificates into a containerized build behind a corporate TLS gateway, without installing them in an image or persisting them in layers, caches, or artifacts. + +## Quick start + +On the host, stage the PEM CA bundle. Bind-mount it writable into the build container, and in the container set `CA_TRUST_DIR` to that mount point (plus `CA_TRUST_JVM=1` for a JVM build), then source `container.sh`. The Java PKCS#12 trust store is generated in-container into the same directory, alongside the bundle. + +```bash +# Host side +source tools/yscope-dev-utils/exports/docker/ca-trust/host.sh + +CA_TRUST_HOST_DIR="$(mktemp -d)" +trap 'rm -rf "${CA_TRUST_HOST_DIR}"' EXIT +stage_host_ca_bundle "${CA_TRUST_HOST_DIR}" # creates ${CA_TRUST_HOST_DIR}/ca-bundle.pem, read-only + +docker run --rm \ + --mount "type=bind,src=${CA_TRUST_HOST_DIR},dst=${CA_TRUST_CONTAINER_DIR}" \ + --env "CA_TRUST_DIR=${CA_TRUST_CONTAINER_DIR}" \ + --env "CA_TRUST_JVM=1" \ + --env MAVEN_OPTS \ + bash -c ' + source /repo/tools/yscope-dev-utils/exports/docker/ca-trust/container.sh + # ... run the build; curl/git/pip/Maven now use the host CAs + ' +``` + +Only the PEM bundle is staged on the host; the Java trust store is generated inside the container, which already has a JDK for the build. The generated store is written to the same writable bind mount (not the container's writable overlay), so it never lands on the overlay and cannot be retained by `docker commit`. The caller cleans up the staging directory. + +## Host API (`host.sh`) + +| Function | Args | Effect | +|------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `stage_host_ca_bundle` | `` | Writes `/${CA_TRUST_BUNDLE_FILENAME}` (`0444`). Uses `SSL_CERT_FILE` when set, else searches common Linux CA-bundle locations; creates an empty file if none is found. | + +Constants: `CA_TRUST_BUNDLE_FILENAME` (`ca-bundle.pem`) and `CA_TRUST_CONTAINER_DIR` (`/run/ca-trust`, the in-container mount point for the staged trust directory, passed as `CA_TRUST_DIR`). + +## Container API (`container.sh`) + +Source it in the container after setting `CA_TRUST_DIR`; set `CA_TRUST_JVM=1` as well if the build runs on a JVM (Maven, Gradle, ...) that needs its trust store configured: + +```bash +CA_TRUST_DIR=/trusted +CA_TRUST_JVM=1 +source tools/yscope-dev-utils/exports/docker/ca-trust/container.sh +``` + +It reads `ca-bundle.pem` from `CA_TRUST_DIR`. When the bundle is non-empty it exports `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`, `PIP_CERT`, `REQUESTS_CA_BUNDLE`, and `SSL_CERT_FILE`. When `CA_TRUST_JVM` is set, the bundle is non-empty, and `keytool` is available, it also generates a PKCS#12 trust store from the bundle via `generators/java-pkcs12/generate.sh`, writes it to `${CA_TRUST_DIR}/truststore.p12`, and appends `-Djavax.net.ssl.trustStore*` to `MAVEN_OPTS` (preserving any caller-supplied value). + +**Persistence contract:** `CA_TRUST_DIR` must be a writable host bind-mount or tmpfs, not the container's writable overlay. `container.sh` verifies this with `findmnt` and refuses (with an error) to write to the overlay, since a file there would be retained by `docker commit`. If `findmnt` is unavailable it warns but proceeds. A generation failure errors. + +JVM trust configuration is opt-in via `CA_TRUST_JVM`, since not every caller runs on a JVM; it's also skipped when the bundle is empty or `keytool` is absent. A no-op when `CA_TRUST_DIR` is unset, so CI builds that don't mount a trust directory are unaffected. + +The caller owns and cleans up the staging directory; the scripts never modify the host or container trust stores, only the staged bundle. The generated PKCS#12 store is a per-build file in the caller's staging directory, removed when the caller cleans up. + +## Extensibility + +Add a backend under `generators/` when a trust format can't consume the PEM bundle directly. Keep host discovery and lifecycle in `host.sh`; keep format-specific conversion in the backend, run in-container. See `generators/java-pkcs12/README.md`. diff --git a/exports/docker/ca-trust/container.sh b/exports/docker/ca-trust/container.sh new file mode 100644 index 0000000..9e059ce --- /dev/null +++ b/exports/docker/ca-trust/container.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash + +# Container-side configuration for CA trust. Source it after setting +# CA_TRUST_DIR to a writable mount of the staged trust directory, which must +# contain ca-bundle.pem. Set CA_TRUST_JVM=1 as well if the build runs on a JVM +# (Maven, Gradle, ...) that needs its trust store configured: a Java PKCS#12 +# trust store is then generated here, inside the container, from the PEM +# bundle using the container's own JDK (keytool) -- no separate generator +# container or host JDK is required -- and written back to CA_TRUST_DIR +# alongside the bundle. +# +# Persistence contract: CA_TRUST_DIR must be a writable host bind-mount (or +# tmpfs), not the container's writable overlay. A file on the overlay is retained +# by `docker commit`; a bind mount is not part of any committed image. This +# script refuses to write to the overlay. + +if [[ -z "${CA_TRUST_DIR:-}" ]]; then + return 0 2>/dev/null || exit 0 +fi + +_ca_trust_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" &>/dev/null && pwd)" +HOST_CA_BUNDLE="${CA_TRUST_DIR}/ca-bundle.pem" + +if [[ -s "${HOST_CA_BUNDLE:-}" ]]; then + export CURL_CA_BUNDLE="${HOST_CA_BUNDLE}" + export GIT_SSL_CAINFO="${HOST_CA_BUNDLE}" + export PIP_CERT="${HOST_CA_BUNDLE}" + export REQUESTS_CA_BUNDLE="${HOST_CA_BUNDLE}" + export SSL_CERT_FILE="${HOST_CA_BUNDLE}" +fi + +# Generate a Java PKCS#12 trust store in-container from the staged PEM bundle and +# point Maven at it. Opt-in via CA_TRUST_JVM=1, since not every caller of this +# library runs on a JVM. Also skipped when the bundle is empty or keytool is +# unavailable, so CI builds without a trust directory and PEM-only staging +# (empty bundle) are unaffected. +if [[ -n "${CA_TRUST_JVM:-}" ]] && [[ -s "${HOST_CA_BUNDLE:-}" ]] && command -v keytool &>/dev/null; then + if ! mkdir -p "${CA_TRUST_DIR}"; then + echo >&2 "ERROR: cannot create Java trust store dir: ${CA_TRUST_DIR}" + return 1 2>/dev/null || exit 1 + fi + + # Refuse to write to the container's writable overlay: a file there is + # retained by `docker commit`, violating the no-persistence invariant. A + # bind mount or tmpfs has its own mount target; the root overlay resolves + # to "/". Warn (but proceed) if findmnt is unavailable to check. + if command -v findmnt &>/dev/null; then + _ca_trust_mount_target="$(findmnt -T "${CA_TRUST_DIR}" -o TARGET -n 2>/dev/null || true)" + if [[ -z "${_ca_trust_mount_target}" || "${_ca_trust_mount_target}" == "/" ]]; then + echo >&2 "ERROR: CA_TRUST_DIR (${CA_TRUST_DIR}) is on the container's writable overlay," + echo >&2 " which docker commit would retain. Mount a writable host directory or tmpfs there." + return 1 2>/dev/null || exit 1 + fi + else + echo >&2 "WARNING: findmnt unavailable; cannot verify CA_TRUST_DIR is off the overlay." + fi + + HOST_CA_JAVA_TRUST_STORE="${CA_TRUST_DIR}/truststore.p12" + if ! bash "${_ca_trust_dir}/generators/java-pkcs12/generate.sh" \ + "${HOST_CA_BUNDLE}" "${HOST_CA_JAVA_TRUST_STORE}"; then + echo >&2 "ERROR: failed to generate Java PKCS#12 trust store from ${HOST_CA_BUNDLE}" + return 1 2>/dev/null || exit 1 + fi + + # Preserve any Maven options supplied by the caller. + _host_ca_maven_opts="${MAVEN_OPTS:-}" + [[ -n "${_host_ca_maven_opts}" ]] && _host_ca_maven_opts="${_host_ca_maven_opts} " + _host_ca_maven_opts="${_host_ca_maven_opts}-Djavax.net.ssl.trustStore=${HOST_CA_JAVA_TRUST_STORE}" + _host_ca_maven_opts="${_host_ca_maven_opts} -Djavax.net.ssl.trustStoreType=PKCS12" + # The store contains only public certificates; this is an integrity password, not a secret. + _host_ca_maven_opts="${_host_ca_maven_opts} -Djavax.net.ssl.trustStorePassword=changeit" + export MAVEN_OPTS="${_host_ca_maven_opts}" + unset _host_ca_maven_opts _ca_trust_mount_target +fi + +unset _ca_trust_dir diff --git a/exports/docker/ca-trust/generators/java-pkcs12/README.md b/exports/docker/ca-trust/generators/java-pkcs12/README.md new file mode 100644 index 0000000..774dd16 --- /dev/null +++ b/exports/docker/ca-trust/generators/java-pkcs12/README.md @@ -0,0 +1,44 @@ +# Java PKCS#12 generator + +A `generators/` backend that produces a Java PKCS#12 trust store from a PEM CA +bundle. Invoked by `container.sh` inside the build container; also runnable +directly. + +## Usage + +```bash +./tools/yscope-dev-utils/exports/docker/ca-trust/generators/java-pkcs12/generate.sh \ + +``` + +It needs a JDK: it locates `keytool` via `JAVA_HOME`, falling back to `keytool` +on `PATH`, then reads the JDK's base trust store (`jssecacerts` if present, else +`cacerts`). Given the inputs, it: + +1. Copies the base JDK trust store into a new PKCS#12 store via + `keytool -importkeystore`, preserving the standard Mozilla CA set alongside + the host's corporate CAs. +2. Imports each certificate from the PEM bundle with `keytool -importcert`, + splitting the bundle first (keytool reads only the first certificate from a + multi-cert PEM file) and using unique `host-ca-` aliases. Certificates + already present under any alias are silently skipped. +3. Writes the result to the output path (store password `changeit`, an + integrity password for public certificates, not a secret). + +`container.sh` runs this and feeds the result to Maven via +`-Djavax.net.ssl.trustStore= -Djavax.net.ssl.trustStoreType=PKCS12 +-Djavax.net.ssl.trustStorePassword=changeit`, appended to `MAVEN_OPTS`, avoiding +edits to the JDK's installed `cacerts`. + +## Notes + +The generator runs in the build container, which already has a JDK for the +build, so no separate generator container or host JDK is required. The output +store is written to the caller-supplied output path, which `container.sh` places +in `CA_TRUST_DIR` -- a writable host bind-mount, not the container's writable +overlay -- so it never enters the image, caches, packages, or layers and is +cleaned up by the caller. + +## Files + +- `generate.sh` -- validates inputs, locates the JDK trust store, runs keytool. \ No newline at end of file diff --git a/exports/docker/ca-trust/generators/java-pkcs12/generate.sh b/exports/docker/ca-trust/generators/java-pkcs12/generate.sh new file mode 100755 index 0000000..0012389 --- /dev/null +++ b/exports/docker/ca-trust/generators/java-pkcs12/generate.sh @@ -0,0 +1,118 @@ +#!/usr/bin/env bash + +# Generates a Java PKCS#12 trust store from a PEM CA bundle, merging the +# selected JDK's default certificates with the bundle's certificates. +# +# Runs inside the build container, which already provides a JDK (keytool + +# cacerts); no separate generator container or host JDK is required. + +set -o errexit +set -o nounset +set -o pipefail + +if (( $# != 2 )) || [[ -z "$1" || -z "$2" ]]; then + echo >&2 "ERROR: generate.sh requires an input CA bundle and output path" + exit 2 +fi + +input_bundle="$1" +output_trust_store="$2" +if [[ ! -f "${input_bundle}" || ! -r "${input_bundle}" ]]; then + echo >&2 "ERROR: input CA bundle is not a readable regular file: ${input_bundle}" + exit 1 +fi +output_dir="$(dirname "${output_trust_store}")" +if [[ ! -d "${output_dir}" || ! -w "${output_dir}" ]]; then + echo >&2 "ERROR: output directory is not writable: ${output_dir}" + exit 1 +fi +if [[ -e "${output_trust_store}" && ! -f "${output_trust_store}" ]]; then + echo >&2 "ERROR: output path is not a regular file: ${output_trust_store}" + exit 1 +fi + +# Integrity password for a store of public CA certificates; not a secret. +readonly STOREPASS=changeit + +# Locate keytool and the JDK's default trust store. Match Java's trust-store +# lookup order: jssecacerts overrides cacerts. +java_home="${JAVA_HOME:-}" +if [[ -n "${java_home}" ]]; then + keytool="${java_home}/bin/keytool" +else + keytool="$(command -v keytool)" || { + echo >&2 "ERROR: keytool was not found in PATH and JAVA_HOME is unset" + exit 1 + } + keytool="$(readlink -f "${keytool}")" + java_home="${keytool%/bin/keytool}" +fi +if [[ ! -x "${keytool}" ]]; then + echo >&2 "ERROR: keytool is not executable: ${keytool}" + exit 1 +fi + +java_security_dir="${java_home}/lib/security" +base_java_trust_store="${java_security_dir}/cacerts" +if [[ -f "${java_security_dir}/jssecacerts" && -s "${java_security_dir}/jssecacerts" ]]; then + base_java_trust_store="${java_security_dir}/jssecacerts" +fi +if [[ ! -f "${base_java_trust_store}" || ! -r "${base_java_trust_store}" \ + || ! -s "${base_java_trust_store}" ]]; then + echo >&2 "ERROR: JDK default trust store is not readable: ${base_java_trust_store}" + exit 1 +fi + +# Append each certificate from the PEM bundle. keytool -importcert reads only +# the first certificate from a multi-cert PEM file, so split the bundle into +# per-cert buffers and import each under a unique alias. +work_dir="$(mktemp -d)" +trap 'rm -rf "${work_dir}"' EXIT + +# Start from a copy of the JDK's default trust store as PKCS#12. This keeps the +# standard Mozilla CA set alongside the host's corporate CAs, so downloads to +# public mirrors (not behind the corporate gateway) still verify. keytool prints +# one progress line per entry to stderr; capture it so success is quiet but a +# failure still surfaces the cause. +if ! "${keytool}" -importkeystore -noprompt \ + -srckeystore "${base_java_trust_store}" -srcstoretype JKS -srcstorepass "${STOREPASS}" \ + -destkeystore "${output_trust_store}" -deststoretype PKCS12 -deststorepass "${STOREPASS}" \ + >/dev/null 2>"${work_dir}/import.err"; then + echo >&2 "ERROR: keytool -importkeystore failed:" + cat >&2 "${work_dir}/import.err" + exit 1 +fi + +count=0 +cert_buf="" +cert_file="${work_dir}/cert.pem" +while IFS= read -r line || [[ -n "${line}" ]]; do + cert_buf+="${line}"$'\n' + if [[ "${line}" == "-----END CERTIFICATE-----" ]]; then + printf '%s' "${cert_buf}" > "${cert_file}" + # -noprompt skips the "trust this certificate?" prompt. A certificate + # already present under any alias is silently skipped by keytool, so + # duplicates in the bundle (or shared with cacerts) are harmless. + if ! "${keytool}" -importcert -noprompt \ + -alias "host-ca-${count}" -file "${cert_file}" \ + -keystore "${output_trust_store}" -storetype PKCS12 -storepass "${STOREPASS}" \ + >/dev/null 2>"${work_dir}/import-cert.err"; then + echo >&2 "ERROR: failed to import certificate #${count} from bundle:" + cat >&2 "${work_dir}/import-cert.err" + exit 1 + fi + count=$((count + 1)) + cert_buf="" + fi +done < "${input_bundle}" + +if (( count == 0 )); then + echo >&2 "ERROR: input bundle contains no complete PEM certificates: ${input_bundle}" + exit 1 +fi + +if [[ ! -s "${output_trust_store}" ]]; then + echo >&2 "ERROR: generated trust store is empty: ${output_trust_store}" + exit 1 +fi +echo "==> Generated Java PKCS#12 trust store: ${output_trust_store} (${count} bundle certificate(s) processed)" diff --git a/exports/docker/ca-trust/host.sh b/exports/docker/ca-trust/host.sh new file mode 100644 index 0000000..5d50d9e --- /dev/null +++ b/exports/docker/ca-trust/host.sh @@ -0,0 +1,145 @@ +#!/usr/bin/env bash + +# Host-side CA discovery and staging shared by Docker build and run workflows. + +if [[ "${_CA_TRUST_HOST_SH_LOADED:-}" == "1" ]]; then + return 0 +fi +readonly _CA_TRUST_HOST_SH_LOADED=1 + +# Conventional staged filename for the host CA bundle. container.sh reads it +# from CA_TRUST_DIR by this name (HOST_CA_BUNDLE) and generates the Java +# PKCS#12 trust store in-container from it. +readonly CA_TRUST_BUNDLE_FILENAME="ca-bundle.pem" + +# In-container mount point for the staged trust directory. Callers bind-mount +# the staging directory here (writable) and pass it as CA_TRUST_DIR so +# container.sh consumes the staged PEM bundle and writes the generated Java +# PKCS#12 trust store back into it. Kept in host.sh so the path is defined +# once on the host side rather than hardcoded by each caller. +readonly CA_TRUST_CONTAINER_DIR="/run/ca-trust" + +# Copies to , dropping any certificate whose validity period has +# already ended. A stale corporate CA bundle otherwise gets propagated +# verbatim into CURL_CA_BUNDLE, where OpenSSL (unlike macOS's SecureTransport) +# treats the file as the exclusive trust store: one expired cert anywhere in +# it is enough to break TLS verification for any download whose chain happens +# to rely on it, even though the destination server's own certificate is +# fine. Falls back to a plain copy if openssl isn't on the host, so this never +# becomes a new hard dependency. +# +# Args: +_stage_ca_bundle_without_expired_certs() { + local src="$1" dest="$2" + if ! command -v openssl &>/dev/null; then + cp "${src}" "${dest}" + return + fi + + local total=0 dropped=0 + local cert="" line + : > "${dest}" + while IFS= read -r line || [[ -n "${line}" ]]; do + cert+="${line}"$'\n' + if [[ "${line}" == "-----END CERTIFICATE-----" ]]; then + total=$((total + 1)) + if printf '%s' "${cert}" | openssl x509 -noout -checkend 0 &>/dev/null; then + printf '%s' "${cert}" >> "${dest}" + else + dropped=$((dropped + 1)) + fi + cert="" + fi + done < "${src}" + + if (( dropped > 0 )); then + echo >&2 "==> Dropped ${dropped}/${total} expired certificate(s) from host CA bundle" + fi +} + +# Stages the host CA bundle at /${CA_TRUST_BUNDLE_FILENAME} for a +# temporary Docker mount. Creates an empty file when the host has no CA bundle; +# returns nonzero only on an error. +# +# Args: +stage_host_ca_bundle() { + if (( $# != 1 )) || [[ -z "$1" ]]; then + echo >&2 "ERROR: stage_host_ca_bundle requires a trust directory" + return 2 + fi + local trust_dir="$1" + if [[ -L "${trust_dir}" || ( -e "${trust_dir}" && ! -d "${trust_dir}" ) ]]; then + echo >&2 "ERROR: stage_host_ca_bundle target is not a directory: ${trust_dir}" + return 1 + fi + if ! mkdir -p "${trust_dir}"; then + echo >&2 "ERROR: failed to create trust directory: ${trust_dir}" + return 1 + fi + trust_dir="$(cd "${trust_dir}" &>/dev/null && pwd)" || return + local dest="${trust_dir}/${CA_TRUST_BUNDLE_FILENAME}" + if [[ -L "${dest}" || ( -e "${dest}" && ! -f "${dest}" ) ]]; then + echo >&2 "ERROR: host CA bundle destination is not a regular file: ${dest}" + return 1 + fi + local source_path="" + local candidates=() + + if [[ -n "${SSL_CERT_FILE:-}" ]]; then + if [[ ! -f "${SSL_CERT_FILE}" || ! -s "${SSL_CERT_FILE}" ]]; then + echo >&2 "ERROR: SSL_CERT_FILE is not a nonempty regular file: ${SSL_CERT_FILE}" + return 1 + fi + candidates=("${SSL_CERT_FILE}") + else + candidates=( + /etc/ssl/certs/ca-certificates.crt + /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem + /etc/pki/tls/certs/ca-bundle.crt + /etc/ssl/ca-bundle.pem + /etc/pki/tls/cacert.pem + /etc/ssl/cert.pem + ) + fi + + local candidate + for candidate in "${candidates[@]}"; do + if [[ -f "${candidate}" && -s "${candidate}" ]]; then + source_path="${candidate}" + break + fi + done + + if [[ -n "${source_path}" && -e "${dest}" && "${source_path}" -ef "${dest}" ]]; then + echo >&2 "ERROR: host CA bundle source and destination must differ: ${dest}" + return 1 + fi + + local staged_bundle + if ! staged_bundle="$(mktemp "${trust_dir}/.ca-bundle.XXXXXX")"; then + echo >&2 "ERROR: failed to create temporary host CA bundle in: ${trust_dir}" + return 1 + fi + if [[ -n "${source_path}" ]]; then + echo >&2 "==> Staging host CA bundle: ${source_path} -> ${dest}" + if ! _stage_ca_bundle_without_expired_certs "${source_path}" "${staged_bundle}"; then + rm -f "${staged_bundle}" + echo >&2 "ERROR: failed to stage host CA bundle: ${source_path}" + return 1 + fi + else + echo >&2 "==> No host CA bundle found; continuing without host CA context." + fi + + # BuildKit and runtime containers consume the staged bundle read-only. + if ! chmod 0444 "${staged_bundle}"; then + rm -f "${staged_bundle}" + echo >&2 "ERROR: failed to set host CA bundle permissions: ${dest}" + return 1 + fi + if ! mv -f "${staged_bundle}" "${dest}"; then + rm -f "${staged_bundle}" + echo >&2 "ERROR: failed to replace host CA bundle: ${dest}" + return 1 + fi +} From b20ea7a7b3e6491e9c2a8a95eb82c59f0d9259ca Mon Sep 17 00:00:00 2001 From: Jack Luo Date: Thu, 23 Jul 2026 11:41:42 -0400 Subject: [PATCH 2/2] docs(docker): Polish ca-trust READMEs and link the library from the docs index. Explain the /repo mount and submodule-path assumptions in the quick start, document expired-cert filtering, and tighten both READMEs to the repo's markdownlint style. Co-Authored-By: Claude Fable 5 --- docs/index.md | 1 + exports/docker/ca-trust/README.md | 59 ++++++++++++------- exports/docker/ca-trust/container.sh | 20 ++++++- .../ca-trust/generators/java-pkcs12/README.md | 44 +++----------- exports/docker/ca-trust/host.sh | 5 +- 5 files changed, 70 insertions(+), 59 deletions(-) diff --git a/docs/index.md b/docs/index.md index b923266..184dc80 100644 --- a/docs/index.md +++ b/docs/index.md @@ -18,4 +18,5 @@ To use the repo's artifacts in your project: For language/tool-specific guides, see: +* [CA trust for containerized builds](../exports/docker/ca-trust/README.md) * [Linting for C++ projects](lint-tools-cpp.md) diff --git a/exports/docker/ca-trust/README.md b/exports/docker/ca-trust/README.md index b0c85f8..e1835d7 100644 --- a/exports/docker/ca-trust/README.md +++ b/exports/docker/ca-trust/README.md @@ -1,10 +1,20 @@ # CA trust -A reusable library for propagating the host's trusted certificates into a containerized build behind a corporate TLS gateway, without installing them in an image or persisting them in layers, caches, or artifacts. +A library for propagating the host's trusted CA certificates into containerized builds that run behind a TLS-inspecting (e.g., corporate) gateway. Trust is wired up through environment variables and a bind-mounted staging directory, so certificates are never installed into an image or persisted in layers, caches, or artifacts. + +The examples below assume the consuming project has this repo as a submodule at `tools/yscope-dev-utils` (see the [usage docs](../../../docs/index.md#usage)) and mounts itself at `/repo` inside the build container; adjust the paths to your layout. + +## Requirements + +* Host: `bash`, plus a container runtime that supports bind mounts (the examples use Docker). + * `openssl` (optional): used to drop expired certificates during staging; without it, the bundle is copied as-is. +* Container: `bash`. + * `findmnt` (optional): used to verify `CA_TRUST_DIR` is not on the container's writable overlay; without it, a warning is printed and the build proceeds. + * A JDK providing `keytool` (JVM builds only): used to generate the PKCS#12 trust store; without it, JVM trust setup is skipped. ## Quick start -On the host, stage the PEM CA bundle. Bind-mount it writable into the build container, and in the container set `CA_TRUST_DIR` to that mount point (plus `CA_TRUST_JVM=1` for a JVM build), then source `container.sh`. The Java PKCS#12 trust store is generated in-container into the same directory, alongside the bundle. +On the host, stage the CA bundle into a temporary directory. Bind-mount that directory (writable) into the container, point `CA_TRUST_DIR` at the mount, and source `container.sh` before running the build: ```bash # Host side @@ -12,47 +22,54 @@ source tools/yscope-dev-utils/exports/docker/ca-trust/host.sh CA_TRUST_HOST_DIR="$(mktemp -d)" trap 'rm -rf "${CA_TRUST_HOST_DIR}"' EXIT -stage_host_ca_bundle "${CA_TRUST_HOST_DIR}" # creates ${CA_TRUST_HOST_DIR}/ca-bundle.pem, read-only + +# Creates ${CA_TRUST_HOST_DIR}/ca-bundle.pem (read-only). Check the status: running +# the build without host CA trust is the failure this library exists to avoid. +stage_host_ca_bundle "${CA_TRUST_HOST_DIR}" || exit 1 docker run --rm \ + --mount "type=bind,src=${PWD},dst=/repo" \ --mount "type=bind,src=${CA_TRUST_HOST_DIR},dst=${CA_TRUST_CONTAINER_DIR}" \ --env "CA_TRUST_DIR=${CA_TRUST_CONTAINER_DIR}" \ --env "CA_TRUST_JVM=1" \ --env MAVEN_OPTS \ - bash -c ' + \ + bash -c ' source /repo/tools/yscope-dev-utils/exports/docker/ca-trust/container.sh - # ... run the build; curl/git/pip/Maven now use the host CAs + # Run the build; curl, git, pip, and Maven now trust the host CAs. ' ``` -Only the PEM bundle is staged on the host; the Java trust store is generated inside the container, which already has a JDK for the build. The generated store is written to the same writable bind mount (not the container's writable overlay), so it never lands on the overlay and cannot be retained by `docker commit`. The caller cleans up the staging directory. +`CA_TRUST_JVM=1` and `--env MAVEN_OPTS` are only needed for JVM builds; see [JVM builds](#jvm-builds). ## Host API (`host.sh`) -| Function | Args | Effect | -|------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `stage_host_ca_bundle` | `` | Writes `/${CA_TRUST_BUNDLE_FILENAME}` (`0444`). Uses `SSL_CERT_FILE` when set, else searches common Linux CA-bundle locations; creates an empty file if none is found. | +`stage_host_ca_bundle ` writes the host's CA bundle to `/${CA_TRUST_BUNDLE_FILENAME}` (read-only, `0444`): + +* The bundle is taken from `SSL_CERT_FILE` when set; otherwise, common Linux CA-bundle locations are searched. If none is found (e.g., on macOS without `SSL_CERT_FILE`), an empty file is created and the build proceeds without host CA context. +* Expired certificates are dropped during staging (when `openssl` is available on the host), since a single expired certificate in a bundle can break TLS verification for otherwise-valid chains. -Constants: `CA_TRUST_BUNDLE_FILENAME` (`ca-bundle.pem`) and `CA_TRUST_CONTAINER_DIR` (`/run/ca-trust`, the in-container mount point for the staged trust directory, passed as `CA_TRUST_DIR`). +Constants: + +* `CA_TRUST_BUNDLE_FILENAME` (`ca-bundle.pem`): the staged bundle's filename; `container.sh` reads it from `CA_TRUST_DIR` by this name. +* `CA_TRUST_CONTAINER_DIR` (`/run/ca-trust`): the conventional in-container mount point for the staged trust directory, passed to the container as `CA_TRUST_DIR`. + +The caller owns the staging directory and cleans it up (e.g., with `trap`, as above). The scripts never modify the host's or the container's installed trust stores. ## Container API (`container.sh`) -Source it in the container after setting `CA_TRUST_DIR`; set `CA_TRUST_JVM=1` as well if the build runs on a JVM (Maven, Gradle, ...) that needs its trust store configured: +Source it after setting `CA_TRUST_DIR` to the (writable) mount of the staged trust directory. It's a no-op when `CA_TRUST_DIR` is unset, so builds that don't mount a trust directory are unaffected. -```bash -CA_TRUST_DIR=/trusted -CA_TRUST_JVM=1 -source tools/yscope-dev-utils/exports/docker/ca-trust/container.sh -``` +When the staged bundle is non-empty, it exports `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`, `PIP_CERT`, `REQUESTS_CA_BUNDLE`, and `SSL_CERT_FILE`, covering most TLS clients used in builds. -It reads `ca-bundle.pem` from `CA_TRUST_DIR`. When the bundle is non-empty it exports `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`, `PIP_CERT`, `REQUESTS_CA_BUNDLE`, and `SSL_CERT_FILE`. When `CA_TRUST_JVM` is set, the bundle is non-empty, and `keytool` is available, it also generates a PKCS#12 trust store from the bundle via `generators/java-pkcs12/generate.sh`, writes it to `${CA_TRUST_DIR}/truststore.p12`, and appends `-Djavax.net.ssl.trustStore*` to `MAVEN_OPTS` (preserving any caller-supplied value). +### JVM builds -**Persistence contract:** `CA_TRUST_DIR` must be a writable host bind-mount or tmpfs, not the container's writable overlay. `container.sh` verifies this with `findmnt` and refuses (with an error) to write to the overlay, since a file there would be retained by `docker commit`. If `findmnt` is unavailable it warns but proceeds. A generation failure errors. +JVM tools (Maven, Gradle, ...) don't read the environment variables above, so JVM support is opt-in via `CA_TRUST_JVM=1`. When it's set, the bundle is non-empty, and `keytool` is available, `container.sh` uses the container's own JDK to generate a PKCS#12 trust store from the bundle at `${CA_TRUST_DIR}/truststore.p12`, then appends the corresponding `-Djavax.net.ssl.trustStore*` options to `MAVEN_OPTS`, preserving any caller-supplied value (forward `MAVEN_OPTS` into the container, as in the quick start). A generation failure is an error. See [generators/java-pkcs12](generators/java-pkcs12/README.md) for details. -JVM trust configuration is opt-in via `CA_TRUST_JVM`, since not every caller runs on a JVM; it's also skipped when the bundle is empty or `keytool` is absent. A no-op when `CA_TRUST_DIR` is unset, so CI builds that don't mount a trust directory are unaffected. +## Persistence contract -The caller owns and cleans up the staging directory; the scripts never modify the host or container trust stores, only the staged bundle. The generated PKCS#12 store is a per-build file in the caller's staging directory, removed when the caller cleans up. +`CA_TRUST_DIR` must be a writable host bind-mount or tmpfs, not the container's writable overlay: a file on the overlay would be retained by `docker commit`, while a bind mount is not part of any committed image. `container.sh` verifies this with `findmnt` and refuses to write to the overlay; if `findmnt` is unavailable, it warns and proceeds. All staged and generated files live in the caller's staging directory and disappear when the caller cleans it up. ## Extensibility -Add a backend under `generators/` when a trust format can't consume the PEM bundle directly. Keep host discovery and lifecycle in `host.sh`; keep format-specific conversion in the backend, run in-container. See `generators/java-pkcs12/README.md`. +Add a backend under `generators/` when a trust format can't consume the PEM bundle directly. Keep host discovery and lifecycle in `host.sh`; keep format-specific conversion in the backend, run in-container. See [generators/java-pkcs12](generators/java-pkcs12/README.md) as a template. diff --git a/exports/docker/ca-trust/container.sh b/exports/docker/ca-trust/container.sh index 9e059ce..254b944 100644 --- a/exports/docker/ca-trust/container.sh +++ b/exports/docker/ca-trust/container.sh @@ -19,6 +19,12 @@ if [[ -z "${CA_TRUST_DIR:-}" ]]; then fi _ca_trust_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" &>/dev/null && pwd)" +# Spelled out rather than read from host.sh's CA_TRUST_BUNDLE_FILENAME: host.sh +# is the host's half of the library and isn't guaranteed to be beside this file +# in the container. Renaming the bundle means changing both. +# +# Internal to this file, like the other bare names here; the caller gets the +# environment variables exported below, and all of these are unset at the end. HOST_CA_BUNDLE="${CA_TRUST_DIR}/ca-bundle.pem" if [[ -s "${HOST_CA_BUNDLE:-}" ]]; then @@ -29,6 +35,18 @@ if [[ -s "${HOST_CA_BUNDLE:-}" ]]; then export SSL_CERT_FILE="${HOST_CA_BUNDLE}" fi +# Say why when an explicit opt-in does nothing. Silence here surfaces much later +# as a PKIX path-building error inside the JVM build, far from the cause. +if [[ -n "${CA_TRUST_JVM:-}" ]]; then + if [[ ! -s "${HOST_CA_BUNDLE:-}" ]]; then + echo >&2 "WARNING: CA_TRUST_JVM is set but ${HOST_CA_BUNDLE} is empty;" \ + "skipping the JVM trust store." + elif ! command -v keytool &>/dev/null; then + echo >&2 "WARNING: CA_TRUST_JVM is set but keytool isn't on PATH;" \ + "skipping the JVM trust store." + fi +fi + # Generate a Java PKCS#12 trust store in-container from the staged PEM bundle and # point Maven at it. Opt-in via CA_TRUST_JVM=1, since not every caller of this # library runs on a JVM. Also skipped when the bundle is empty or keytool is @@ -73,4 +91,4 @@ if [[ -n "${CA_TRUST_JVM:-}" ]] && [[ -s "${HOST_CA_BUNDLE:-}" ]] && command -v unset _host_ca_maven_opts _ca_trust_mount_target fi -unset _ca_trust_dir +unset _ca_trust_dir HOST_CA_BUNDLE HOST_CA_JAVA_TRUST_STORE diff --git a/exports/docker/ca-trust/generators/java-pkcs12/README.md b/exports/docker/ca-trust/generators/java-pkcs12/README.md index 774dd16..e20bcde 100644 --- a/exports/docker/ca-trust/generators/java-pkcs12/README.md +++ b/exports/docker/ca-trust/generators/java-pkcs12/README.md @@ -1,44 +1,16 @@ # Java PKCS#12 generator -A `generators/` backend that produces a Java PKCS#12 trust store from a PEM CA -bundle. Invoked by `container.sh` inside the build container; also runnable -directly. - -## Usage +A `generators/` backend that produces a Java PKCS#12 trust store from a PEM CA bundle. Invoked by `container.sh` inside the build container; also runnable directly: ```bash -./tools/yscope-dev-utils/exports/docker/ca-trust/generators/java-pkcs12/generate.sh \ - +tools/yscope-dev-utils/exports/docker/ca-trust/generators/java-pkcs12/generate.sh \ + ``` -It needs a JDK: it locates `keytool` via `JAVA_HOME`, falling back to `keytool` -on `PATH`, then reads the JDK's base trust store (`jssecacerts` if present, else -`cacerts`). Given the inputs, it: - -1. Copies the base JDK trust store into a new PKCS#12 store via - `keytool -importkeystore`, preserving the standard Mozilla CA set alongside - the host's corporate CAs. -2. Imports each certificate from the PEM bundle with `keytool -importcert`, - splitting the bundle first (keytool reads only the first certificate from a - multi-cert PEM file) and using unique `host-ca-` aliases. Certificates - already present under any alias are silently skipped. -3. Writes the result to the output path (store password `changeit`, an - integrity password for public certificates, not a secret). - -`container.sh` runs this and feeds the result to Maven via -`-Djavax.net.ssl.trustStore= -Djavax.net.ssl.trustStoreType=PKCS12 --Djavax.net.ssl.trustStorePassword=changeit`, appended to `MAVEN_OPTS`, avoiding -edits to the JDK's installed `cacerts`. - -## Notes - -The generator runs in the build container, which already has a JDK for the -build, so no separate generator container or host JDK is required. The output -store is written to the caller-supplied output path, which `container.sh` places -in `CA_TRUST_DIR` -- a writable host bind-mount, not the container's writable -overlay -- so it never enters the image, caches, packages, or layers and is -cleaned up by the caller. +It requires a JDK (`keytool` is located via `JAVA_HOME`, falling back to `PATH`); the build container already has one for the build, so no separate generator container or host JDK is needed. Given the inputs, it: -## Files +1. Copies the JDK's base trust store (`jssecacerts` if present, else `cacerts`) into a new PKCS#12 store, keeping the standard public CA set alongside the bundle's CAs so downloads from hosts not behind the gateway still verify. +2. Imports each certificate from the PEM bundle under a unique `host-ca-` alias, splitting the bundle first since `keytool -importcert` reads only the first certificate of a multi-cert file. Certificates already present in the store are silently skipped. +3. Writes the store to the output path with password `changeit` (an integrity password for public certificates, not a secret). -- `generate.sh` -- validates inputs, locates the JDK trust store, runs keytool. \ No newline at end of file +`container.sh` points the JVM at the result via `-Djavax.net.ssl.trustStore*` options appended to `MAVEN_OPTS`, avoiding edits to the JDK's installed `cacerts`. The store is written to the caller-supplied output path -- for `container.sh`, inside `CA_TRUST_DIR`, a writable bind mount rather than the container's overlay -- so it never enters an image, cache, or artifact, and is removed when the caller cleans up the staging directory. diff --git a/exports/docker/ca-trust/host.sh b/exports/docker/ca-trust/host.sh index 5d50d9e..3ca009c 100644 --- a/exports/docker/ca-trust/host.sh +++ b/exports/docker/ca-trust/host.sh @@ -76,7 +76,10 @@ stage_host_ca_bundle() { echo >&2 "ERROR: failed to create trust directory: ${trust_dir}" return 1 fi - trust_dir="$(cd "${trust_dir}" &>/dev/null && pwd)" || return + if ! trust_dir="$(cd "${trust_dir}" &>/dev/null && pwd)"; then + echo >&2 "ERROR: failed to resolve trust directory: $1" + return 1 + fi local dest="${trust_dir}/${CA_TRUST_BUNDLE_FILENAME}" if [[ -L "${dest}" || ( -e "${dest}" && ! -f "${dest}" ) ]]; then echo >&2 "ERROR: host CA bundle destination is not a regular file: ${dest}"