Add notebook image CI build (mdx2:1.0.2-1) and remove broken docker job

[Abdelsalam-Abbas]

Summary

Adds a CI build for diffuseproject/mdx2-jhub:1.0.2-1, the JupyterHub singleuser image used by the Diffuse SSH gateway. Reproduces the existing :test image's scientific stack via a captured conda lockfile, pins the mdx2 source to the same commit :test was built from, and adds openssh-client, openssh-sftp-server, and rsync so scp, sftp, and rsync work through the gateway.

Also removes the pre-existing broken docker: job from the workflow, strips dead weight from the new Dockerfile, and adds workflow safety guards to prevent silent tag overwrites.

Why

ssh into a JupyterHub notebook pod via the Diffuse SSH gateway works (shipped via webapp PRs #287–#294). But scp, rsync, and sftp fail because they exec the corresponding binaries inside the pod, and the running :test image has none of them:

$ kubectl -n jupyterhub exec jupyter-<user> -- sh -c "which scp; which rsync; ls /usr/lib/openssh/sftp-server"
which scp:           (empty)
ls /usr/lib/openssh/sftp-server: No such file or directory
type scp:            scp: not found  (exit 127)

Modern OpenSSH (≥9.0) defaults scp to the SFTP subsystem, which is why even basic scp started failing.

Approach

The previous :test image (built 2026-02-26) was produced from a now-deleted feature branch (feat/jupyterhub-singleuser, PR #56) by docker build on someone's laptop. Its source Dockerfile was never committed anywhere. To rebuild faithfully:

1. Conda env via explicit lockfile (jhub-env.lock, 385 packages from conda-forge). Captured with micromamba env export --explicit -n mdx2-dev from the live :test pod, so the new image gets byte-equivalent DIALS 3.24.1, dxtbx 3.24.1, hdf5 1.14.3, Python 3.10.19, and 381 other packages — same build hashes, same versions.
2. mdx2 source pinned to 327bf6e1541e3e0b63a22c8aff100b92c4aa6e39 — PR #56's head commit. mdx2/VERSION = 1.0.2 and the package directory matches the live pod exactly (including the absence of io.py, which only exists on main). Reachable via refs/pull/56/head only since the branch was deleted; the Dockerfile fetches that ref explicitly.
3. Pip versions pinned to match the live pod: jupyterhub==5.4.3 and jupyter-vscode-proxy==0.7.
4. Apt additions: openssh-client, openssh-sftp-server, rsync.

The tag 1.0.2-1 follows Debian-revision style: upstream 1.0.2 (mdx2 version) + -1 (our first build with this upstream).

Future upgrade story

The lockfile decouples mdx2 source upgrades from scientific stack upgrades:

Shape	What changes	Tag bump	Trigger
Source-only	MDX2_COMMIT env var in workflow	1.0.3-1, 1.0.4-1, ...	mdx2 ships a new release
Stack-only	Regenerate jhub-env.lock	1.0.2-2, 1.0.2-3, ...	DIALS bump, security patches, quarterly refresh
Both	Both files	<new-mdx2>-1	New mdx2 needs new deps

Lockfile regeneration: spin up a temp container, micromamba env export --explicit -n mdx2-dev > jhub-env.lock, commit. Bump IMAGE_TAG env var in the workflow at the same time — the immutability check will refuse to overwrite the existing tag if you forget.

Step-by-step recipes for each scenario (with concrete commands, smoke tests, rollback procedure, and common pitfalls): see docs/upgrading-the-jhub-image.md.

Workflow safety

The workflow has explicit guards against accidental tag overwrites and unnecessary builds:

• Workflow-level env: IMAGE_NAME, IMAGE_TAG, MDX2_COMMIT are hoisted to a single review surface at the top of the workflow. Bumping a version is a single diff line.
• Pre-push immutability check: before pushing on main, the workflow runs docker manifest inspect against Dockerhub. If ${IMAGE_NAME}:${IMAGE_TAG} already exists, the workflow fails with an explicit error message telling the maintainer to bump IMAGE_TAG. Prevents silent overwrites of published-and-deployed tags.
• Path filter on push trigger: push: to main only fires when Dockerfile.jhub, jhub-env.lock, or the workflow itself changes. Docs-only PRs that get merged don't trigger an unnecessary rebuild. PR-CI stays unfiltered for predictable required-status-check behavior.
• Least-privilege permissions: workflow declares contents: read, actions: write (only the latter for type=gha cache writes). No packages: write since we don't push to ghcr.

Files changed

• New: Dockerfile.jhub (71 lines). 3-stage build: mambaorg/micromamba:1.5.5 (micromamba binary) → debian:stable-slim (mdx2 git clone, isolated stage so git stays out of final image) → debian:stable-slim (final, with conda env from lockfile + mdx2 editable + pip pins + ssh tooling).
• New: jhub-env.lock (389 lines). Conda explicit lockfile; do not edit by hand.
• New: docs/upgrading-the-jhub-image.md (304 lines). Operational runbook for the three upgrade scenarios (mdx2 source bump, conda env refresh, OS apt additions) with concrete commands, rollback procedure, and common pitfalls.
• Modified: .github/workflows/docker.yml. The pre-existing docker: job is removed. The new notebook: job replaces it with: env-var-driven version, paths-filtered push trigger, conditional login, immutability check, and least-privilege permissions.

Why remove the existing docker: job

It was added 2026-03-25 with two pre-existing bugs that prevented it from ever working: file: dockerfile (lowercase, fatal on Linux runners) and unconditional login (failed every PR before Dockerhub creds existed). It has never successfully pushed an image. The image it would build (.github/Dockerfile, the :latest-flavored standalone Jupyter Lab launcher with CMD jupyter lab) is not used in the Diffuse deployment chain — only mdx2-workflows/Dockerfile itself references :1.0.0 as a base image, and that base already exists on Dockerhub from a manual push by jlee in March 2026.

.github/Dockerfile itself is kept around as a reference if anyone wants to revive :1.0.0 automation in a focused follow-up PR.

Pre-merge checklist

• Maintainer adds repo-level secrets:
  • vars.DOCKERHUB_USERNAME (verified working — login step succeeds in CI)
  • secrets.DOCKERHUB_TOKEN (same)
• PR's CI build of Dockerfile.jhub passes (no push, just verifies the build succeeds in a clean ubuntu-latest runner)
• On merge, main's CI run pushes diffuseproject/mdx2-jhub:1.0.2-1 to Dockerhub

Verification done locally

Check	Result
docker buildx build	succeeded; image at sha256:4616b2f03c42…, 4.61 GB uncompressed (~50 MB smaller than :test after stripping unused python_stage and /opt/conda)
which scp; which rsync; ls /usr/lib/openssh/sftp-server	all present
jupyterhub-singleuser --version	5.4.3
jupyterhub-singleuser boot with realistic env	extensions load (jupyterhub, jupyterlab, jupyterlab_h5web, notebook_shim, jupyter_lsp, jupyter_server_terminals)
import mdx2; mdx2.__version__	1.0.2 (matches live pod)
mdx2.io attribute	absent (matches live pod — io.py only exists on main)
mdx2 source dir contents	matches live pod listing (VERSION, command_line, data.py, dxtbx_machinery.py, geometry.py, scaling.py, utils.py); .git removed
dxtbx.__version__	3.24.1 (matches live)
pip-pinned versions (jupyterhub, jupyter-vscode-proxy, mdx2)	match live
Lockfile parity (385 packages, name+version+build normalized)	md5 d5efd15da8bcbddebfa20b20ef2f2ff6 matches lockfile exactly
mdx2.import_data --help	functional, prints usage
sftp-server -h, scp (no args), rsync --version	all run
actionlint on workflow	passes (no warnings)

Cutover plan (after merge)

1. CI on main pushes diffuseproject/mdx2-jhub:1.0.2-1 to Dockerhub. (One-time: the new Dockerhub repo diffuseproject/mdx2-jhub needs to exist as public and the vars.DOCKERHUB_USERNAME account needs push access. Lazy creation on first push usually works for orgs with auto-create-on-push enabled; eager creation in the Dockerhub UI is the boring/reliable path.)
2. Webapp PR (separate, against diff-use/webapp) updates the JupyterHub singleuser image config from diffuseproject/mdx2:test to diffuseproject/mdx2-jhub:1.0.2-1. See app/config.py:217 plus the repo-name constant if the webapp keeps repo and tag separate.
3. After webapp merge + deploy, restart the JupyterHub user notebook from the Hub Control Panel. KubeSpawner pulls fresh because imagePullPolicy: Always.
4. Verify from a laptop:

   echo hello > /tmp/scp-smoke
   scp -P 30022 /tmp/scp-smoke <user>@jupyter-<cluster>.hub.diffuse.science:/home/jovyan/
   ssh -p 30022 <user>@jupyter-<cluster>.hub.diffuse.science 'cat /home/jovyan/scp-smoke'   # → "hello"
   

Rollback: revert the webapp PR (single-line). Old :test stays parked on Dockerhub.

Notes for future maintenance

• 327bf6e lives only at refs/pull/56/head. If the PR is ever purged the commit becomes unreachable. Suggested follow-up: push that SHA as a durable git tag in diff-use/mdx2 (e.g. singleuser-source-pin) so future builds can pin to a tag instead of a PR ref.
• .github/Dockerfile is kept in the repo despite no longer being referenced by any workflow. It's the (verified) source for what produced :1.0.0 and :latest on Dockerhub. Useful as a reference if the standalone Lab image needs reviving later.
• The mdx2 v1.0.4 upgrade is not bundled here. Separate, deliberate PR once 1.0.2-1 is verified in prod, with its own rollback boundary.

Commit history

Commit	Purpose
1db4c76	Add notebook image CI build (mdx2:1.0.2-1) — the main change
cdf2221	Remove broken docker: job from workflow
14ec2e8	Strip dead weight from notebook image (unused python_stage, /opt/conda, build-context .git) — saves ~50 MB
3249501	Add workflow guards: env vars, immutability check, path filter, permissions
fc51467	Document MDX2_COMMIT: what the SHA is and when to bump it
daaa8c1	Add upgrade guide at docs/upgrading-the-jhub-image.md
7443d20	Rename image from diffuseproject/mdx2 to diffuseproject/mdx2-notebook to disambiguate from the standalone Lab image at :1.0.0/:latest
d002c88	Rename image and supporting files from notebook to jhub (image, Dockerfile, lockfile, doc, workflow refs) — operational-role naming matches the team's verbal shorthand

[jlee733]

Looks good to me