paper: Reproducible Machine Learning Workflows for Scientists with Pixi

Author: matthewfeickert

papers/matthew_feickert/acknowledgements.md

@@ -0,0 +1,6 @@
+## Acknowledgements
+
+Matthew Feickert is supported by the U.S. National Science Foundation (NSF) under Cooperative Agreement PHY-2323298 (IRIS-HEP) and by the US Research Software Sustainability Institute (URSSI) via grant G-2022-19347 from the Sloan Foundation.
+Ruben Arts is supported by prefix.dev GmbH.
+John Kirkham is supported by NVIDIA.
+The described work [@reproducible_machine_learning_scipy_2025_tutorial] was created in association with [@Feickert_Reproducible_Machine_Learning].

papers/matthew_feickert/banner.png

@@ -0,0 +1,2 @@
+# SCM syntax highlighting & preventing 3-way merges
+pixi.lock merge=binary linguist-language=YAML linguist-generated=true

papers/matthew_feickert/code/ml-example/.gitattributes

@@ -0,0 +1,2 @@
+# pixi environments
+.pixi

papers/matthew_feickert/code/ml-example/.gitignore

@@ -0,0 +1,31 @@
+ARG CUDA_VERSION="12"
+ARG ENVIRONMENT="gpu"
+
+FROM ghcr.io/prefix-dev/pixi:noble AS build
+
+# Redeclaring ARGS in a stage without a value inherits the global default
+ARG CUDA_VERSION
+ARG ENVIRONMENT
+
+WORKDIR /app
+COPY . .
+ENV CONDA_OVERRIDE_CUDA=$CUDA_VERSION
+RUN pixi install --locked --environment $ENVIRONMENT
+RUN echo "#!/bin/bash" > /app/entrypoint.sh && \
+    pixi shell-hook --environment $ENVIRONMENT -s bash >> /app/entrypoint.sh && \
+    echo 'exec "$@"' >> /app/entrypoint.sh
+
+FROM ghcr.io/prefix-dev/pixi:noble AS final
+
+ARG ENVIRONMENT
+
+WORKDIR /app
+COPY --from=build /app/.pixi/envs/$ENVIRONMENT /app/.pixi/envs/$ENVIRONMENT
+COPY --from=build /app/pixi.toml /app/pixi.toml
+COPY --from=build /app/pixi.lock /app/pixi.lock
+# The ignore files are needed for 'pixi run' to work in the container
+COPY --from=build /app/.pixi/.gitignore /app/.pixi/.gitignore
+COPY --from=build /app/.pixi/.condapackageignore /app/.pixi/.condapackageignore
+COPY --from=build --chmod=0755 /app/entrypoint.sh /app/entrypoint.sh
+
+ENTRYPOINT [ "/app/entrypoint.sh" ]

papers/matthew_feickert/code/ml-example/Dockerfile

@@ -0,0 +1,37 @@
+[workspace]
+channels = ["conda-forge"]
+name = "ml-example"
+platforms = ["linux-64", "osx-arm64", "win-64"]
+version = "0.1.0"
+
+[tasks]
+
+[dependencies]
+python = ">=3.13.5,<3.14"
+
+[feature.cpu.dependencies]
+pytorch-cpu = ">=2.7.1,<3"
+torchvision = ">=0.22.0,<0.23"
+
+[feature.cpu.tasks.train-cpu]
+description = "Train MNIST on CPU"
+cmd = "python src/torch_MNIST.py --epochs 2 --save-model --data-dir data"
+
+[feature.gpu.system-requirements]
+cuda = "12"
+
+[feature.gpu.target.linux-64.dependencies]
+pytorch-gpu = ">=2.7.1,<3"
+torchvision = ">=0.22.0,<0.23"
+
+[feature.gpu.tasks.train-gpu]
+description = "Train MNIST on GPU"
+cmd = "python src/torch_MNIST.py --epochs 14 --save-model --data-dir data"
+
+[feature.inference.dependencies]
+matplotlib = ">=3.10.3,<4"
+
+[environments]
+cpu = ["cpu"]
+gpu = ["gpu"]
+inference = ["gpu", "inference"]

papers/matthew_feickert/code/ml-example/pixi.toml

@@ -0,0 +1,14 @@
+## Conda packages
+
+Conda packages (`.conda` files) are language agnostic file archives that contain built code distributions and metadata.
+This is quite powerful, as it allows for arbitrary code to be built for any target platform and then packaged with its metadata.
+When a conda package is downloaded and then unpacked with a conda package management tool (e.g. Pixi, conda, mamba) it is then "installed" by copying the package's file directory tree to the base of the environment's directory tree.
+Package contents are also simple; they can only contain files and symbolic links.
+
+### conda-forge
+
+Conda packages can be distributed on package indexes that support the concept of "channels" which redirect URLs to directory trees of conda packages.
+Channel names serve as the base path for hosting packages.
+The most broadly used community channel for conda-packages is the `conda-forge` channel, which hosts the conda packages generated from builds on the global conda-forge community cyberinfrastructure.
+The conda-forge community operates in a GitHub organization that hosts Git repositions that contain conda package build recipes as well as automation infrastructure and continuous integration (CI) and continuous delivery (CD) workflows.
+This allows for conda-forge community members to submit and maintain recipes — instructions for conda package build systems — to build and distribute conda packages for multiple variants of computing platforms — combinations of operating systems and hardware architectures — for Linux, macOS, and Windows.

papers/matthew_feickert/conda-packages.md

@@ -0,0 +1,54 @@
+## CUDA
+
+CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model developed by NVIDIA for general computing on graphical processing units (GPUs). [@CUDA_paper; @CUDA_slides]
+The CUDA ecosystem provides Software Development Kits (SDKs) with APIs to CUDA that allow for software developers to write hardware accelerated programs with CUDA in various languages for NVIDIA GPUs.
+CUDA has official language support for C++, Fortran, and Python, with community support for Julia and other languages.
+While there are other types of hardware acceleration development platforms, as of 2025 CUDA is the most abundant platform for scientific computing that uses GPUs and effectively the default choice for major machine learning libraries and applications.
+
+CUDA is closed source and proprietary to NVIDIA, which means that NVIDIA has historically limited the download access of the CUDA toolkits and drivers to registered NVIDIA developers (while keeping the software free (monetarily) to use).
+CUDA then required a multi-step installation process [@CUDA_install_guide] with manual steps and decisions based on the target platform and particular CUDA version.
+This meant that when CUDA enabled environments were setup on a particular machine they were powerful and optimized, but brittle to change and could easily be broken if system wide updates (like for security fixes) occurred.
+CUDA software environments were bespoke and not many scientists understood how to construct and curate them.
+
+## CUDA packages on conda-forge
+
+### Initial implementation
+
+After discussion in late 2018 [@conda-forge_github_io_issue_687] to better support the scientific developer community, the CUDA packaging community agreed to use the Anaconda `defaults` channel's [@anaconda-defaults-channel] `cudatoolkit` package.
+Initially the `cudatoolkit` package was designed around Numba's CUDA needs [@conda-recipe-cudatoolkit], though it evolved to a bundle of redistributable CUDA libraries.
+In 2019, NVIDIA began packaging the `cudatoolkit` package in the [`nvidia` conda channel](https://anaconda.org/nvidia).
+With help from the broader community, the `cudatoolkit` package was added to `conda-forge` in 2020 [@staged-recipes-pr-12882].
+For the first time, this provided users the _ability to specify different versions of CUDA libraries_ and download them in newly created conda environments.
+
+Supporting initial conda-forge CUDA builds required additional components:
+* [A conda-forge Docker image](https://github.com/conda-forge/docker-images/pull/93) using [the NVIDIA CUDA Docker images](https://hub.docker.com/r/nvidia/cuda/), which provided the NVIDIA build tools for compiling packages.
+* [A shim package](https://github.com/conda-forge/staged-recipes/pull/8229) to leverage the NVIDIA build tools within a conda package build.
+* [A CUDA build matrix in conda-forge's global pinnings](https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/285), which tied these two pieces together.
+
+These ideas were tied together in the first package build on September 20, 2019 [@ucx-split-feedstock-pr-14], and the initial implementation of this work was completed later in 2019.
+In 2020, support was expanded to [Windows CUDA builds](https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/914).
+Lots of iteration on this work happened after, all using the same basic foundation.
+
+### Revised implementation
+
+After some time using these packages and build process, a few observations became clear.
+First, some packages used only a subset of the libraries, like the driver, the CUDA runtime library, or particular library components like cuBLAS.
+However, the `cudatoolkit` package shipped considerably more than that, so having finer specifications of dependencies would provide a better package maintainer and end-user experience.
+Second, some packages needed components that were not part of the `cudatoolkit` bundle like other libraries or parts of the build toolchain.
+Having some way to depend on these components would improve usability.
+Third, the infrastructure management overhead of custom Docker images and their integration into the conda-forge build matrix was cumbersome for conda-forge maintainers.
+Being able to install and use the build tools directly would simplify maintenance and benefit end-users wishing to use these build tools.
+
+To address these issues, NVIDIA began working on a revised set of packages.
+These more closely matched packages in other distribution channels (like Linux distribution package managers) and were adapted to the conda user experience.
+For example, Linux distributions often install packages at the system level, which differs from the first-class userspace environment experience that conda package environments provides.
+As a result, some distinctions that a Linux distribution provides are unneeded in conda.
+There are additional differences around behavior in pinning versions of dependencies or how compilers are packaged and expected to work in their installed environments.
+Initial production of the packages were made on the `nvidia` channel, however, all of this work was being done internally in NVIDIA and published to a separate channel.
+This made the packages less visible and required additional knowledge to use.
+
+In [2023](https://youtu.be/WgKwlGgVzYE?si=hfyAo6qLma8hnJ-N), NVIDIA began adding the releases of CUDA conda packages from the `nvidia` channel to conda-forge, making it easier to discover and allowing for community support.
+Given the new package structure, NVIDIA added the packages for CUDA `12.0` to indicate the breaking change.
+Also with significant advancements in system driver specification support, CUDA `12` became the first version of CUDA to be released as conda packages through conda-forge and included all CUDA libraries from the [CUDA compiler `nvcc`](https://github.com/conda-forge/cuda-nvcc-feedstock) to the [CUDA development libraries](https://github.com/conda-forge/cuda-libraries-dev-feedstock).
+[CUDA metapackages](https://github.com/conda-forge/cuda-feedstock/) were also released, which allow users to easily describe the version of CUDA they require (e.g. `cuda-version=12.5`) and the CUDA conda packages they want (e.g. `cuda`).
+This significantly improved the ability for researchers to easily create CUDA accelerated computing environments.

papers/matthew_feickert/cuda.md

@@ -0,0 +1,57 @@
+version: 6
+environments:
+  cpu:
+    channels:
+    - url: https://conda.anaconda.org/conda-forge/
+    packages:
+      linux-64:
+
+...
+
+      - conda: https://conda.anaconda.org/conda-forge/linux-64/python-3.13.5-hec9711d_102_cp313.conda
+      - conda: https://conda.anaconda.org/conda-forge/noarch/python_abi-3.13-8_cp313.conda
+      - conda: https://conda.anaconda.org/conda-forge/linux-64/pytorch-2.7.1-cpu_mkl_py313_h58dab0e_103.conda
+      - conda: https://conda.anaconda.org/conda-forge/linux-64/pytorch-cpu-2.7.1-cpu_mkl_hc60beec_103.conda
+
+...
+
+  gpu:
+    channels:
+    - url: https://conda.anaconda.org/conda-forge/
+    packages:
+      linux-64:
+
+...
+
+      - conda: https://conda.anaconda.org/conda-forge/linux-64/cuda-nvcc-tools-12.9.86-he02047a_2.conda
+      - conda: https://conda.anaconda.org/conda-forge/linux-64/cuda-nvdisasm-12.9.88-hbd13f7d_0.conda
+      - conda: https://conda.anaconda.org/conda-forge/linux-64/cuda-nvrtc-12.9.86-h5888daf_0.conda
+      - conda: https://conda.anaconda.org/conda-forge/linux-64/cuda-nvtx-12.9.79-h5888daf_0.conda
+      - conda: https://conda.anaconda.org/conda-forge/linux-64/cuda-nvvm-tools-12.9.86-h4bc722e_2.conda
+      - conda: https://conda.anaconda.org/conda-forge/noarch/cuda-version-12.9-h4f385c5_3.conda
+      - conda: https://conda.anaconda.org/conda-forge/linux-64/cudnn-9.10.1.4-hbcb9cd8_1.conda
+
+...
+
+      - conda: https://conda.anaconda.org/conda-forge/linux-64/python-3.13.5-hec9711d_102_cp313.conda
+      - conda: https://conda.anaconda.org/conda-forge/noarch/python_abi-3.13-8_cp313.conda
+      - conda: https://conda.anaconda.org/conda-forge/linux-64/pytorch-2.7.1-cuda129_mkl_py313_h1e53aa0_304.conda
+      - conda: https://conda.anaconda.org/conda-forge/linux-64/pytorch-gpu-2.7.1-cuda129_mkl_h43a4b0b_304.conda
+
+...
+
+packages:
+
+...
+
+- conda: https://conda.anaconda.org/conda-forge/linux-64/pytorch-gpu-2.7.1-cuda129_mkl_h43a4b0b_304.conda
+  sha256: af54e6535619f4e484d278d015df6ea67622e2194f78da2c0541958fc3d83d18
+  md5: e374ee50f7d5171d82320bced8165e85
+  depends:
+  - pytorch 2.7.1 cuda*_mkl*304
+  license: BSD-3-Clause
+  license_family: BSD
+  size: 48008
+  timestamp: 1753886159800
+
+...

papers/matthew_feickert/example.lock

@@ -0,0 +1,24 @@
+## Introduction
+
+A critical component of research software sustainability is the reproducibility of the software and computing environments software operates and "lives" in.
+Providing software as a "package" — a standardized distribution of all source or binary components of the software required for use along with identifying metadata — goes a long way to improved reproducibility of software libraries.
+However, more researchers are consumers of libraries than developers of them, but still need reproducible computing environments for research software applications that may be run across multiple computing platforms — e.g. scientific analyses, visualization tools, data transformation pipelines, and artificial intelligence (AI) and machine learning (ML) applications on hardware accelerator platforms (e.g. GPUs).
+While workflow engines and Linux containers offer a gold standard for scientific computing reproducibility, they require additional layers of training and software engineering knowledge.
+Modern open source multi-platform environment management tools, e.g. [@pixi], provide automatic multi-platform hash-level lock file support for all dependencies — down to the compiler level — of software on public package indexes (e.g. PyPI [@PyPI_website] and conda-forge [@conda-forge_community]) while still providing a high level interface well suited for researchers.
+Combined with the arrival of the full CUDA [@CUDA_paper] stack on conda-forge, it is now possible to declaratively specify a full CUDA accelerated software environment.
+We are now at a point where well supported, robust technological solutions exist, even for applications with highly complex software environments.
+What is currently lacking is the education and training by the broader scientific software community to adopt these technologies and build community standards of practice around them, as well as an understanding of what are the most actionably useful features of adopting computational reproducibility tools.
+
+## Reproducibility
+
+"Reproducible" research is a term that can mean multiple things across various fields.
+Some fields may view work as "reproducible" if the full process is documented, and other may view "reproducible" as meaning that all computations will give the same numerical outputs barring entropy variations.
+As there are multiple levels of reproducibility, we will restrict "reproducibility" to software environment reproducibility.
+We define this as be limited to the ability to define and programmatically create a software environment composed of packages that specifies all software, and its dependencies, with exact URLs and binary digests ("hashes").
+Reproducible environments need to be machine agnostic in that for a specified computing platform in the environment they must be installable without modification across multiple instances.
+
+### Hardware accelerated environments
+
+Software the involves hardware acceleration on computing resources like GPUs requires additional information to be provided for full computational reproducibility.
+In addition to the computer platform, information about the hardware acceleration device, its supported drivers, and compatible hardware accelerated versions of the software in the environment (GPU enabled builds) are required.
+Traditionally this has been very difficult to do, but multiple recent technological advancements (made possible by social agreements and collaborations) in the scientific open source world now provide solutions to these problems.