You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: papers/matthew_feickert/acknowledgements.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,4 +3,4 @@
3
3
Matthew Feickert is supported by the U.S. National Science Foundation (NSF) under Cooperative Agreement PHY-2323298 (IRIS-HEP) and by the US Research Software Sustainability Institute (URSSI) via grant G-2022-19347 from the Sloan Foundation.
4
4
Ruben Arts is supported by prefix.dev GmbH.
5
5
John Kirkham is supported by NVIDIA.
6
-
The described tutorial was created in association with [@Feickert_Reproducible_Machine_Learning].
6
+
The described work [@reproducible_machine_learning_scipy_2025_tutorial] was created in association with [@Feickert_Reproducible_Machine_Learning].
Often researchers are running scientific and machine learning workflows on remote computational resources that use batch computing systems (e.g. HTCondor, SLURM).
4
+
Some of these systems may not have shared filesystems, requiring that each worker node receive its own copy of the software environment.
5
+
While locked Pixi environments significantly help with this, it is often advantageous to distribute the environment in the form of a Linux container image to the compute resources.
6
+
These systems are able to mount Linux container images to worker nodes in ways that reduce the disk and memory cost to the user's session, compared to installing Pixi and then downloading all dependencies of the software environment from the package indexes used.
7
+
This also reduces the bandwidth use as the Linux container image can be cached at the compute resource host and efficiently replicated to the worker nodes, paying the bandwidth cost of download once.
8
+
While Linux container technology historically has presented additional engineering and design overhead to researchers, Linux container construction of Pixi environments is simple and can be reduced to templated format.
9
+
An example in the form of a templated Dockerfile is seen in @example-pixi-dockerfile.[^docker_footnote]
10
+
The template requires user input to define the target CUDA version (`CUDA_VERSION`) and the name of the Pixi environment to install (`ENVIRONMENT`).
11
+
As the Pixi environment is already fully defined and locked it can be directly installed as normal in the `build` stage of the container image build, along with an entrypoint shell script that will activate the environment, and then copied from the `build` stage into the `final` stage to reduce the total image size by removing the cache and reducing the total number of layers in the final image.
12
+
13
+
```{literalinclude} code/ml-example/Dockerfile
14
+
:label: example-pixi-dockerfile
15
+
:caption: The template structure of a Dockerfile for a locked Pixi environment with CUDA dependencies. The only values that need user input are the CUDA version and the name of the target environment.
16
+
```
17
+
18
+
The Dockerfile can then be built into a Linux container image binary file which can be distributed to a container image registry.
19
+
Batch computing system workflow definition files can use these container images to provide the software environment for the computing jobs, which pull the images from the container image registry when requested by the job.
20
+
21
+
[^docker_footnote]: As many compute facilities do not allow for use of Docker directly given security concerns, Apptainer container image formats are more common.
22
+
Apptainer definition files are similarly easy to write as compared to Dockerfiles and Docker container images can be converted into a format that Apptainer can use.
23
+
As Docker is a more common format in the broader computing world, including commercial settings, it has been used for this example.
24
+
These workflows are not limited to a single container image format.
Copy file name to clipboardExpand all lines: papers/matthew_feickert/main.md
+7-1Lines changed: 7 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ title: Reproducible Machine Learning Workflows for Scientists with Pixi
4
4
abstract: |
5
5
Scientific researchers need reproducible software environments for complex applications that can run across heterogeneous computing platforms.
6
6
Modern open source tools, like Pixi, provide automatic reproducibility solutions for all dependencies while providing a high level interface well suited for researchers.
7
-
Combined with the recent emergence of the entire CUDA software stack — from compilers to development libraries — being supported on conda-forge, researchers are now able to easily specify their exact hardware acceleration requirements and software dependencies and get portable computational environments locked down to the hash level.
7
+
Combined with the recent emergence of the entire CUDA software stack — from compilers to development libraries — being supported on conda-forge, researchers are now able to easily specify their exact hardware acceleration requirements and software dependencies and get portable computational environments locked down to the digest level.
Copy file name to clipboardExpand all lines: papers/matthew_feickert/pixi.md
+131-3Lines changed: 131 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -30,10 +30,12 @@ This declarative nature allows for users to efficiently specify their project re
30
30
### CUDA hardware accelerated environment creation
31
31
32
32
Combining the features of modern CUDA `12` conda packages with Pixi's environment management, it is now possible to efficiently manage multiple software environments that can include both hardware accelerated and CPU environments.
33
-
An example Pixi workspace is presented below
33
+
An example Pixi workspace is presented in @pixi-ml-example-workspace
34
34
35
35
```{literalinclude} code/ml-example/pixi.toml
36
36
:linenos:
37
+
:label: pixi-ml-example-workspace
38
+
:caption: Example of a multi-platform and multi-environment Pixi manifest with all required information and constraints to resolve and install CUDA accelerated conda packages.
37
39
```
38
40
39
41
where the definition of multiple platforms allows for solving the declared environments for all platforms while on other platforms
@@ -47,6 +49,132 @@ where the definition of multiple platforms allows for solving the declared envir
47
49
48
50
the `cpu` feature defines `dependencies` and `tasks` that are accessible from the `cpu` environment
The `gpu` feature does the same for the `gpu` environment, but it also importantly defines a [`system-requirements` table](https://pixi.sh/v0.50.2/workspace/system_requirements/) that define the system specifications needed to install and run a Pixi workspace's environments.
`system-requirements` build upon the concept of conda "[virtual packages](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-virtual.html)", allowing for the dependency resolver to enforce constraints declared by defining compatibility of the system with virtual packages, like `__cuda`.
98
+
In the particular case of CUDA, the `system-requirements` table specifies the CUDA version the workspace expects the host system to support, as detected through the host system's NVIDIA driver API.
99
+
While the `system-requirements` field values do not correspond to lower or upper bounds, specifying that the workspace is expected to work on systems that support CUDA 12
100
+
101
+
```{code} toml
102
+
:filename: pixi.toml
103
+
104
+
...
105
+
106
+
[feature.gpu.system-requirements]
107
+
cuda = "12"
108
+
109
+
...
110
+
111
+
```
112
+
113
+
ensures that packages depending on `__cuda >= 12` are resolved correctly.
114
+
This effectively means that declaring the system requirement will cause the Pixi dependency resolver to find CUDA enabled packages that are compatible with CUDA 12, disallowing for incompatible package builds to be resolved.
115
+
Once these package dependencies have been resolved and locked, this ensures that any system capable of meeting the system requirement will get working CUDA accelerated conda packages installed.
116
+
117
+
Not all machines will have an NVIDIA GPU on them to allow for the system requirements to be resolved correctly.
118
+
To allow for non-CUDA-supported-machines to still resolve Pixi workspace requirements, shell environment overrides exist through the `CONDA_OVERRIDE_CUDA` environmental variable.
119
+
Setting `CONDA_OVERRIDE_CUDA=12` on a machine that doesn't meet the CUDA version requirements, will override the supported virtual packages and set a value of `__cuda=12` for the system.
120
+
This can be clearly understood from setting the override and then querying the workspace summary with `pixi info`, as seen in @conda-override-cuda-example.
121
+
This is a powerful functionality as it allows for environment specification, resolution, and locking for target platforms that users might not have access to, but can be assured are valid.
122
+
123
+
```{code} console
124
+
:label: conda-override-cuda-example
125
+
:caption: Demonstration of using the `CONDA_OVERRIDE_CUDA` environmental variable on a system with no CUDA support (an Apple silicon machine) to allow dependency resolution as if it supported CUDA 12.
126
+
127
+
% pixi info
128
+
System
129
+
------------
130
+
Pixi version: 0.50.2
131
+
Platform: osx-arm64
132
+
Virtual packages: __unix=0=0
133
+
: __osx=15.3.2=0
134
+
: __archspec=1=m2
135
+
...
136
+
137
+
% CONDA_OVERRIDE_CUDA=12 pixi info
138
+
System
139
+
------------
140
+
Pixi version: 0.50.2
141
+
Platform: osx-arm64
142
+
Virtual packages: __unix=0=0
143
+
: __osx=15.3.2=0
144
+
: __cuda=12=0
145
+
: __archspec=1=m2
146
+
...
147
+
```
148
+
149
+
Pixi also allows for feature composition to efficiently create new environments.
150
+
@pixi-ml-example-workspace's `gpu` and `inference` features are combined and resolved collectively to provide a new CUDA accelerated `inference` environment that does not affect the `gpu` environment.
151
+
152
+
```{code} toml
153
+
:filename: pixi.toml
154
+
155
+
...
156
+
157
+
[feature.inference.dependencies]
158
+
matplotlib = ">=3.10.3,<4"
159
+
160
+
[environments]
161
+
...
162
+
gpu = ["gpu"]
163
+
inference = ["gpu", "inference"]
164
+
```
165
+
166
+
### Locked environments
167
+
168
+
Once the workspace has been defined, any Pixi operation on the workspace will result in all environments in the workspace having their dependencies resolved and then fully specified ("locked") at the digest ("hash") level in a single `pixi.lock` Pixi lock file, as seen in @example-pixi-lockfile.
169
+
The lock file is a YAML file that contains two definition groups: `environments` and `packages`.
170
+
The `environments` group lists every environment in the workspace for every platform with a complete listing of all packages in the environment.
171
+
The `packages` group lists a full definition of every package that appears in the `environments` lists, including the package's URL and digests (e.g. sha256, md5).
172
+
These groups provide a full description of every package described in the Pixi workspace and its dependencies and constraints on other packages.
173
+
Versioning the lock file along with the manifest file in a version control system allows for workspaces to be fully reproducible to the byte level indefinitely into the future, conditioned on the continued existence of the package indexes the workspace pulls from (e.g. conda-forge, PyPI, the nvidia conda channel).
174
+
In the event that long term preservation and reproducibility are of importance, there are community projects [@pixi-pack] that allow for downloading all dependencies of a Pixi environment and generating a tar archive containing all of the packages, which can later be unpacked and installed.
175
+
176
+
```{literalinclude} example.lock
177
+
:filename: pixi.lock
178
+
:label: example-pixi-lockfile
179
+
:caption: Example structure of a `pixi.lock` Pixi lock file showing the definition of the environments as well as a full description of each package used in each environment.
As hardware accelerated code becomes more common across scientific computing, especially CUDA accelerated software for machine learning, the need for simple but powerful solutions for software environment management has grown too.
4
+
The simple and flexible structure of conda packages allows for complex projects to be packaged as directory trees of built binaries on a platform specific level.
5
+
This has allowed for the complexity of the CUDA software stack to be efficiently built as conda packages using the conda-forge cyberinfrastructure and then distributed on the conda-forge conda channel for public use.
6
+
Distribution of CUDA conda packages on conda-forge additionally allows for other conda-forge projects to use the CUDA conda packages in their builds, resulting in a wide selection of CUDA enabled projects, including many machine learning packages.
7
+
Through use of Pixi's declarative specification of dependencies in the project manifest and non-optional digest level lock file generation, software environments can now be declaratively and rapidly constructed, resolved, and locked using semantic operations well designed for scientific researchers.
8
+
With these powerful technologies and abstractions, researchers can now construct machine learning and data science environments for multiple platforms at once and use trusted patterns to develop locally and deploy to remote computational resources.
9
+
10
+
In addition to the long term reproducibility provided by the combination of these technologies, the maintenance burden and complexity reduction should not be overlooked.
11
+
With the CUDA v12 distributions on conda-forge, researches no longer need to have experience in CUDA internals and distribution installation to accelerate their software projects.
12
+
They need only know the supported versions of CUDA by the NVIDIA drivers on their target machines.
13
+
Researchers also no longer need to use multiple tools build bespoke workflows for constructing and maintaining lock files for multiple environments and platforms, while keeping environment definition files and lock files synced.
14
+
Pixi provides a single tool and unified interface to achieve the same results faster while using high level abstractions — removing most of the work of software environment reproducibility from the user workflow.
15
+
Having the full specification of the software environment including the CUDA dependencies also removes runtime failures due to missing, unspecified, or incompatible system-level requirements on remote compute resources.
16
+
Most importantly, reducing cognitive overhead and the latency to reach a useable software environment reduces the time to insight for researchers, transferring the problems of scientific computing back into their domains of expertise.
0 commit comments