diff --git a/docs/software/container-engine/known-issue.md b/docs/software/container-engine/known-issue.md index 7569df1b..d1e7038e 100644 --- a/docs/software/container-engine/known-issue.md +++ b/docs/software/container-engine/known-issue.md @@ -52,3 +52,16 @@ Mounting individual home directories (usually located on the `/users` filesystem It is generally NOT recommended to mount home folders inside containers, due to the risk of exposing personal data to programs inside the container. Defining a mount related to `/users` in the EDF should only be done when there is a specific reason to do so, and the container image being deployed is trusted. + +[](){#ref-ce-why-no-sbatch-env} +## Why `--environment` as `#SBATCH` is discouraged + +The use of `--environment` as `#SBATCH` is known to cause **unexpected behaviors** and is exclusively reserved for highly customized workflows. This is because `--environment` as `#SBATCH` puts the entire SBATCH script in a container from the EDF file. The following are a few known associated issues. + + - **Slurm availability in a container**: Either Slurm components are not completely injected inside a container, or injected Slurm components do not function properly. + + - **Non-host execution context**: Since the SBATCH script runs inside a container, most host resources are inaccessible by default unless EDF explicitly exposes them. Affected resources include: filesystems, devices, system resources, container hooks, etc. + + - **Nested use of `--environment`**: running `srun --environment` in `#SBATCH --environment` results in double-entering EDF containers, causing unexpected errors in the underlying container runtime. + +To avoid any unexpected confusion, users are advised **not** to use `--environment` as `#SBATCH`. If users encounter a problem while using this, it's recommended to move `--environment` from `#SBATCH` to each `srun` and see if the problem disappears. diff --git a/docs/software/container-engine/run.md b/docs/software/container-engine/run.md index 6d399d91..4b51af1d 100644 --- a/docs/software/container-engine/run.md +++ b/docs/software/container-engine/run.md @@ -34,17 +34,30 @@ Use `--environment` with the Slurm command (e.g., `srun` or `salloc`): #SBATCH --job-name=edf-example #SBATCH --time=00:01:00 ... - - # Run job step srun --environment=ubuntu cat /etc/os-release ``` +Multiple Slurm commands may have different EDF environments; this is useful when a single environment is not feasible due to compatibility issues or keep EDF files modular. + +!!! example "`srun`s with different EDFs" + ```bash + #!/bin/bash + #SBATCH --job-name=edf-example + #SBATCH --time=00:01:00 + ... + srun --environment=env1 ... # (1)! + ... + srun --environment=env2 ... # (2)! + ``` + + 1. Assuming `env1.toml` is at `EDF_PATH`. See [EDF search path][ref-ce-edf-search-path] below. + 2. Assuming `env2.toml` is at `EDF_PATH`. See [EDF search path][ref-ce-edf-search-path] below. + Specifying the `--environment` option with an `#SBATCH` option is **experimental**. Such usage is discouraged as it may result in unexpected behaviors. -!!! note - Specifying `--environment` with `#SBATCH` will put the entire batch script inside the containerized environment, requiring the Slurm hook to use any Slurm commands within the batch script (e.g., `srun` or `scontrol`). - The hook is controlled by the `ENROOT_SLURM_HOOK` environment variable and activated by default on most vClusters. +!!! warning + The use of `--environment` as an `#SBATCH` option is reserved for highly customized workflows, and it may result in several **counterintuitive, hard-to-diagnose failures**. See [Why `--environment` as `#SBATCH` is discouraged][ref-ce-why-no-sbatch-env] for details. [](){#ref-ce-edf-search-path} ### EDF search path