Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions docs/software/container-engine/known-issue.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,29 @@ Mounting individual home directories (usually located on the `/users` filesystem

It is generally NOT recommended to mount home folders inside containers, due to the risk of exposing personal data to programs inside the container.
Defining a mount related to `/users` in the EDF should only be done when there is a specific reason to do so, and the container image being deployed is trusted.

[](){#ref-ce-why-no-sbatch-env}
## Why `--environment` as `#SBATCH` is discouraged

Due to how Slurm works, when using `--environment` as an `#SBATCH` option, the entire contents of the SBATCH script is executed within a container created by the EDF file.
This can lead to subtle and hard-to-diagnose failures, some of which are described below.

- **Slurm availability in the container**: In some cases, CE does not inject essential Slurm components in containers, which result in crashes on basic Slurm operations (e.g., `srun`) inside the SBATCH script. Even if they were injected, it's not guaranteed to cover the complete feature set of Slurm.

- **The execution context is not the host system**: Since the entire SBATCH script runs inside a container (shaped with EDF), all commands in the script are affected by the environment defined by EDF. This primarily includes filesystem mounts, where any directories not explicitly mounted in EDF are invisible to all commands inside the SBATCH script.

- **Nested use of `--environment`**: `--environment` in the SBATCH script _and_ for a `srun` command results in entering the EDF environment twice, causing unexpected errors due to double-entering containers.

For these reasons, we encourage using `--environment` for each `srun` as shown below.

```bash
#!/bin/bash
#SBATCH --cpus-per-task=4
...
srun --environment=my_edf echo 'this'
...
srun --environment=my_edf echo 'that'
...
```

As the use of `--environment` as an `#SBATCH` option is reserved for highly customized workflows, users should have a high level of proficiency and a full understanding of the risk to encounter cryptic behaviors. Should users encounter a problem while using `--environment` as `#SBATCH`, it's recommended to move `--environment` from `#SBATCH` to each `srun` and see if the problem disappears.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could also mention a benefit of only using --environment as an argument to srun: if you have more than one srun command in the batch script, you can easily use different containers for the different commands.

5 changes: 2 additions & 3 deletions docs/software/container-engine/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,8 @@ Use `--environment` with the Slurm command (e.g., `srun` or `salloc`):
Specifying the `--environment` option with an `#SBATCH` option is **experimental**.
Such usage is discouraged as it may result in unexpected behaviors.

!!! note
Specifying `--environment` with `#SBATCH` will put the entire batch script inside the containerized environment, requiring the Slurm hook to use any Slurm commands within the batch script (e.g., `srun` or `scontrol`).
The hook is controlled by the `ENROOT_SLURM_HOOK` environment variable and activated by default on most vClusters.
!!! warning
The use of `--environment` as an `#SBATCH` option is reserved for highly customized workflows, and it may result in several **counterintuitive, hard-to-diagnose failures**. See [Why `--environment` as `#SBATCH` is discouraged][ref-ce-why-no-sbatch-env] for details.

[](){#ref-ce-edf-search-path}
### EDF search path
Expand Down