Unable to use Singularity runtime on Fred Hutch's `rhino`

[victorlin]

Description

Some commands fail with:

mkdir: cannot create directory ‘/nextstrain/env.d’: Operation not supported

This happens on latest nextstrain/base:build-20250708T012131Z, but still works with nextstrain/base:build-20240130T210714Z from nextstrain/cli#360 (comment).

I bisected other runtime versions to find that build-20250226T202705Z works but build-20250304T041009Z does not. This points to #241 as introducing breaking changes.

Scope

• nextstrain version --verbose
• nextstsrain build

nextstrain shell is not affected.

How to reproduce

Steps to reproduce the current behavior:

1. Connect to rhino

2. Run nextstrain version --verbose

3. See error:

     singularity (default)
       docker://nextstrain/base:build-20250708T012131Z (/home/vlin/.nextstrain/runtimes/singularity/images/nextstrain/base/build-20250708T012131Z.sif)
   mkdir: cannot create directory ‘/nextstrain/env.d’: Operation not supported
   

Additional context

Here is the output of nextstrain shell . -c "ls -al /nextstrain" with Docker:

drwxrwxrwt  1 nextstrain nextstrain 4096 Jul 15 19:07 .
drwxr-xr-x  1 root       root       4096 Jul 15 19:07 ..
drwxr-xr-x  1 nextstrain nextstrain 4096 Jul  8 01:39 augur
drwxr-xr-x 14 nextstrain nextstrain 4096 Jul  8 01:23 auspice
-rw-r--r--  1        501 dialout    1243 Jul 15 19:07 bash_history
drwxr-xr-x 13        501 dialout     416 Jul 14 23:44 build
-rw-rw-rw-  1 nextstrain nextstrain   25 Jul  8 01:42 config
drwx------  2        501 dialout      64 Jul 15  2025 env.d
drwxr-xr-x 11 nextstrain nextstrain 4096 Jul  8 01:27 fauna

and with Singularity:

drwxr-xr-x  1 vlin g_vlin  60 Jul 15 12:07 .
drwxr-xr-x  1 vlin g_vlin  60 Jul 15 12:07 ..
drwxr-xr-x 11 root root   646 Jul  7 18:26 augur
drwxr-xr-x 14 root root   798 Jul  7 18:23 auspice
-rw-rw----  1 vlin g_vlin  50 Jul 15 12:07 bash_history
drwxrwx---  7 vlin g_vlin 232 Jul 15 10:50 build
-rw-rw-rw-  1 root root    25 Jul  7 18:41 config
drwxrwx---  2 vlin g_vlin  40 Jul 15 12:07 env.d
drwxr-xr-x 11 root root   427 Jul  7 18:26 fauna

It looks like permissions configured for /nextstrain are overridden by permissions on the host machine since that directory is mounted. This may have broader implications for specific commands run inside the image.

Possible solution

Reconfigure permissions / directory creation to work with Singularity.

[victorlin]

It doesn't look like Nextstrain CLI's Singularity runner uses /nextstrain/env.d, so I think everything in create-envd can be skipped when running under Singularity.

WIP with 5aebd2f, nextstrain/cli@e776ac6

[tsibley]

My preference would be to fix the underlying issue if we can reasonably do so rather than paper over it by special-casing/skipping.

It looks like permissions configured for /nextstrain are overridden by permissions on the host machine since that directory is mounted.

What directory is mounted?

[tsibley]

I cannot access rhino at the moment due to FH network and device policies, but I'm unable to reproduce this on my own machine. I also note that the ls -al output under Singularity (that you included above) looks correct to me and the nextstrain shell command indeed succeeds rather than fails like the nextstrain version --verbose command. I suspect there is an issue with how we're invoking Singularity to collect component versions within the container, rather than something more fundamental about how we're handling /nextstrain/env.d.

[victorlin]

It looks like permissions configured for /nextstrain are overridden by permissions on the host machine since that directory is mounted.

What directory is mounted?

Oh, I meant /nextstrain/build instead of nextstrain:

Mapped volumes:
  /nextstrain/build is from /home/vlin/repos

[tsibley]

Ok, I can reproduce on rhino and so I investigated. It's notable that nextstrain shell works but nextstrain version --verbose and nextstrain build do not, esp. since they all go thru singularity run and more importantly since nextstrain shell and nextstrain build both go thru the same nextstrain.cli.runner.singularity code path. That latter point means that the difference must be in the specific invocations of singularity (as influenced by shell vs. build). For example, the first command here works, the second does not, but they both should.

$ nextstrain shell --singularity --exec bash ~/empty -c 'pwd; ls -la /nextstrain'
Entering the Nextstrain runtime (singularity)

Mapped volumes:
  /nextstrain/build is from /home/tsibley/empty

Run the command "exit" to leave the runtime.

/nextstrain/build
total 59
drwxr-xr-x  1 tsibley g_tsibley   60 Sep  4 20:21 .
drwxr-xr-x  1 tsibley g_tsibley   60 Sep  4 20:21 ..
drwxr-xr-x 11 root    root       646 Sep  2 18:03 augur
drwxr-xr-x 14 root    root       798 Sep  2 17:59 auspice
-rw-rw----  1 tsibley g_tsibley 1025 Sep  4 20:21 bash_history
drwxrwx---  2 tsibley g_tsibley    0 Sep  4 20:10 build
-rw-rw-rw-  1 root    root        25 Sep  2 18:19 config
drwxr-xr-x  2 tsibley g_tsibley   40 Sep  4 20:21 env.d
drwxr-xr-x 11 root    root       427 Sep  2 18:03 fauna

$ nextstrain build --singularity --exec bash ~/empty -c 'pwd; ls -la /nextstrain'
mkdir: cannot create directory '/nextstrain/env.d': Operation not supported

I used NEXTSTRAIN_DEBUG=1 to pull out the underlying singularity commands being run and then ran and tweaked them directly myself to start to narrow things down.

This command, slightly cut down from what nextstrain shell above runs, works:

singularity run \
    --contain \
    --no-home \
    --home : \
    --writable-tmpfs \
    --cleanenv \
    --pwd / \
    --bind /home/tsibley/empty:/nextstrain/build:rw \
    --bind /home/tsibley/.nextstrain/shell-history:/nextstrain/bash_history:rw \
    --pwd /nextstrain/build \
    /home/tsibley/.nextstrain/runtimes/singularity/images/nextstrain/base/build-20250902T175636Z.sif \
    bash -c 'pwd; ls -la /nextstrain'

but it fails when the shell-history bind is removed. It seems that if there's a single bind to /nextstrain/… then mkdir /nextstrain/env.d fails (/nextstrain is read-only, maybe?), but if there's multiple binds under /nextstrain/… (e.g. /nextstrain/x and /nextstrain/y) then it all works. This seems like a quirk of how Singularity achieves binds and directory overlays. We can work around it, probably without too much fuss.