[Bug]: Venv race condition

[TrapsterDK]

What happened?

When running bazel build it can result in a race condition in the virtual environment code making the code crash.

Version

Development (host) and target OS/architectures:

Output of bazel --version:
8.5.1

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:
bazel_dep(name = "aspect_rules_py", version = "1.8.4") // Should not be fixed in newer versions either

Language(s) and/or frameworks involved:

How to reproduce

venv_race_demo.py

from importlib import import_module

print(import_module("coverage").__version__)


BUILD.bazel

load("@aspect_rules_py//py:defs.bzl", "py_binary")
load(":defs.bzl", "make_venv_race_repro")

package(default_visibility = ["//visibility:public"])

py_binary(
    name = "venv_race_demo",
    srcs = ["venv_race_demo.py"],
    tags = ["manual"],
    deps = ["@pip//coverage"],
)

make_venv_race_repro()


defs.bzl

def make_venv_race_repro(count = 128):
    srcs = []

    for i in range(count):
        name = "race_%s" % str(i)
        out = name + ".txt"
        srcs.append(":" + name)

        native.genrule(
            name = name,
            outs = [out],
            # One Bazel build runs many local actions that all execute the same
            # aspect_rules_py py_binary tool target in parallel.
            cmd = "$(location :venv_race_demo) >/dev/null && touch $@",
            local = True,
            tags = ["manual", "no-sandbox"],
            tools = [":venv_race_demo"],
        )

    native.filegroup(
        name = "race",
        srcs = srcs,
        tags = ["manual"],
    )


bazel build //...

Any other information?

Traceback when race condition happens:

XXX@XX:~/XX/workspace-1$ bazel build //XX:race
INFO: Invocation ID: 27a1d5f5-4656-4cca-aed7-c8351a31963d
INFO: Analyzed target //XX:race (275 packages loaded, 5658 targets configured).
ERROR: /home/XXX/XX/workspace-1/XX/BUILD.bazel:5:21: Executing genrule //XX:race_9 failed: (Exit 1): bash failed: error executing Genrule command (from target //XX:race_9) /bin/bash -c ... (remaining 1 argument skipped)
Error:   × Unable to run command:
  ├─▶ Unable to create symlink: /home/XXX/.cache/bazel/
  │   _bazel_XXX/9b5d2fa0c214848a59c1b3ab52d3254b/execroot/
  │   _main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/tools/python/
  │   venv_race_demo.runfiles/rules_python++pip+pip_313_coverage/
  │   site-packages/coverage/collector.py -> /home/XXX/.cache/
  │   bazel/_bazel_XXX/9b5d2fa0c214848a59c1b3ab52d3254b/execroot/
  │   _main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/tools/python/
  │   venv_race_demo.runfiles/.venv_race_demo.venv/lib/python3.13/site-
  │   packages/coverage/collector.py
  ╰─▶ No such file or directory (os error 2)

ERROR: /home/XXX/XX/workspace-1/XX/BUILD.bazel:5:21: Executing genrule //XX:race_4 failed: (Exit 1): bash failed: error executing Genrule command (from target //XX:race_4) /bin/bash -c ... (remaining 1 argument skipped)
Error:   × Unable to run command:
  ├─▶ Unable to create symlink: /home/XXX/.cache/bazel/
  │   _bazel_XXX/9b5d2fa0c214848a59c1b3ab52d3254b/execroot/
  │   _main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/tools/python/
  │   venv_race_demo.runfiles/rules_python++pip+pip_313_coverage/
  │   site-packages/coverage-7.13.4.dist-info/top_level.txt -> /home/
  │   XXX/.cache/bazel/_bazel_XXX/9b5d2fa0c214848a59c1b3ab52d3254b/
  │   execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/tools/python/
  │   venv_race_demo.runfiles/.venv_race_demo.venv/lib/python3.13/site-
  │   packages/coverage-7.13.4.dist-info/top_level.txt
  ╰─▶ File exists (os error 17)

ERROR: /home/XXX/XX/workspace-1/XX/BUILD.bazel:5:21: Executing genrule //XX:race_8 failed: (Exit 1): bash failed: error executing Genrule command (from target //XX:race_8) /bin/bash -c ... (remaining 1 argument skipped)
Error:   × Unable to run command:
  ├─▶ Unable to remove venv_root directory
  ╰─▶ Directory not empty (os error 39)

ERROR: /home/XXX/XX/workspace-1/XX/BUILD.bazel:5:21: Executing genrule //XX:race_5 failed: (Exit 1): bash failed: error executing Genrule command (from target //XX:race_5) /bin/bash -c ... (remaining 1 argument skipped)
Error:   × Unable to run command:
  ╰─▶ The directory `bazel-out/k8-opt-exec-ST-d57f47055a04/bin/tools/python/
      venv_race_demo.runfiles/.venv_race_demo.venv` exists, but it's not a
      virtual environment

ERROR: /home/XXX/XX/workspace-1/XX/BUILD.bazel:5:21: Executing genrule //XX:race_10 failed: (Exit 1): bash failed: error executing Genrule command (from target //XX:race_10) /bin/bash -c ... (remaining 1 argument skipped)
Error:   × Unable to run command:
  ╰─▶ The directory `bazel-out/k8-opt-exec-ST-d57f47055a04/bin/tools/python/
      venv_race_demo.runfiles/.venv_race_demo.venv` exists, but it's not a
      virtual environment

ERROR: /home/XXX/XX/workspace-1/XX/BUILD.bazel:5:21: Executing genrule //XX:race_1 failed: (Exit 1): bash failed: error executing Genrule command (from target //XX:race_1) /bin/bash -c ... (remaining 1 argument skipped)
Error:   × Unable to run command:
  ├─▶ Unable to remove venv_root directory
  ╰─▶ Directory not empty (os error 39)

ERROR: /home/XXX/XX/workspace-1/XX/BUILD.bazel:5:21: Executing genrule //XX:race_6 failed: (Exit 1): bash failed: error executing Genrule command (from target //XX:race_6) /bin/bash -c ... (remaining 1 argument skipped)
Error:   × Unable to run command:
  ╰─▶ The directory `bazel-out/k8-opt-exec-ST-d57f47055a04/bin/tools/python/
      venv_race_demo.runfiles/.venv_race_demo.venv` exists, but it's not a
      virtual environment

ERROR: /home/XXX/XX/workspace-1/XX/BUILD.bazel:5:21: Executing genrule //XX:race_63 failed: (Exit 1): bash failed: error executing Genrule command (from target //XX:race_63) /bin/bash -c ... (remaining 1 argument skipped)
Error:   × Unable to run command:
  ├─▶ Unable to create symlink: /home/XXX/.cache/bazel/
  │   _bazel_XXX/9b5d2fa0c214848a59c1b3ab52d3254b/execroot/
  │   _main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/tools/python/
  │   venv_race_demo.runfiles/rules_python++pip+pip_313_coverage/site-
  │   packages/coverage-7.13.4.dist-info/INSTALLER -> /home/XXX/.cache/
  │   bazel/_bazel_XXX/9b5d2fa0c214848a59c1b3ab52d3254b/execroot/
  │   _main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/tools/python/
  │   venv_race_demo.runfiles/.venv_race_demo.venv/lib/python3.13/site-
  │   packages/coverage-7.13.4.dist-info/INSTALLER
  ╰─▶ No such file or directory (os error 2)

Target //XX:race failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.145s, Critical Path: 0.12s
INFO: 12 processes: 4 action cache hit, 12 internal.
ERROR: Build did NOT complete successfully
XXX@XX:~/XX/workspace-1$ 

Something like the following fixes the issue, in that it doesn't chrash consistenly, but I have no real knowledge of the repo so it could have unknown side effects.

--- a/py/private/run.tmpl.sh
+++ b/py/private/run.tmpl.sh
@@ -37,13 +37,21 @@
 VENV_TOOL="$(rlocation {{VENV_TOOL}})"
-VIRTUAL_ENV="$(alocation "${RUNFILES_DIR}/{{ARG_VENV_NAME}}")"
+VENV_ROOT="$(alocation "${RUNFILES_DIR}")"
+VENV_TEMPLATE="${VENV_ROOT}/{{ARG_VENV_NAME}}.XXXXXX"
+VIRTUAL_ENV="$(mktemp -d "${VENV_TEMPLATE}")"
 export VIRTUAL_ENV
 
+cleanup() {
+  rm -rf "${VIRTUAL_ENV}"
+}
+
+trap cleanup EXIT
+
 "${VENV_TOOL}" \
     --location "${VIRTUAL_ENV}" \
     --python "$(python_location)" \
     --pth-file "$(rlocation {{ARG_PTH_FILE}})" \
     --collision-strategy "{{ARG_COLLISION_STRATEGY}}" \
-    --venv-name "{{ARG_VENV_NAME}}"
+    --venv-name "$(basename "${VIRTUAL_ENV}")"
 
 PATH="${VIRTUAL_ENV}/bin:${PATH}"
 export PATH