Implement run-time Python interpreter searching

There is currently an assumption that the Python interpreter used while
building .pex files (i.e. the value of `DefaultInterpreter`) is also the
Python interpreter that runs them. This can be overridden with the
`shebang` parameter on individual `python_binary` and `python_test`
targets, which sets the shebang in the generated .pex file to a specific
value. The functionality this provides is limited:

- If additional interpreter arguments are specified by the target, the
  generated .pex file will depend on a shell at `/bin/sh` in the
  run-time environment (which isn't guaranteed to exist, e.g. in
  distroless containers), as well as `/usr/bin/env` (also not guaranteed
  to exist).
- It is possible to build the .pex file using a Python interpreter
  defined as a build target in the repo, but it isn't possible to run
  the .pex file using that interpreter because the .pex file's shebang
  is (by definition) static, but the interpreter's path relative to the
  .pex file's path varies depending on the context (e.g. whether the
  .pex file exists inside a build environment vs outside of one). This
  was discussed in please-build#247.

Provide more flexibility in how .pex files are run by prepending a
native-code preamble that implements dynamic searching of Python
interpreters. The behaviour of this preamble is controlled at run time
by a special zip member within the .pex file,
`.bootstrap/PLZ_PREAMBLE_CONFIG`, which contains a list of candidate
paths for Python interpreters under which the .pex file should be
executed and a list of arguments to pass to the interpreter when
attempting to execute it. The lists can be customised per
`python_binary` or `python_test` target via the `runtime_interpreters`
and `runtime_interpreter_args` parameters respectively; new plugin
configuration options, `DefaultRuntimeInterpreters` and
`DefaultRuntimeInterpreterArgs`, provide default values. If both are
unspecified, the plugin's current behaviour is maintained (i.e. the same
interpreter is used to both build and run the code). The flexibility
offered by the new .pex preamble is sufficient to be able to remove
`python_binary` and `python_test`'s `shebang` parameter (and the
`DefaultShebang` plugin configuration option) altogether.

The new .pex preamble is compact enough to prepend to all .pex files in
a repo without collectively making them prohibitively large: ~98KB
statically linked with musl on Linux, ~83KB dynamically linked on Darwin
(which doesn't have a stable kernel ABI, and strongly discourages static
linking).

Among other possibilities, this allows for the creation of .pex files
that are truly portable between run-time environments of the same
operating system and architecture - e.g., the preamble can be configured
to attempt to execute an interpreter that only exists within a
production environment (such as a distroless container with its
interpreter at `/python/bin/python3`), falling back on an interpreter
defined as a build target in the repo if the .pex file is still located
within Please's development/testing environment.

Author: chrisnovakovic

.github/workflows/plugin_test.yaml

@@ -28,6 +28,8 @@ jobs:
     runs-on: ${{ (inputs.platform == 'freebsd_amd64' || inputs.platform == 'linux_amd64') && 'ubuntu-24.04' || (inputs.platform == 'linux_arm64' && 'ubuntu-24.04-arm' || (inputs.platform == 'darwin_amd64' && 'macos-15-intel' || (inputs.platform == 'darwin_arm64' && 'macos-15' || 'unknown'))) }}
     steps:
       - name: Install Python in CI environment
+        # actions/setup-python doesn't support FreeBSD - instead, interpreters for the supported Python versions are
+        # pre-installed in the FreeBSD runner image.
         if: ${{ inputs.platform != 'freebsd_amd64' }}
         id: python
         uses: actions/setup-python@v6
@@ -36,14 +38,11 @@ jobs:
           update-environment: false
       - name: Check out code
         uses: actions/checkout@v5
-      - name: Configure plugin's default Python interpreter
-        run: |
-          _python_path="${{ inputs.platform == 'freebsd_amd64' && format('python{0}', inputs.python) || steps.python.outputs.python-path }}"
-          echo "PLZ_ARGS=-o plugin.python.defaultinterpreter:$_python_path" >> $GITHUB_ENV
       - name: Configure plugin to use please_pex tool built from source
         if: inputs.please_pex_from_repo
-        run: |
-          echo "PLZ_ARGS=${PLZ_ARGS:+$PLZ_ARGS }-o plugin.python.pextool://tools/please_pex" >> $GITHUB_ENV
+        # PexTool can't be set to //tools/please_pex for e2e tests because it can't be built inside the e2e test
+        # environment - skip the e2e tests if we're using //tools/please_pex as the pex tool.
+        run: echo "PLZ_ARGS=${PLZ_ARGS:+$PLZ_ARGS }-o plugin.python.pextool://tools/please_pex -e plz_e2e_test" >> $GITHUB_ENV
       - name: Run tests (in nested runner)
         if: ${{ inputs.platform == 'freebsd_amd64' }}
         uses: cross-platform-actions/[email protected]
@@ -55,11 +54,22 @@ jobs:
           shell: bash
           environment_variables: PLZ_ARGS
           shutdown_vm: true
-          run: ./pleasew test --keep_going --log_file plz-out/log/test.log ${{ inputs.test_targets }}
+          run: |
+            _external_interpreter="python${{ inputs.python }}"
+            _in_repo_interpreter="//third_party/cc/cpython:cpython_${{ inputs.python }}|python"
+            echo "*** Running tests using external Python interpreter: $_external_interpreter"
+            ./pleasew -o "plugin.python.defaultinterpreter:$_external_interpreter" test --rerun --keep_going --log_file plz-out/log/test-external_interpreter.log ${{ inputs.test_targets }}
+            echo "*** Running tests using in-repo Python interpreter: $_in_repo_interpreter"
+            ./pleasew -o "plugin.python.defaultinterpreter:$_in_repo_interpreter" test --rerun --keep_going --log_file plz-out/log/test-in_repo_interpreter.log ${{ inputs.test_targets }}
       - name: Run tests (on host)
         if: ${{ inputs.platform != 'freebsd_amd64' }}
         run: |
-          ./pleasew test --keep_going --log_file plz-out/log/test.log ${{ inputs.test_targets }}
+          _external_interpreter="${{ steps.python.outputs.python-path }}"
+          _in_repo_interpreter="//third_party/cc/cpython:cpython_${{ inputs.python }}|python"
+          echo "*** Running tests using external Python interpreter: $_external_interpreter"
+          ./pleasew -o "plugin.python.defaultinterpreter:$_external_interpreter" test --rerun --keep_going --log_file plz-out/log/test-external_interpreter.log ${{ inputs.test_targets }}
+          echo "*** Running tests using in-repo Python interpreter: $_in_repo_interpreter"
+          ./pleasew -o "plugin.python.defaultinterpreter:$_in_repo_interpreter" test --rerun --keep_going --log_file plz-out/log/test-in_repo_interpreter.log ${{ inputs.test_targets }}
       - name: Archive logs
         uses: actions/upload-artifact@v4
         with:

.plzconfig

@@ -1,11 +1,31 @@
 [Please]
-Version = >=17.21.0
+Version = >=17.25.1
 
 [Build]
 hashcheckers = sha256
 
 [Plugin "cc"]
 Target = //plugins:cc
+LdGarbageCollection = true
+
+DefaultOptCFlags = -std=c99
+DefaultOptCFlags = -Os
+DefaultOptCFlags = -DNDEBUG
+DefaultOptCFlags = -Wall
+DefaultOptCFlags = -Wextra
+DefaultOptCFlags = -Werror
+DefaultOptCFlags = -flto
+DefaultOptCFlags = -D_LARGEFILE64_SOURCE
+DefaultOptCFlags = -D_FILE_OFFSET_BITS=64
+
+DefaultDbgCFlags = -std=c99
+DefaultDbgCFlags = -g3
+DefaultDbgCFlags = -DDEBUG
+DefaultDbgCFlags = -Wall
+DefaultDbgCFlags = -Wextra
+DefaultDbgCFlags = -Werror
+DefaultDbgCFlags = -D_LARGEFILE64_SOURCE
+DefaultDbgCFlags = -D_FILE_OFFSET_BITS=64
 
 [Plugin "go"]
 Target = //plugins:go
@@ -21,6 +41,7 @@ Target = //plugins:shell
 
 [Plugin "e2e"]
 Target = //plugins:e2e
+PleaseVersion = 17.25.1
 DefaultPlugin = //test:python-rules
 
 [PluginDefinition]
@@ -30,25 +51,34 @@ name = python
 ConfigKey = DefaultInterpreter
 DefaultValue = python3
 Inherit = true
-Help = The Python interpreter to use for any case where one is not explicitly specified
+Help = The default Python interpreter to use when building .pex files.
 
-[PluginConfig "default_shebang"]
+[PluginConfig "interpreter_options"]
+ConfigKey = InterpreterOptions
+Optional = true
+Repeatable = true
+Help = A list of additional arguments to pass to the Python interpreter when building .pex files.
+
+[PluginConfig "default_runtime_interpreters"]
+ConfigKey = DefaultRuntimeInterpreters
+Optional = true
+Repeatable = true
 Inherit = true
+Help = A list of possible paths to Python interpreters with which to run .pex files.
+
+[PluginConfig "default_runtime_interpreter_args"]
+ConfigKey = DefaultRuntimeInterpreterArgs
 Optional = true
-Help = The shebang to set on binaries / tests where one is not explicitly specified
+Repeatable = true
+Inherit = true
+Help = A list of additional arguments to pass to the Python interpreter when running .pex files.
 
 [PluginConfig "pex_tool"]
 ConfigKey = PexTool
 DefaultValue = //tools:please_pex
 Inherit = true
 Help = A path or build label for the pex tool, which is used to create .pex files from python sources
 
-[PluginConfig "interpreter_options"]
-ConfigKey = InterpreterOptions
-Optional = true
-Repeatable = true
-Help = Additional command line arguments to pass to the Python interpreter.
-
 [PluginConfig "test_runner"]
 ConfigKey = TestRunner
 DefaultValue = unittest
@@ -146,6 +176,12 @@ DefaultValue = "info"
 Help = Passes verbosity level to wheel_tool, for example: wheel_tool --verbosity info
 Optional = true
 
+[PluginConfig "default_preamble_verbosity"]
+ConfigKey = DefaultPreambleVerbosity
+DefaultValue = error
+Inherit = true
+Help = The default logging level for the .pex preamble. Must be one of trace, debug, info, warn, error or fatal.
+
 [PluginConfig "feature_flags"]
 DefaultValue = ""
 Repeatable = true