Prioritize core packages when included in lock file

[mbrancato]

I've noticed several issues which are related to EnvCommandError or CalledProcessError. Many of these seem to be an issue that I have experienced and can reproduce related to a race condition with core packages. When packages like setuptools, wheel, or pip are included in the lock file, commands like poetry update and poetry install can fail when performing installations. The process seems to be related to parallel install workers. When, for example, the package pip is being updated, other workers calling pip will fail.

This can be resolved by setting POETRY_INSTALLER_MAX_WORKERS=1 on the env. Part of the issue here is that the packages do not explicitly call out a dependency on pip or others, and when poetry builds a graph for ordering installation, it is not taking into account its own use of pip and others to install the packages.

Example:

  - Updating pip (23.3.2 -> 24.3.1)
  - Updating psutil (5.9.7 -> 5.9.8)
  - Updating pyproject-api (1.6.1 -> 1.8.0)
  - Updating pytest (7.4.0 -> 8.3.4): Failed

  CalledProcessError

  Command '['/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/bin/python', '-m', 'pip', 'uninstall', 'pytest', '-y']' returned non-zero exit status 2.

  at ~/.pyenv/versions/3.11.10/lib/python3.11/subprocess.py:571 in run
       567│             # We don't call process.wait() as .__exit__ does that for us.
       568│             raise
       569│         retcode = process.poll()
       570│         if check and retcode:
    →  571│             raise CalledProcessError(retcode, process.args,
       572│                                      output=stdout, stderr=stderr)
       573│     return CompletedProcess(process.args, retcode, stdout, stderr)
       574│ 
       575│ 

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/bin/python', '-m', 'pip', 'uninstall', 'pytest', '-y'] errored with the following return code 2
  
  Output:
  ERROR: Exception:
  Traceback (most recent call last):
    File "/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    File "/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/_internal/commands/uninstall.py", line 65, in run
    File "/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 95, in get_default_session
    File "/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 122, in _build_session
    File "/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/_internal/network/session.py", line 342, in __init__
    File "/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/_internal/network/session.py", line 175, in user_agent
    File "/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/_internal/metadata/importlib/_envs.py", line 189, in get_distribution
    File "/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/_internal/metadata/importlib/_envs.py", line 184, in <genexpr>
    File "/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/_internal/metadata/base.py", line 626, in iter_all_distributions
    File "/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/_internal/metadata/importlib/_envs.py", line 177, in _iter_distributions
    File "/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs
    File "/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/_internal/metadata/importlib/_envs.py", line 113, in _find_eggs_in_dir
  ImportError: cannot import name 'pkg_resources' from 'pip._internal.metadata' (/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/_internal/metadata/__init__.py)
  

  at ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/utils/env/base_env.py:342 in _run
      338│                 output = subprocess.check_output(
      339│                     cmd, stderr=stderr, env=env, text=True, encoding=encoding, **kwargs
      340│                 )
      341│         except CalledProcessError as e:
    → 342│             raise EnvCommandError(e)
      343│ 
      344│         return output
      345│ 
      346│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

Cannot install pytest.

With only 1 worker, this does not happen:

  - Updating pip (23.3.2 -> 24.3.1)
  - Updating psutil (5.9.7 -> 5.9.8)
  - Updating pyproject-api (1.6.1 -> 1.8.0)
  - Updating pytest (7.4.0 -> 8.3.4)
  - Updating rich (13.5.2 -> 13.9.4)

[abn]

@mbrancato looks like you have poetry installed in the same environment that you are managing. This is bad. And in any case, and Poetry does not use the pip or packaging from within the project's virtual env (#2826).

[abn]

It is for the very issue you have encountered, we recommend you install Poetry in isolated environments and not to mix your project and poetry envs.

https://python-poetry.org/docs/

Poetry should always be installed in a dedicated virtual environment to isolate it from the rest of your system. Each of the above described installation methods ensures that. It should in no case be installed in the environment of the project that is to be managed by Poetry. This ensures that Poetry’s own dependencies will not be accidentally upgraded or uninstalled. In addition, the isolated virtual environment in which poetry is installed should not be activated for running poetry commands.

[mbrancato]

@abn

Are you sure that is the issue? I don't have poetry installed in the env I'm managing and the poetry binary points to its own venv.

% which poetry
/Users/mike/.local/bin/poetry
% ls -la /Users/mike/.local/bin/poetry
lrwxr-xr-x@ 1 mike  staff  64 Jan 19 11:15 /Users/mike/.local/bin/poetry -> /Users/mike/Library/Application Support/pypoetry/venv/bin/poetry
% python -m pip freeze | grep poetry
% 

[dimbleby]

poetry still uses pip to uninstall packages

pip does not expose any stable API so depending on it in your project is almost certainly not what you want. The answer is therefore likely just: don't do that.

[abn]

Are you sure that is the issue? I don't have poetry installed in the env I'm managing and the poetry binary points to its own venv.

Maybe not. The error you posted is very common in the scenario I mentioned. Maybe I was quick to assume, my bad.

@dimbleby should that be using the intercepted pip (ie. from Poetry's system env?) Will check a bit later if that has changed somehow.

Edit: Looks like I switched to "if pip is available in the environment use it" behaviour in #4011.

[abn]

@mbrancato #10091 should resolve the issue you are facing. Would be good if you can test it out.

Can you please try the fix at #PULL_ID.

Using pipx

pipx install --suffix=@10091 'poetry @ git+https://github.com/python-poetry/poetry.git@refs/pull/10091/head'
alias poetry=poetry@10091
poetry install

Using a container (podman | docker)

podman run --rm -i --entrypoint bash python:latest <<EOF
set -xe
python -m pip install --disable-pip-version-check -q git+https://github.com/python-poetry/poetry.git@refs/pull/10091/head
...
poetry install
EOF

[mbrancato]

@abn

Thanks, I installed it as requested, but it still seems to call pip from the local env for uninstall. I installed pipx with. homebrew and installed it using pipx, which I normally do not do because I didn't have pipx installled.

I normally need to add this to support a private repo:

poetry self add keyrings.google-artifactregistry-auth

But that too was failing due to an attempt to downgrade cffi. I had to run this to prevent the downgrade.

poetry self add cffi==1.17.1

output from poetry@10091 install:

  - Updating pytest (7.4.0 -> 8.3.4)
  - Updating rich (13.5.2 -> 13.9.4)
  - Updating setuptools (75.2.0 -> 75.8.0)
  - Updating types-pyopenssl (23.3.0.20240106 -> 24.1.0.20240722)

  CalledProcessError

  Command '['/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/bin/python', '-m', 'pip', 'uninstall', 'types-pyopenssl', '-y']' returned non-zero exit status 1.

  at /opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py:577 in run
       573│             # We don't call process.wait() as .__exit__ does that for us.
       574│             raise
       575│         retcode = process.poll()
       576│         if check and retcode:
    →  577│             raise CalledProcessError(retcode, process.args,
       578│                                      output=stdout, stderr=stderr)
       579│     return CompletedProcess(process.args, retcode, stdout, stderr)
       580│ 
       581│ 

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/bin/python', '-m', 'pip', 'uninstall', 'types-pyopenssl', '-y'] errored with the following return code 1
  
  Output:
  Traceback (most recent call last):
    File "<frozen runpy>", line 198, in _run_module_as_main
    File "<frozen runpy>", line 88, in _run_code
    File "/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/__main__.py", line 22, in <module>
      from pip._internal.cli.main import main as _main
    File "/Users/mike/.pyenv/versions/3.11.10/envs/test-3.11/lib/python3.11/site-packages/pip/_internal/__init__.py", line 3, in <module>
      from pip._internal.utils import _log
  ModuleNotFoundError: No module named 'pip._internal.utils'
  

  at ~/.local/pipx/venvs/poetry@10091/lib/python3.13/site-packages/poetry/utils/env/base_env.py:342 in _run
      338│                 output = subprocess.check_output(
      339│                     cmd, stderr=stderr, env=env, text=True, encoding=encoding, **kwargs
      340│                 )
      341│         except CalledProcessError as e:
    → 342│             raise EnvCommandError(e)
      343│ 
      344│         return output
      345│ 
      346│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

Cannot install types-pyopenssl.

[abn]

For completeness, the regression was addressed in #10102.