Regression in 1.2.x when installing fasttext - can't compile due to not finding pybind and/or pip

[cfculhane]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 22.04 LTS

• Poetry version: 1.2.0b3

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/cfculhane/f8f9edee4af3ee99deab0170b814e72d

• Link to the lockfile: https://gist.github.com/cfculhane/aa093d4333e33d297a8da075d9d43e83

Issue

I've been enjoying the speed and other improvements to the 1.2.x betas, however one package is not installing, and it installs fine on 1.1.14 - fasttext.

Below is the log when installing in 1.2.0b3 with poetry install -vvv

(poetrybug-py3.8)  ~/repos/poetrybug      poetry install -vvv                                                                                                                              
Loading configuration file /home/chris/.config/pypoetry/config.toml
Using virtualenv: /home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8
Project environment contains an empty path in sys_path, ignoring.
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals, 3 skipped

  • Installing fasttext (0.9.2): Pending...
  • Installing numpy (1.23.1): Pending...
  • Installing numpy (1.23.1): Skipped for the following reason: Already installed
  • Installing pybind11 (2.10.0): Pending...
  • Installing pybind11 (2.10.0): Skipped for the following reason: Already installed
  • Installing setuptools (63.4.1): Pending...
  • Installing setuptools (63.4.1): Skipped for the following reason: Already installed
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
  • Installing fasttext (0.9.2): Installing...
  • Installing fasttext (0.9.2): Failed

  Stack trace:

  2  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1454 in _run
      1452│                 )
      1453│             else:
    → 1454│                 output = subprocess.check_output(
      1455│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1456│                 )

  1  ~/.pyenv/versions/3.8.11/lib/python3.8/subprocess.py:415 in check_output
       413│         kwargs['input'] = empty
       414│ 
    →  415│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
       416│                **kwargs).stdout
       417│ 

  CalledProcessError

  Command '['/home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8/bin/python', '/home/chris/.local/share/pypoetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-22.2.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8', '--no-deps', '/home/chris/.cache/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz']' returned non-zero exit status 1.

  at ~/.pyenv/versions/3.8.11/lib/python3.8/subprocess.py:516 in run
       512│             # We don't call process.wait() as .__exit__ does that for us.
       513│             raise
       514│         retcode = process.poll()
       515│         if check and retcode:
    →  516│             raise CalledProcessError(retcode, process.args,
       517│                                      output=stdout, stderr=stderr)
       518│     return CompletedProcess(process.args, retcode, stdout, stderr)
       519│ 
       520│ 

The following error occurred when trying to handle this error:


  Stack trace:

  3  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/pip.py:49 in pip_install
       47│ 
       48│     try:
    →  49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
       51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e

  2  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1417 in run_pip
      1415│         pip = self.get_pip_command(embedded=True)
      1416│         cmd = pip + list(args)
    → 1417│         return self._run(cmd, **kwargs)
      1418│ 
      1419│     def run_python_script(self, content: str, **kwargs: Any) -> int | str:

  1  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1707 in _run
      1705│     def _run(self, cmd: list[str], **kwargs: Any) -> int | str:
      1706│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
    → 1707│         return super()._run(cmd, **kwargs)
      1708│ 
      1709│     def get_temp_environ(

  EnvCommandError

  Command ['/home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8/bin/python', '/home/chris/.local/share/pypoetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-22.2.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8', '--no-deps', '/home/chris/.cache/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz'] errored with the following return code 1, and output: 
  Processing /home/chris/.cache/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    error: subprocess-exited-with-error
    
    × Getting requirements to build wheel did not run successfully.
    │ exit code: 1
    ╰─> [25 lines of output]
        /home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8/bin/python: No module named pip
        Traceback (most recent call last):
          File "<string>", line 38, in __init__
        ModuleNotFoundError: No module named 'pybind11'
        
        During handling of the above exception, another exception occurred:
        
        Traceback (most recent call last):
          File "/tmp/tmpoxq8b58i", line 363, in <module>
            main()
          File "/tmp/tmpoxq8b58i", line 345, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/tmp/tmpoxq8b58i", line 130, in get_requires_for_build_wheel
            return hook(config_settings)
          File "/tmp/pip-build-env-z78q_hl2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 177, in get_requires_for_build_wheel
            return self._get_build_requires(
          File "/tmp/pip-build-env-z78q_hl2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires
            self.run_setup()
          File "/tmp/pip-build-env-z78q_hl2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 281, in run_setup
            super(_BuildMetaLegacyBackend,
          File "/tmp/pip-build-env-z78q_hl2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 174, in run_setup
            exec(code, locals())
          File "<string>", line 72, in <module>
          File "<string>", line 41, in __init__
        RuntimeError: pybind11 install failed.
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  

  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1458 in _run
      1454│                 output = subprocess.check_output(
      1455│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1456│                 )
      1457│         except CalledProcessError as e:
    → 1458│             raise EnvCommandError(e, input=input_)
      1459│ 
      1460│         return decode(output)
      1461│ 
      1462│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  Stack trace:

  5  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:245 in _execute_operation
      243│ 
      244│             try:
    → 245│                 result = self._do_execute_operation(operation)
      246│             except EnvCommandError as e:
      247│                 if e.e.returncode == -2:

  4  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:317 in _do_execute_operation
      315│             return 0
      316│ 
    → 317│         result: int = getattr(self, f"_execute_{method}")(operation)
      318│ 
      319│         if result != 0:

  3  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:437 in _execute_install
      435│ 
      436│     def _execute_install(self, operation: Install | Update) -> int:
    → 437│         status_code = self._install(operation)
      438│ 
      439│         self._save_url_reference(operation)

  2  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:480 in _install
      478│         )
      479│         self._write(operation, message)
    → 480│         return self.pip_install(archive, upgrade=operation.job_type == "update")
      481│ 
      482│     def _update(self, operation: Install | Update) -> int:

  1  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:119 in pip_install
      117│     ) -> int:
      118│         try:
    → 119│             pip_install(req, self._env, upgrade=upgrade, editable=editable)
      120│         except EnvCommandError as e:
      121│             output = decode(e.e.output)

  PoetryException

  Failed to install /home/chris/.cache/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz

  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/pip.py:51 in pip_install
       47│ 
       48│     try:
       49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
    →  51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       52│ 

  • Installing numpy (1.23.1): Pending...
  • Installing numpy (1.23.1): Skipped for the following reason: Already installed
  • Installing pybind11 (2.10.0): Pending...
  • Installing pybind11 (2.10.0): Skipped for the following reason: Already installed
  • Installing setuptools (63.4.1): Pending...
  • Installing setuptools (63.4.1): Skipped for the following reason: Already installed

And the log when the same is run on 1.1.14:

 ~/repos/poetrybug2      poetry add fasttext -vvv                                                                                                                                          
Using virtualenv: /home/chris/.cache/pypoetry/virtualenvs/poetrybug2-KxUlQ7WG-py3.8
PyPI: No release information found for fasttext-0.1.0, skipping
PyPI: No release information found for fasttext-0.6.3, skipping
PyPI: 33 packages found for fasttext *
Using version ^0.9.2 for fasttext

Updating dependencies
Resolving dependencies...
   1: fact: poetrybug2 is 0.1.0
   1: derived: poetrybug2
   1: fact: poetrybug2 depends on fasttext (^0.9.2)
   1: selecting poetrybug2 (0.1.0)
   1: derived: fasttext (>=0.9.2,<0.10.0)
PyPI: No release information found for fasttext-0.1.0, skipping
PyPI: No release information found for fasttext-0.6.3, skipping
PyPI: 1 packages found for fasttext >=0.9.2,<0.10.0
   1: fact: fasttext (0.9.2) depends on pybind11 (>=2.2)
   1: fact: fasttext (0.9.2) depends on numpy (*)
   1: selecting fasttext (0.9.2)
   1: derived: numpy
   1: derived: pybind11 (>=2.2)
PyPI: No release information found for numpy-0.9.6, skipping
PyPI: No release information found for numpy-0.9.8, skipping
PyPI: No release information found for numpy-1.0.3, skipping
PyPI: No release information found for numpy-1.0.4, skipping
PyPI: No release information found for numpy-1.0b1, skipping
PyPI: No release information found for numpy-1.0b4, skipping
PyPI: No release information found for numpy-1.0b5, skipping
PyPI: No release information found for numpy-1.0rc1, skipping
PyPI: No release information found for numpy-1.0rc2, skipping
PyPI: No release information found for numpy-1.0rc3, skipping
PyPI: No release information found for numpy-1.1.1, skipping
PyPI: No release information found for numpy-1.2.0, skipping
PyPI: No release information found for numpy-1.2.1, skipping
PyPI: No release information found for numpy-1.4.0, skipping
PyPI: 87 packages found for numpy *
PyPI: No release information found for pybind11-1.6, skipping
PyPI: 22 packages found for pybind11 >=2.2
   1: selecting pybind11 (2.10.0)
PyPI: Getting info for numpy (1.23.1) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: numpy-1.23.1.tar.gz
   1: selecting numpy (1.23.1)
   1: Version solving took 5.889 seconds.
   1: Tried 1 solutions.

Writing lock file

Finding the necessary packages for the current system

Package operations: 3 installs, 0 updates, 0 removals

  • Installing numpy (1.23.1): Pending...
  • Installing numpy (1.23.1): Installing...
  • Installing numpy (1.23.1)
  • Installing pybind11 (2.10.0): Pending...
  • Installing pybind11 (2.10.0): Installing...
  • Installing pybind11 (2.10.0)
  • Installing fasttext (0.9.2): Pending...
  • Installing fasttext (0.9.2): Installing...
  • Installing fasttext (0.9.2)

Tried a few things:

• If I start a fresh project with 1.1.14 and add/install fasttext -> no error
• If I start a fresh project with 1.2.0b3 and add/install fasttext -> fails
• If I start with 1.1.14, then update to 1.2.0b3, and add/install fasttextt -> fails
• If I start with 1.2.0n3, then downgrade to 1.1.14, and add/install fasttext -> fails

Seems to be something with the temp env created for the build? Not quite sure, and it installs totally fine with pip.

[dimbleby]

seems to have been broken at #3835, which prevents the fasttext install process from finding the already-installed (and needed) pybind11

[dimbleby]

However fasttext is probably Doing It Wrong by not declaring pybind11 as a build-time requirement.

Probably you can and should fix this over there by pointing them at https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#build-system-requirement and / or submitting an MR.

[dimbleby]

I see that facebookresearch/fastText#1223 has been open for a while so I expect that you'll have to submit an MR if you want this fixed, rather than hoping that they take it on themselves.

[dimbleby]

I just noticed facebookresearch/fastText#1270, not clear whether this is blocked on no-one being interested or on the contributor signing the CLA.

Anyway poetry is not at fault here, this can be closed.

[cfculhane]

Thanks for looking into this - I will investigate a fix and submit a MR, apologies for taking the blame to poetry!

[juhoinkinen]

While waiting the issue to be fixed on fastText side, Poetry can install fastText using the prebuilt wheels available from fasttext-wheel PyPI project.