Installing 8.5.0 with `--no-binary=:all:` results in `LookupError`

[DnPlas]

This is a very similar issue to #392, but for me it is happening on py3.8.

Steps to reproduce

(on py3.8) pip install --no-binary=:all: importlib-metadata

Relevant output log

(venv) dplascen@templates$: pip install --no-binary=:all: importlib-metadata
Collecting importlib-metadata
  Using cached importlib_metadata-8.5.0.tar.gz (55 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 2
  ╰─> [83 lines of output]
      Collecting setuptools>=61.2
        Using cached setuptools-75.3.2-py3-none-any.whl
      Collecting setuptools_scm>=3.4.1 (from setuptools_scm[toml]>=3.4.1)
        Using cached setuptools_scm-8.3.1.tar.gz (78 kB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting packaging>=20 (from setuptools_scm>=3.4.1->setuptools_scm[toml]>=3.4.1)
        Using cached packaging-25.0.tar.gz (165 kB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting typing-extensions (from setuptools_scm>=3.4.1->setuptools_scm[toml]>=3.4.1)
        Using cached typing_extensions-4.13.2.tar.gz (106 kB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting importlib-metadata>=4.6 (from setuptools_scm>=3.4.1->setuptools_scm[toml]>=3.4.1)
        Using cached importlib_metadata-8.5.0.tar.gz (55 kB)
      ERROR: Exception:
      Traceback (most recent call last):
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 106, in _run_wrapper
          status = _inner_run()
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 97, in _inner_run
          return self.run(options, args)
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
          return func(self, options, args)
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 386, in run
          requirement_set = resolver.resolve(
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
          result = self._result = resolver.resolve(
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
          state = resolution.resolve(requirements, max_rounds=max_rounds)
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve
          failure_causes = self._attempt_to_pin_criterion(name)
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
          criteria = self._get_updated_criteria(candidate)
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 230, in _get_updated_criteria
          self._add_to_criteria(criteria, requirement, parent=candidate)
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
          if not criterion.candidates:
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
          return bool(self._sequence)
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 174, in __bool__
          return any(self)
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 162, in <genexpr>
          return (c for c in iterator if id(c) not in self._incompatible_ids)
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 53, in _iter_built
          candidate = func()
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 187, in _make_candidate_from_link
          base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 233, in _make_base_candidate_from_link
          self._link_candidate_cache[link] = LinkCandidate(
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in __init__
          super().__init__(
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 159, in __init__
          self.dist = self._prepare()
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 236, in _prepare
          dist = self._prepare_distribution()
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 315, in _prepare_distribution
          return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 527, in prepare_linked_requirement
          return self._prepare_linked_requirement(req, parallel_builds)
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 642, in _prepare_linked_requirement
          dist = _get_prepared_distribution(
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 71, in _get_prepared_distribution
          with build_tracker.track(req, tracker_id):
        File "/opt/homebrew/Cellar/python@3.8/3.8.19/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
          return next(self.gen)
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/operations/build/build_tracker.py", line 136, in track
          self.add(req, tracker_id)
        File "/dplascen/venv/lib/python3.8/site-packages/pip/_internal/operations/build/build_tracker.py", line 102, in add
          raise LookupError(message)
      LookupError: https://files.pythonhosted.org/packages/cd/12/33e59336dca5be0c398a7482335911a33aa0e20776128f038019f1a95f1b/importlib_metadata-8.5.0.tar.gz (from https://pypi.org/simple/importlib-metadata/) (requires-python:>=3.8) is already being built: importlib-metadata from https://files.pythonhosted.org/packages/cd/12/33e59336dca5be0c398a7482335911a33aa0e20776128f038019f1a95f1b/importlib_metadata-8.5.0.tar.gz
      [end of output]

Additional context

pypa/setuptools-scm#1131 was recently reported and closed, and may be related.

[RonnyPfannschmidt]

It's related

I solved the loop for setuptools scm itself

But now the build env of importlib metadata requires importlib metadata

@jaraco is there a reasonable way to provide altered pyproject.toml inside a sdist

[RonnyPfannschmidt]

also i strongly recommend never to use source only installs unless you have full control of the build process

and arguably anyone hitting this issue outside of distro packaging is in fact not in control of the build process

[jaraco]

This problem is another manifestation of pypa/packaging-problems#342. Essentially, it's not possible for build backends (or their plugins) to have dependencies that use said build backend.

I've proposed several solutions, but none of which are without tradeoffs, and many of which push the bootstrapping problem onto those environments that wish to build everything from source.

I have one approach that I'm currently exploring in the coherent.build system that addresses the problem by producing sdists that build on a more primitive build backend (like flit). If I were to migrate importlib_metadata to coherent.build (with the experimental feature enabled) (or move to flit), the problem would go away, but for importlib_metadata only. I'd still have to apply the same treatment for every dependency of setuptools_scm (and Setuptools itself once I can get it to unvendor its dependencies).

Unfortunately, support for Python 3.8 is dropped, so there's little hope of these innovations addressing the issue on Python 3.8.

[jaraco]

There are many hacks that system integrators have used to work around the cyclic dependency issues, namely:

• avoiding use of setuptools_scm by hand-coding the version with SETUPTOOLS_SCM_PRETEND_VERSION (or similar)
• converting projects to flit before building

Another workaround that's been proposed - while bootstrapping the build backends (Setuptools+setuptools-scm in this case), ensure all of the (transitive) dependencies are present in the PYTHONPATH, either by pre-loading them from pre-built trusted installs or by manually assembling the Python modules from sources into an importable form, and then building the backend without build isolation.

It's a difficult problem to comprehend and an even more difficult problem to solve, which is why it's still broken. Good luck!

[RonnyPfannschmidt]

@abravalheri @jaraco can we expand setuptools sdist building in a way that allows somthing like setuptools_scm to reasonably replace build dependencies and dynamic fields

the idea would be to move the setuptools_scm dependency into something setuptools can omit for a sdist

same goes for file finer output

that way new sdists could ship whats needed in a more safe manner

[jaraco]

I think the short answer is no, mainly because Setuptools (and by extension every backend that has dependencies that lead to itself) would need to re-write the user's sources to produce the sdist. That is, if the user has declared build-requires=['setuptools-scm'], they've declared that for the build. PEP 518 and 517 doesn't provide enough granularity for the user to indicate build-wheel-requires=['setuptools-scm'] but build-sdist-requires[]. Therefore, to prevent an installer from rightfully attempting to install a cyclic dependency, the build-requires would have to be different for sources than for source distributions. Moreover, that's just a half-step toward what coherent.build is doing in compiling the sources to a different backend for sdists. It's true that Setuptools doesn't rely on setuptools-scm except at the build from source stage (it's no longer needed at the build from sdist stage), but that guarantee doesn't hold in general; it's a special case for setuptools-scm. If Setuptools is going to special-case something for setuptools-scm, it should be to integrate it. Otherwise, whatever solution is devised should apply to any dependency. The landscape is already complicated enough without carving out special cases for specific dependencies.

[RonnyPfannschmidt]

As build backends can report additional dependencies the idea is more around being able to skip some if the metadata is provided by the sdist instead of rewriting the code

[jaraco]

As build backends can report additional dependencies the idea is more around being able to skip some if the metadata is provided by the sdist instead of rewriting the code

Reading between the lines, I think you're suggesting:

• Setuptools-scm users are directed to stop specifying setuptools-scm in build-requires in pyproject.toml.
• Setuptools adds a feature flag to enable setuptools-scm.
• When a user specifies to enable setuptools-scm, setuptools will include 'setuptools-scm' in the response to get_build_requires_for_sdist but not get_build_requires_for_wheel.

That sounds like a lot of work and a very special case. Currently, setuptools relies on the presence or absence of the plugin to determine whether the feature is enabled or not. If setuptools-scm were to require a different paradigm for inclusion/enablement, it also has implications for the designs of other plugins.

In any case, this issue is not a great place to be having this discussion, since the issue relates to setuptools and setuptools-scm and the Python packaging ecosystem as a whole and has nothing to do with importlib_metadata other than that importlib_metadata happens to be caught in the problem because it happens to be a dependency of a build backend's plugin.

The only reason I'm not closing this issue is because I'm reserving it to track replacing the build system to side step the problem.

[RonnyPfannschmidt]

No id prefer to enable setuptools to provide dynamic requires depending on whether its running on a sdist or on a vcs checkout

But if that fails I'll work towards creating a separate build backend library in the dependency loop of build backends that can self build from its sdist and enable any sdist with matching tooling to do the same without loops

[RonnyPfannschmidt]

Additionally most users will not need any changes

But any build backend or dependeny of core build backends will change

[jaraco]

No id prefer to enable setuptools to provide dynamic requires depending on whether its running on a sdist or on a vcs checkout

But if that fails I'll work towards creating a separate build backend library in the dependency loop of build backends that can self build from its sdist and enable any sdist with matching tooling to do the same without loops

Please bring these ideas to packaging-problems or setuptools; I don't want to continue having packaging ecosystem problems in this issue, as this project is the wrong audience. This issue is about fixing the issue importlib_metadata specifically.