Speed up build environment creation (re-usable, faster)

[gaborbernat]

Currently pip always creates all build environments from scratch, requiring network access for this. This causes a significant overhead of creating isolated build environments. Can we somehow speed-up this operation?

Idea: Creating isolated build environments from cached wheels only? Do not recreate isolated build environments from scratch, but have a master copy that we just copy.

[chrahunt]

Having a concrete target can help. Can you provide an example of a slow command, the time it took, and how long you're expecting it to take?

The network access issue can be overcome by invoking pip like pip install --no-index --find-links path/to/deps pkg after a previous invocation of pip wheel -w path/to/deps pkg. That should also remove build environment setup overhead since everything is built by the pip wheel step.

[gaborbernat]

My issue raised here was exactly that pip wheel builds a build environment every-time from scratch via the netwrok; which is quite expensive and could be avoided.

python3.8 -m pip wheel -w . --no-deps .

Looking in indexes: http://localhost:3141/bernat/bb
Processing /Users/bgabor8/git/github/tox
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: tox
  Building wheel for tox (PEP 517) ... done
  Created wheel for tox: filename=tox-3.13.3.dev23+g6cb400e-py2.py3-none-any.whl size=80760 sha256=8cff1ac6b9a01ceac6a15d5d0c415a7d9414a847f866e414a6a224eb8ef9e31a
  Stored in directory: /Users/bgabor8/git/github/tox
Successfully built tox

env PIP_INDEX_URL=12321 python3.8 -m pip wheel -w . --no-deps . 

Looking in indexes: 12321
Processing /Users/bgabor8/git/github/tox
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/bgabor8/.pyenv/versions/3.8.0/bin/python3.8 /Users/bgabor8/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-build-env-cfk5t5ih/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i 12321 -- 'setuptools >= 40.0.4' 'setuptools_scm >= 2.0.0, <4' 'wheel >= 0.29.0'
       cwd: None
  Complete output (4 lines):
  Looking in indexes: 12321
  WARNING: Url '12321/setuptools/' is ignored. It is either a non-existing path or lacks a specific scheme.
  ERROR: Could not find a version that satisfies the requirement setuptools>=40.0.4 (from versions: none)
  ERROR: No matching distribution found for setuptools>=40.0.4
  ----------------------------------------

[chrahunt]

Thanks for the example! On my machine running this script I'm showing it takes about two seconds to build tox:

script.sh

#!/bin/sh
cd "$(mktemp -d)"
python -m venv venv
PYTHON="$PWD/venv/bin/python"
"$PYTHON" -V

git clone https://github.com/tox-dev/tox.git
cd tox
time "$PYTHON" -m pip wheel -w . --no-deps .
time "$PYTHON" -m pip wheel -w . --no-deps .
time "$PYTHON" -m pip wheel -w . --no-deps .

Output

$ ./repro.sh
Python 3.8.0
Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-19.3.1
Cloning into 'tox'...
remote: Enumerating objects: 10807, done.
remote: Total 10807 (delta 0), reused 0 (delta 0), pack-reused 10807
Receiving objects: 100% (10807/10807), 10.30 MiB | 37.03 MiB/s, done.
Resolving deltas: 100% (3663/3663), done.
Processing /tmp/user/1000/tmp.hiF75TCOBr/tox
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: tox
  Building wheel for tox (PEP 517) ... done
  Created wheel for tox: filename=tox-3.14.1.dev6+g3980da7-py2.py3-none-any.whl size=80773 sha256=c0200254a4e10b86c1f1e17dbcb17f42a24202c08d0ff8a5227db6e29cb7ac40
  Stored in directory: /tmp/user/1000/tmp.hiF75TCOBr/tox
Successfully built tox
1.82user 0.19system 0:02.03elapsed 99%CPU (0avgtext+0avgdata 48488maxresident)k
352inputs+42280outputs (0major+112397minor)pagefaults 0swaps
Processing /tmp/user/1000/tmp.hiF75TCOBr/tox
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: tox
  Building wheel for tox (PEP 517) ... done
  Created wheel for tox: filename=tox-3.14.1.dev6+g3980da7-py2.py3-none-any.whl size=80773 sha256=d258f3485f842e82b8ba4ac05b69e2d057ec12477af0bf1969ec4b42c77a65d3
  Stored in directory: /tmp/user/1000/tmp.hiF75TCOBr/tox
Successfully built tox
1.74user 0.21system 0:01.95elapsed 100%CPU (0avgtext+0avgdata 48484maxresident)k
0inputs+42400outputs (0major+112203minor)pagefaults 0swaps
Processing /tmp/user/1000/tmp.hiF75TCOBr/tox
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: tox
  Building wheel for tox (PEP 517) ... done
  Created wheel for tox: filename=tox-3.14.1.dev6+g3980da7-py2.py3-none-any.whl size=80773 sha256=fb850196ad8cd3449fb16b6e2d812c4c54a7fd16a5036f766b6562710a2c4703
  Stored in directory: /tmp/user/1000/tmp.hiF75TCOBr/tox
Successfully built tox
1.75user 0.23system 0:01.98elapsed 100%CPU (0avgtext+0avgdata 48588maxresident)k
0inputs+42400outputs (0major+112101minor)pagefaults 0swaps

The important lines are:

• 1.82user 0.19system 0:02.03elapsed 99%CPU (0avgtext+0avgdata 48488maxresident)k
  352inputs+42280outputs (0major+112397minor)pagefaults 0swaps
• 1.74user 0.21system 0:01.95elapsed 100%CPU (0avgtext+0avgdata 48484maxresident)k
  0inputs+42400outputs (0major+112203minor)pagefaults 0swaps
• 1.75user 0.23system 0:01.98elapsed 100%CPU (0avgtext+0avgdata 48588maxresident)k
  0inputs+42400outputs (0major+112101minor)pagefaults 0swaps

How long does it take for you? If you're seeing much longer times, it may be caused by something other than the build environment setup (e.g. #2195). If you run pip with --log pip.log and post it here (attached as a file if it is too long for a comment) then we can help identify what is taking the most time. That should give us a concrete idea about how much the build environment setup factors in to the overall build time.

Sorry if I'm asking something that seems obvious, I'm trying to cover all bases so we can say without a doubt that a change in this area would help and identify how much it would help. That helps with prioritization since there are several existing issues related to pip being slow to install (#4768, #4497 kind of, #825).

[pradyunsg]

(just noting -- @gaborbernat is a virtualenv maintainer)

The build environments could be created from cached wheels -- optimizing things to minimize network requests is a good idea. However as #7132 and similar show, it is kinda tricky to do. Note that these are intended as optimizations, so we'd want to not have subtle+nuanced behavior changes.

If someone wants to implement the approach I propose there and a way for pip to aggressively use the cache, that'd be a good way to avoid network requests in this specific scenario.

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.