Improve UX and Performance of Install step

[notatallshaw]

What's the problem this feature will solve?

At the moment when the final install step starts pip gives no output what it is doing, in some real world cases (e.g. large pytorch installations or airflow installs) this steps can take over 30 seconds on fast machines, so minutes on slow machines. The user is left wondering if anything is happening.

Describe the solution you'd like

I would like to see the following improvements:

• 1. Log a message that pip is starting to install packages
• 2. Present a progress bar that tracks the number of packages installed out of the total packages to be installed
• 3. Improve any obvious performance bottlenecks (see follow up post with profile)
• 4. Run installs in parallel (made seperate issue Install packages in parallel #12742)

Alternative Solutions

I think at a bare minimum there should be a log message that lets the user know what's happening.

Additional context

uv runs installs in parallel, and following their issue tracker it does not appear to be problematic, to do this a cli option to control the maximum number would need to be added, the same as how there is a PR for parallel downloads to do this.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

This scenario is artifically constructed to best profile the installer code by removing the need to download, build sdists, or resolve:

1. python3.12 -m venv .venv
2. source .venv/bin/activate
3. <install latest/dev pip>
4. wget https://raw.githubusercontent.com/apache/airflow/constraints-2.9.1/constraints-3.12.txt
5. python -m pip download -d downloads -r constraints-3.12.txt
6. cd downloads
7. for file in $(ls *.tar.gz); do pip wheel --no-deps "$file" && mv "$file" "$file".built ; done
8. for file in $(ls *.zip); do pip wheel --no-deps "$file" && mv "$file" "$file".built ; done
9. cd -
10. python -m pip install --only-binary=:all: --no-index --ignore-installed --no-deps --find-links file://${PWD}/downloads -r constraints-3.12.txt

I ran with and without --dry-run to see the timing difference:
Dry Run: 32s
Regular Install: 144s

I profiled with and without --dry-run to see the profile difference:

Dry Run Profile

Regular Install Profile

There are some clear hotspots here, I will take a look when I have time if there are some easy ways to reduce those hotspots if no one else does.

[ichard26]

The get_dist_name() hot spot should be vastly improved by #12656 FWIW. I scheduled the PR for 24.2 as it feels a bit risky to ship in 24.1 final. Please say something if anyone feels differently.

[pfmoore]

I see no issues with the UI proposal, but I'd want parallel installs to be a separate feature. I can imagine pathological cases where things could break when installing in parallel, and while the experience of uv is encouraging (as is the fact that normal cases are clearly safe) my instinct is that every pathological case is being exercised by some user of pip, somewhere. So we should isolate the risk here by making it a separate feature.

[notatallshaw]

The get_dist_name() hot spot should be vastly improved by #12656 FWIW. I scheduled the PR for 24.2 as it feels a bit risky to ship in 24.1 final. Please say something if anyone feels differently.

Great, I'll reprofile with this PR. I personally wasn't imagining any of these ideas would land for 24.1.

I'd want parallel installs to be a separate feature

Agree, I'll make a seperate issue for that.

Honestly, the others I feel like I could make PRs that safely improve pip, I'm unsure about parralel installs, I think it would at a minimum carefully need to look at what current multiple installs tests there are and potentially expanding them to have a good matrix of different possibilities.

[notatallshaw]

Log a message that pip is starting to install packages

Btw, I was looking at this recently because I noticed pip does tell you it's installing packages. The specific scenario I was seeing was the following:

1. You install a large number of packages
2. You then install a large number of semi-overlapping packages

On step two this produces the following behavior:

1. Packages are resolved and pip tells you what packages it is going to install
2. Pip then quickly uninstalls old packages, filling up the screen
3. There is a long wait with no update on the screen while pip is installing
4. Pip then lists all packages it installed

The real world situaiton this happens is installing large machine learning packages, particularly because you install a bunch of packages from the pytorch index, and then install a bunch of packages from pypi.

I think there are a couple of possible solutions:

1. Re-order or add additional messages, e.g. move or add and "install" message after the uninstalls have completed
2. Add progress bars to both uninstalling packages and installing packages, so it's clear pip is doing things

I will take a look at PRs when I have a chance.

[ichard26]

Caching the result of utils.compatibility_tags.get_supported() in the resolver factory should be another easy win¹ (~3% or 4s in the example above)

pip/src/pip/_internal/resolution/resolvelib/factory.py

Lines 608 to 612 in 86b8b23

	return self._wheel_cache.get_cache_entry(
	link=link,
	package_name=name,
	supported_tags=get_supported(),
	)

I'll submit a PR when I get the chance.

Footnotes

1. I strongly suspect that get_supported() is only "slow" (as in, 1-5ms) on Linux due to the large amount of supported tags per system. ↩

[ichard26]

While taking a look at #12601, I was curious to how easy it would be to add an installation progress bar. The progress bar was pretty trivial to add by extending the pre-existing progress logic... However, it did not play nicely with the logging stack, so any intervening logs would break the progress bar. To fix this, I had to redo how rich was initialized in the logging stack which took a bit 🙂

Anyway, here's a demo:

Screencast.from.2024-07-15.22-31-35.webm

What do you think @notatallshaw?

• I'll note that your mental model for how pip installs packages is wrong. The uninstalls occur "on-demand" right before its replacement package is about to be installed (i.e. the uninstalls/installs are interwoven), so an uninstallation progress bar doesn't really make sense.
• I also chose to include the package currently being installed in the progress bar. Yes, in most situations, the per-package installation time is so low that most packages are never shown to the user (like in the demo), but there are exceptions. If we're installing some massive package, it'd be nice to let the user know we're stuck on $package. I don't feel strongly about this though so I'm fine dropping it.
• I chose to hide the progress bar after the installation step since the "successfully installed ..." printout is going to be emitted anyway.
• The progress bar does feel a bit choppy. This is intentional. I set the progress bar refresh rate to 5 times per second to minimize the added overhead. I don't want a fancy progress bar to add a ton of overhead, haha.¹

Footnotes

1. Ideally, the presentation logic would simply disable the progress bar outright when writing to a non-TTY, but that's a future thing to think about. ↩

[ichard26]

Hmm, it would definitely look less rough if I left-justified the package name. Here's another demo, but the package name is justified to the longest name length seen so far (as doing it properly feels like going against the API contract of pips' progress bars).

Screencast.from.2024-07-15.23-06-07.webm

It does kinda look weird. Perhaps after the bar?

Screencast.from.2024-07-15.23-11-40.webm

I think this looks the best out of all of them :)