High CPU usage with MustInstall/MustInstallAll during parallel downloads

[ndyanx]

I've observed significantly higher CPU usage when using MustInstall() or MustInstallAll() compared to using a system-installed yt-dlp. This becomes particularly noticeable during parallel downloads.

without MustInstall or MustInstallAll

with

Is this expected behavior?

[lrstanley]

Both Mustinstall (just yt-dlp), or MustInstallAll (yt-dlp, ffmpeg, ffprobe) would definitely use a good amount of CPU during download, as it's actively decompressing data from archives and also doing shasum validation (to ensure the thing we're downloading was actually published by the yt-dlp folks). MustInstallAll in particular concurrently downloads and decompresses all three. Network and disk usage would also drive up CPU to some level, during those stages, but I suspect it's specifically the decompressing that you're seeing.

Worth noting that if you have multiple, separated processes, which are invoking MustInstall, and each of those invokes MustInstall* methods, they would all be trying to download the same artifacts at the same time, and you should have some external locking to prevent cross-process concurrency issues, as I think that's out of the scope of go-ytdlp. This wouldn't be an issue if the invocation of go-ytdlp is in one process, as MustInstall* methods have locking to prevent concurrent invocation.

If you're referring to CPU usage with the version of yt-dlp that go-ytdlp downloads (AFTER MustInstall* invocations), there is nothing different I do to invoke the cached versions downloaded through MustInstall* vs system-installed versions, other than we add the cache folder to PATH, so it invokes the version we cached. It's possible the system-installed binary you have is different from the ones which we download (something seems slightly different, as the icons shown in those .exe's in the screenshot is different). For example, on Windows x64, this is the binary we download:

https://github.com/yt-dlp/yt-dlp/releases/download/2025.07.21/yt-dlp.exe

Also, have you confirmed that the system installed yt-dlp version is the exact same version matching that of the version of go-ytdlp you're using?

[ndyanx]

I only call MustInstall once at application startup, not repeatedly.

System Version: I'm using yt-dlp installed via pip (PyPI) at version yt-dlp which works correctly with go-ytdlp. without MustInstall

Observed Behavior: The higher CPU usage persists even after the initial installation phase, during normal download operations. All my tests so far have been downloading only images, not even videos.

example code

[lrstanley]

System Version: I'm using yt-dlp installed via pip (PyPI) at version yt-dlp which works correctly with go-ytdlp.

Specifically, the version difference. E.g. what go-ytdlp tag are you using, and what yt-dlp version (number) is being used by PyPI? Can use that to find the version difference. If there is a difference, you could try seeing if there is any difference by matching the versions (e.g. upgrading/downgrading the Python version to match that of go-ytdlp).

I could see the version through PyPI being faster for a few reasons:

1. through PyPI probably uses Python itself during runtime, and thus Python caches through pyc files. Not sure what python -> exe compiler the yt-dlp team uses, but the approach used could mean it's not as "efficient" during code execution.
2. Security scanning -- Python might be excluded by default through your security scanner (which is the actual true process it's running under), where as yt-dlp.exe binaries might not be, and thus depending on what security scanner is used, it can hook into all system calls done by that binary, slowing it down/causing additional overhead.
   • ref: Long Start-Up Time for yt-dlp_macos yt-dlp/yt-dlp#10425 (comment)

   • Every issue that's been opened on the pyinstaller tracker about slow startup time (and there are several) has always reached a conclusion of "antivirus/security scanning interference" (or else "don't use --onefile", which is not a solution here).

In either case, there is nothing go-ytdlp can do, or is doing special from the normal binary downloads for yt-dlp.exe. If you think it's worth reporting upstream to yt-dlp, you can run your go-ytdlp based app with an env var, specifically YTDLP_DEBUG=true, which will print the exact yt-dlp commands it is running to stdout, so you can see what exactly is being invoked, and replicate that manually in both to confirm.

[lrstanley]

For additional context, on linux for example, with the yt-dlp_linux binaries, I see:

1. the binary temporarily unwraps itself and runs under a /tmp/ dir (bindata style exe's), so there is a tiny CPU spike for the first 400-500ms while it decompresses itself into that directory. I think this is a pyinstaller thing and how it can make self-contained binaries.
2. It then uses 0.1%-2% CPU (single core) to download 4 relatively short videos (30s long each).

[lrstanley]

Also worth noting that I think most of the CPU you would probably see is just initial with those yt-dlp binaries, and if you're downloading screenshots, that technically would probably be 99% of the process run since the download would be so fast. For videos, it should theoretically quickly fall off and use almost no CPU. I think it's an inherent flaw to pyinstaller style binaries created from Python projects.