PyPI mirror in enclosed environments

[stohrendorf]

Found some issues regarding "private" PyPI mirrors for pypi.org, especially regarding environments where the build environment is restricted such that it can't access anything outside of a corporate network and has to go through some sort of proxy/mirror with a different URL, and where people want to define that mirror globally instead of within pyproject.toml (sadly, can't provide any issues/discussions here right now since I'm in a different environment and discussions about this specific issue seem to be sparse). Most of these issues contained some discussion about authentification, but I doubt that a mirror should require any authentification at all. If you call it a mirror, it should be a mirror. Arguments were made that when you change URLs, it would lead to possibly non-reproducible lockfiles. That is acceptable. However, I think it is possible to globally redirect pypi.org/pythonhosted.org requests to a company mirror. Here's what I'm thinking about.

The assumptions:

• If you call it a mirror, it should be a mirror. As such, if you replace pypi.org or pythonhosted.org with some other domain, behaviour/responses must not change. (I'm not sure how poetry would behave if the mirror would block/remove packages for some reason. If it affects poetry, that's an issue, but let's assume it's a true mirror without package removal.)
• The mirror does not require any authorization.

This leads to the following:

• You could switch the domain (and even path) on request level without any problems.
• As it's a 1:1 mirror, the integrity of poetry.lock is guaranteed inside and outside of a restricted environment.
• If the mirror does, for some reason, modify requests, the behaviour of poetry and the integrity of poetry.lock is undefined. I.e., it might work outside the restriced environment, but it is neither a guaranteed nor defined behaviour.
• As such, you could define a mirror within a restricted environment as this: if a request is made to pythonhosted.org, and a mirror is configured (either within pyproject.toml or globally), the request path from the request to pythonhosted.org is taken and appended to the provided mirror URL. E.g., if poetry would do a request for https://files.pythonhosted.org/packages/c6/85/..., and it would have a configured mirror as https://company.xyz/mirror/, it would result in an effective mirror URL of https://company.xyz/mirror/packages/c6/85/...

Disclaimer:

• I'm not familiar with the domains/URLs involved regarding package search or downloads. Thus, I'm somewhat sure it will be more complicated than my simple idea.
• Looking at the code, I think it would be possible to do such URL substitutions, but I couldn't figure out what problems could arise.

Possible issues:

• Maintainability: This would add another feature requiring maintenance, and it seems to be a possible solution for too few people. As such, the benefits are questionable.
• Implementation: I think it would take quite some effort to even implement a first version of this. Even while it is "simple URL re-writing," it needs to go from the most abstract interface (CLI/global config) down to the deepest level of firing requests.

[radoering]

Related issues: #1632 and #5958

Replacing PyPI with a mirror should be easier than defining arbitrary custom sources globally because we do not include the URL of PyPI in the lock file. Actually, there is already a plugin that allows to replace PyPI: https://github.com/arcesium/poetry-plugin-pypi-mirror/

[stohrendorf]

Thanks for referencing these issues, these were the ones I wanted to reference, too.

Regarding the plugin, I'm aware of that, and it is (as of right now) a valid alternative compared to modifying/extending poetry itself. However, that plugin does not seem to be used very widely (it has only 45 stars, which is a strong indicator for that), and as such, there's a high risk that it will not be maintained some time in the future, and if that's the case, you're at some time forced to use an outdated version of poetry that's compatible with the plugin.