`.pth` files located in `app_packages` are not processed

[timrid]

Describe the bug

.pth that are located in the app_packages folder are not correctly processed on all platforms.

This leads to import errors with some packages. A prominent example is pywin32 (see here and here).

In addition, any Python code can be executed via .pth file at startup. This can also be used, for example, to establish a connection with a remote debugger (see here and here).

The following platforms are affected by this:

• Windows App
• Windows VisualStudio
• Linux Flatpak
• Linux System
• Linux AppImage (I guess, but I am not 100% sure. I could not test this so far.)
• macOS App
• macOS XCode
• iOS XCode
• Web

The following platforms are not affected by this:

• Android (but there is another bug as noted here)

Steps to reproduce

Add the following to pyproject.toml

...
[tool.briefcase.app.helloworld]
requires = [
    "git+https://github.com/timrid/pth-execution-checker@main"
]
...

and somewhere in the app call

import pth_execution_checker
if not pth_execution_checker.pth_executed:
    raise RuntimeError(".pth file not processed...")

Expected behavior

.pth files should be processed correctly on all platforms.

Screenshots

No response

Environment

• Operating System: Windows, Linux, macOS
• Python version: 3.13
• Software versions:
  • Briefcase: 0.3.22

Logs

Additional context

One solution would be to remove app and app_packages entirely and move all packages to site-packages, as described in #2195. But this was rejected by @freakboy3742. But he made the following suggestion (Link):

The site_packages folder isn't inherently special - it's just on the site list by default because it's a location tied to a the Python interpreter's libdir. The method site.addsitedir() exists to add additional site folders - I'm fairly certain the fix here is to use this method to add app_packages as a site dir, rather than just adding it to PYTHONPATH/sys.path as the stub is currently doing.

[timrid]

I have fixed this via calling site.addsitedir() as @freakboy3742 suggested for the following platforms:

• Windows VisualStudio: Add processing of .pth files from "app_packages". briefcase-windows-VisualStudio-template#52
• Linux Flatpak: Add processing of .pth files from "app_packages". briefcase-linux-flatpak-template#58
• Linux System: Add processing of .pth files from "app_packages". briefcase-linux-system-template#36
• iOS XCode: Add processing of .pth files from "app_packages". briefcase-iOS-Xcode-template#49
• macOS XCode: Add processing of .pth files from "app_packages". briefcase-macOS-Xcode-template#72

The following platforms requires a core developer to step in as noted here:

• Windows App
• macOS App

[timrid]

I also investigated a little bit into the web platform: There is no app_packages folder because the installation of the requirements and the app are done by pyscript/pyodide. So everything from app and app_packages is located in /lib/python3.12/site-packages, which is the default location that should be loaded by site.py. So everything should work as expected.

But I guess the problem is, that the packages are loaded via loadPackage which is called after loadPyodide as shown in this example from the pyodide docs:

let pyodide;
async function main() {
  pyodide = await loadPyodide();
  await pyodide.loadPackage("matplotlib");
  // matplotlib is now available
}
main();

So when the python interpreter is started and site.py is processed inside loadPyodide, I guess the site-packages folder is empty... So there are two options to get processing of .pth files:

1. The packages have to be available before the interpreter starts. I guess this has to be fixed in pyodide of pyscript.
2. Call site.addsitedir("/lib/python3.12/site-packages") again in index.html right before runpy.run_module is called. At that time all packages should be available.

I am unsure which one is the best...

[freakboy3742]

I am unsure which one is the best...

Thanks for that analysis - it's really thorough.

My initial inclination is that we will likely need to do both - (2) as an immediate workaround, and reporting (1) as a bug that needs to be fixed in Pyodide. Once (1) is resolved, we'll be able to revert (2); and in the brief period between the bug being fixed and us being able to roll out a template reversion, there shouldn't be any particularly drastic side effects to having the site-packages folder listed twice as a site dir.

I can't find an obvious existing bug in the Pyodide repo that covers this; I'll check with my Pyodide contacts to see if this is might be a known issue/limitation.

[freakboy3742]

I've just approved all the template modifications.

The next step is a PR on Briefcase that adds a removal changenote noting the change to PYTHONPATH ordering, and a bugfix changenote for #381. Once that is in place, we can merge the templates and the changenote PR, and then tag the macOS and Windows repos to get new binaries (which will fix the issue for the app templates).

This issue won't be closed at that point, because the issue will still exist for Android and Web, but #381 will be.

[freakboy3742]

I've spoken with the Pyodide folk - pyodide/pyodide#5539 was the result. That mechanism will allow injecting content into the emscripten filesystem prior to initialising the Pyodide interpreter, which will allow .pth handling to work.

[freakboy3742]

I've also logged pyscript/pyscript#2319; even if Pyodide fixes the issue, we'll need a fix in Pyscript to make use of it (and the external surface of any fix will also be different).

I've also merged all the templates, tagged the macOS and Windows stub binaries, and updated the app templates to include the new stubs:

• Bump the stub binary revision. briefcase-macOS-app-template#74
• Bump the stub binary revision. briefcase-windows-app-template#64

I've added a test to Python-support-testbed to validate .pth handling:

• Add a test of .pth handling. Python-support-testbed#118

Lastly, I've added #2212 to add Briefcase change notes for this set of changes.

[freakboy3742]

macOS, Linux, Windows and iOS changes have now landed; these platforms should now process .pth files correctly.

The remaining platforms:

• Android will correctly read and process .pth files; however, if you try to access the standard library in whatever logic is triggered by the .pth, that code will break. Fixing this requires a fix to Chaquopy (site.addsitedir order is incorrect chaquo/chaquopy#1338); this is on the radar for the next Chaquopy release.

• On Web, there's not currently any way to inject content prior to Pyodide initialisation (so .pth files can't be processed); and there's no way to reprocess a directory that is already on the site-packages path (so site.addsitedir() approaches won't work). This requires fixes upstream; see Add fsInit hook pyodide/pyodide#5539 and .pth and sitecustomize modules are not processed on interpreter startup pyscript/pyscript#2319.