Utility to validate stubfile signatures against .py implementations

[dbarnett]

It would be useful to have utilities to validate stubfiles against the .py implementations to ensure the definitions are compatible and complete.

If a stubfile foo.pyi exists in a library the typecheckers seem to completely ignore the corresponding foo.py file (at least in the case of mypy), so the typechecker will spit out "no such attribute" errors if you try to reference definitions you add in foo.py but forget to add it in foo.pyi. It would be nice to have a validator for library maintainers to easily include in their library's test suites to make sure the .pyi is up to date and contains at least as much type information as the .py file.

Note this is geared towards library maintainers that want to maintain compatibility with python 3.4 and earlier. Otherwise you can simply embed the type hints directly in the .py and there's no duplicate definition to keep in sync.

[dbarnett]

Besides validating, there could be a fixer tool to add missing definitions to the stubfile, and if there are incompatible definitions I'd expect both tools to notice and warn you.

[Stewori]

Note on the use case:
Advertising this mainly for Python 3.4 feels somewhat disappointing or like an understatement at least.
I think there are use cases for stubfiles way beyond Python 3.4 compatibility. (Isn't, Python 3.4 EOL anyway?)
C-extensions and typeshed are other use cases and some people still feel that type annotations pollute Python code and thus prefer stubfiles.

A starting point may be https://github.com/Stewori/pytypes/blob/master/pytypes/stubfile_manager.py#L55
It recursively walks nested classes and I think it would cause an attribute error if the pyi contains classes that are missing in the py. It would have to be extended (i.e. under new name and without inserting the ._match_type) to check for other members and compare function/method signatures. Finally a roof function would call it twice, i.e. additionally with the roles of py and pyi swapped to assert totality.

Note:
The route to do this based on inspection might not be the most desirable one. As it has to import both modules (py and pyi) it is not suitable for untrusted code; but then Python isn't in general. A safer approach would be based on the AST module. However the inspection approach is much easier and closer to the current spirit of pytypes. For some reason I once decided to go the inspection route and literally everything in pytypes works inspection-based. So I'd suggest not to change this now. And btw it has the advantage that it also works with C-extension modules seamlessly.

[dbarnett]

That's true stubfiles aren't just for py 3.4 compatibility. I thought C-extensions wouldn't be feasible to cross-check and typeshed seemed "legacy" per comments like python/typeshed#3991 (comment). Anyway, sorry if I undersold the value.

Actually mypy does have a stubtest tool that seems related, but it doesn't seem to complain if I delete things from pyi or mess up signatures.

[Stewori]

I thought C-extensions wouldn't be feasible to cross-check

I think that should work based on inspection just like ordinary python modules, except that type annotations are only possible via stubfiles. But disclaimer: I did not actually test this so far.

typeshed seemed "legacy"

Honestly, I did not follow the fate of typeshed recently but I just had a quick look at their landing page and there is no deprecation statement whatsoever. I learned that mypy bundles typeshed, so it is much invisible to the user though. (Maybe I or we should consider how to improve typeshed integration for pytypes but that's again a different story.)

Actually mypy does have a stubtest tool that seems related

If you already figured it out, would you paste a usage instruction or ideally a small script that applies it to the upcoming pytypes' .pyi files? This may be a trivial thing but all these tiny things add up, and having it as a script to make it a true nobrainer helps to save some time...