Scripts for producing PyPI-compatible manylinux wheel files #1028
Conversation
|
Turns out PyPI has file size limit of 100MB by default. There is a process to request the increase of file size limit. For instance, pytorch packages are ~800MB and they have successfully uploaded to PyPI. I have followed PyPI's suggested process for requesting limit increase. Reference to track progress: pypi/support#4520 |
|
update: the limit was increased this morning, and I have uploaded packages to PyPI, which are compatible with both CUDA and intel MKL backends. |
| # this wont work if pybind11 is git submodule | ||
| #find_package(pybind11 REQUIRED) | ||
|
|
||
| # NOTE: this property must be set before including pybind11 |
There was a problem hiding this comment.
This comment seems confusing given the commented line below. Please check if it's okay and explain in the comment or fix.
| "sacremoses", | ||
| "pyqt5", | ||
| "sentence-splitter@git+https://github.com/mediacloud/sentence-splitter", | ||
| # "sentence-splitter@git+https://github.com/mediacloud/sentence-splitter", |
There was a problem hiding this comment.
Please explain in the comment why it's commented (because not used yet but will be in pymarian-webapp?) or remove.
|
Moved this PR to internal fork. Leaving it open here in case somebody wants to build manylinux wheels and we should close this PR once the code is synced |
* pymarian: manylinux whl builder; * bind main() function. Upon pip install, a "pymarian" is made available which has same functionality as "marian" CLI. * fix github CI and devops CI with recent changes to cmake build for multiple python versions This PR was originally on public fork * #1029 * #1028
Description
While the cmake build produces *.whl files, they are not distributable via PyPI.
PyPI enforces certain rules to improve compatibility for different Linux distributions.
This PR adds scripts for producing pymarian wheel files that can be distributable on PyPI.
List of changes:
src/python/build.shandsrc/python/build-manylinux.shscripts.The former invokes
docker runwhereas the latter runs within docker env to create wheels for python version 3.8 to 3.12set(PYBIND11_NOPYTHON On)before adding pybind11huggingface-hubversion as the strict version causes conflicts with other libs (such as transformers) and also unavailable for some version of pythonAdded dependencies: require docker
How to test
Run
src/python/build.shto produce wheel files atbuild-python/manylinux/*.whlDescribe how you have tested your code, including OS and the cmake command.
Checklist