How to create two dependency groups where dependencies are all URLs

[fernandobperezm]

Hello,

I'm trying to install torch on my laptop (macOS M1) and two linux servers (one with CPU and the other with GPU). I'm trying to have a single pyproject.toml and handle the dependencies using groups as follows:

[tool.poetry.dependencies]
python = "~3.10"

[tool.poetry.group.torchcpu]
optional = true

[tool.poetry.group.torchcpu.dependencies]
torch = [
    {platform = 'linux', url="https://download.pytorch.org/whl/cpu/torch-1.13.1%2Bcpu-cp310-cp310-linux_x86_64.whl"},
    {platform = 'darwin', markers = "platform_machine == 'arm64'", url="https://download.pytorch.org/whl/cpu/torch-1.13.1-cp310-none-macosx_11_0_arm64.whl"},
]

[tool.poetry.group.torchcu116]
optional = true

[tool.poetry.group.torchcu116.dependencies]
torch = [
    {platform = 'linux', url="https://download.pytorch.org/whl/cu116/torch-1.13.1%2Bcu116-cp310-cp310-linux_x86_64.whl"},
]

What I want to achieve is that on installation, I decide whether to install the CPU versions (laptop or server with only CPU) or the GPU version (server with GPU). I tried following the comments on 6409 6409-2 with no luck.

My installation command on macOS and linux+cpu server is:
poetry install --with torchcpu --sync

On the GPU server is:
poetry install --with torchcu116 --sync

However, anytime I run those commands it fails with the following error:
... depends on both torch (1.13.1) @ https://download.pytorch.org/whl/cpu/torch-1.13.1-cp310-none-macosx_11_0_arm64.whl and torch (1.13.1+cu116) @ https://download.pytorch.org/whl/cu116/torch-1.13.1%2Bcu116-cp310-cp310-linux_x86_64.whl, version solving failed.

Is there a way to achieve what I want with Poetry?

Thanks in advance!

[mquillot]

I am asking a similar question. I would like to let the user choose which cuda version they would like to use. cuda 11.8, coda 12.4 etc. I tried the following:

[tool.poetry.dependencies]
python = "^3.9"

torch = [
    { version = "^2.4.0", source = "pytorch_cu124", markers = "extra == 'cuda124'" },
    { version = "^2.4.0", source = "pytorch_cu118", markers = "extra == 'cuda118'" }
]
torchaudio = [
    { version = "^2.4.0", source = "pytorch_cu124", markers = "extra == 'cuda124'" },
    { version = "^2.4.0", source = "pytorch_cu118", markers = "extra == 'cuda118'" }
]


[tool.poetry.extras]
cuda124 = []
cuda118 = []



[[tool.poetry.source]]
name = "pytorch_cu124"
url = "https://download.pytorch.org/whl/cu124"
priority = "explicit"

[[tool.poetry.source]]
name = "pytorch_cu118"
url = "https://download.pytorch.org/whl/cu118"
priority = "explicit"

Then, I run the following command:

poetry install --extras cuda124

But it will respond that I have incompatible constraints:

Incompatible constraints in requirements of speech-tools (0.1.0):
torch (>=2.4.0,<3.0.0) ; extra == "cuda118" ; source=pytorch_cu118
torch (>=2.4.0,<3.0.0) ; extra == "cuda124" ; source=pytorch_cu124

The only solution I found for now is to not include torch on the dependencies and to let the user to install it but adding it to its own requirement file or pyproject file.