Using your cuBLAS and cuDNN libraries in my app

[kaixxx]

Hi Purfview,
thank you very much making the cuBLAS and cuDNN libraries available as a standalone packed. I would like to use them in conjunction with my app which is also based on faster-whisper: https://github.com/kaixxx/noScribe

May I ask: How do you compile your binary so that people can use it with and without the cuda libraries installed?
If I compile with the non-cuda-version of PyTorch (using PyInstaller) and set 'device' to 'auto' in the faster-whisper initialization, my app crashes on devices with a cuda enabled graphics card. If I compile with the cuda-version of PyTorch, everything seems to work, but my app-size grows by several gigabytes. I think in this case, cuBLAS etc. is already included in the binary, right?
What I would like to do is having a single, small binary that runs without cuda on all devices but can be ‘upgraded’ to cuda by installing your libraries (given a suitable graphics card). Is this possible?
(I don’t have a NVIDIA graphics card, so it’s difficult for me to figure this out by trial and error.)

Thank you very much!
Kai

[Purfview]

your cuBLAS and cuDNN libraries

It's not "my" libs, it's NVIDIA's.

without cuda on all devices

Then set device to cpu by default.

I set to cpu by default only if CUDA is not found:

parser.add_argument("--device", "-d", default="cuda" if ctranslate2.get_cuda_device_count() > 0 else "cpu", help="")

And that's how I set device_index:

    # if cuda:1 -> device=cuda, device_index=1
    device = args.device
    device_index = 0

    if device.startswith('cuda:'):
        device, device_index = device.split(':')
        device_index = int(device_index)

[Purfview]

Do you compile your app with the cuda or the non-cuda version of PyTorch?

faster-whisper doesn't have such dependency,

[kaixxx]

You are right, I've mixed it up with pyannote that I use for diarization. Thank you again.

[Purfview]

With PyTorch you can make two separate builds one with CPU only and another with both CUDA and CPU,

Like standalone Whisper -> https://github.com/Purfview/whisper-standalone-win/releases/tag/Whisper-OpenAI

For example CPU only PyTorch install:

pip install torch==1.13.1+cpu torchvision==0.14.1+cpu torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cpu

[kaixxx]

Yeah, maybe I need to do that.
I already have two versions for macOS (Apple Silicon/Intel) and I would really like to avoid splitting up the windows version as well. Compiling, packing, testing and distributing all these different versions can get messy and annoying.
The other option would be to distribute the cuda version only, since it also works on the CPU. But this would bump the size of the installer to around 8GB which might be a bit excessive... I'll have to think about this.

[Purfview]

Compiling, packing, testing and distributing all these different versions can get messy and annoying.

Yeah, that's why mac/Linux builds I release every ~few months, only Windows gets the bleeding edge builds.