Multi-channel input for annotate_with_whisper

[WangHelin1997]

Hi,
I found that only mono input is supported now for the annotate_with_whisper function. Will the multi-channel input work in the future? Or I only need to keep the first channel, what should I do? Thanks so much.

            logging.warning(
                f"Skipping recording '{recording.id}'. It has {recording.num_channels} channels, "
                f"but we currently only support mono input."
            )
            continue

[pzelasko]

I think it should be straightforward to modify the code to iterate over every channel in a recording, create supervisions specifically for that channel, and bind it all together into a MultiCut. Would you be willing to make a PR with these improvements?

[pzelasko]

CC maybe @desh2608 would also be interested

[desh2608]

Yeah, that should be the most straightforward approach. If all channels are different instances of the same speech (i.e. same audio recorded with different mics, instead of 2 channels of a telephone conversations, for example), you can also just pick the first channel to transcribe and share the supervision for all channels. This would save a lot of compute.

[WangHelin1997]

Thanks for your help.
In addition, I found that annotate_with_whisper function takes very long time for mutiple audio files. Have you ever tested the speed of it?

[pzelasko]

I never optimized it for speed. There is a lot of discussion about it in the Whisper repo, maybe you can find something from there useful. I'd be happy to accept contributions with perf improvements :)

[desh2608]

At some point we may consider using one of the Python wrappers around whisper.cpp instead of the original Whisper to speed things up. At the moment, it seems people have come up with several ways to wrap it in Python, such as with cython, ctypes, and pybind11. They seem promising, but most of them seem to have small issues such as incompatibility with Windows etc.

[entn-at]

At some point we may consider using one of the Python wrappers around whisper.cpp instead of the original Whisper to speed things up. At the moment, it seems people have come up with several ways to wrap it in Python, such as with cython, ctypes, and pybind11. They seem promising, but most of them seem to have small issues such as incompatibility with Windows etc.

I've actually started working on a second workflow that uses faster-whisper powered by CTranslate2's implementation (see #1017). It's a lot faster and uses far less memory.

[desh2608]

Cool! Would you be interested in making a PR? I think it would be good to fall back to regular Whisper if people don't have the compute capability required for CTranslate2.