对qwen3omni进行语音榜单的评测失败

[wuyongtai7]

请教下在lmms-eval框架中，使用部署的qwen3_omni模型的api的方式进行librispeech语音榜单的评测时，怎么取不到预测的结果呢
单条数据评测结果：
{"doc_id": 14, "target": "BY HARRY QUILTER M A", "filtered_resps": ["Please recognize the speech and only output the recognized content:"], "doc_hash": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", "wer": {"gt": "BY HARRY QUILTER M A", "pred": "Please recognize the speech and only output the recognized content:", "source": "librispeech_dev_clean", "task": "asr_en"}, "input": "Please recognize the speech and only output the recognized content:"}

[Luodian]

@YichenG170

[wuyongtai7]

很奇怪，这个框架真的支持hugging face上音频数据的加载吗？doc_to_messages()中音频的格式是dict，和hugging face上的音频数据格式为<datasets.features._torchcodec.AudioDecoder object at 0x7f826af61310>明显不一致，这里是怎么转换的？

[wuyongtai7]

麻烦两位大佬帮忙解答下

[YichenG170]

def librispeech_doc_to_audio(doc):
audio_data = doc["audio"]
return [{"array": audio_data["array"], "sampling_rate": audio_data["sampling_rate"]}]

像这样改一下数据格式。我这里刚刚实测过了没问题，晚点儿会PR上去。

[wuyongtai7]

这个看起来像是加载huggingface上的音频数据时格式没有对齐，huggingface上的音频数据是Audio decoder对象，但是评测框架中的音频输入确实数组的格式，我看到有些同学正在提交一些代码来解决这个问题，但是我在复用这些方案时遇到问题，遇到报错：Error converting AudioDecoder object: The frame has 0 channels, expected 1. If you are hitting this, it may be because you are using a buggy FFmpeg version. FFmpeg4 is known to fail here in some valid scenarios. Try to upgrade FFmpeg?

if str(type(audio_file).__name__) == 'AudioDecoder':
            try:
                if hasattr(audio_file, 'get_all_samples'):
                    decoded_audio = audio_file.get_all_samples()
                    
                    if hasattr(decoded_audio, 'samples'):
                        audio_array = decoded_audio.samples
                    elif hasattr(decoded_audio, 'array'):
                        audio_array = decoded_audio.array
                    elif hasattr(decoded_audio, 'data'):
                        audio_array = decoded_audio.data
                    else:
                        audio_array = decoded_audio
                    
                    if hasattr(audio_array, 'cpu') and hasattr(audio_array, 'numpy'):
                        audio_array = audio_array.cpu().numpy()
                    elif hasattr(audio_array, 'detach'):
                        audio_array = audio_array.detach().cpu().numpy()
                    elif str(type(audio_array).__name__) == 'Tensor':
                        try:
                            audio_array = audio_array.cpu().numpy()
                        except:
                            try:
                                audio_array = audio_array.detach().cpu().numpy()
                            except:
                                audio_array = np.array(audio_array)
                    
                    sampling_rate = 16000  # default
                    if hasattr(decoded_audio, 'sample_rate'):
                        sampling_rate = decoded_audio.sample_rate
                    elif hasattr(decoded_audio, 'sampling_rate'):
                        sampling_rate = decoded_audio.sampling_rate
                    elif hasattr(audio_file, 'metadata') and audio_file.metadata:
                        if hasattr(audio_file.metadata, 'sample_rate'):
                            sampling_rate = audio_file.metadata.sample_rate
                        elif isinstance(audio_file.metadata, dict) and 'sample_rate' in audio_file.metadata:
                            sampling_rate = audio_file.metadata['sample_rate']
                    elif hasattr(audio_file, '_desired_sample_rate') and audio_file._desired_sample_rate:
                        sampling_rate = audio_file._desired_sample_rate
                    
                    audio_dict = {
                        'array': audio_array,
                        'sampling_rate': sampling_rate
                    }
                    return [audio_dict]

[YichenG170]

这个看起来像是加载huggingface上的音频数据时格式没有对齐，huggingface上的音频数据是Audio decoder对象，但是评测框架中的音频输入确实数组的格式，我看到有些同学正在提交一些代码来解决这个问题，但是我在复用这些方案时遇到问题，遇到报错：Error converting AudioDecoder object: The frame has 0 channels, expected 1. If you are hitting this, it may be because you are using a buggy FFmpeg version. FFmpeg4 is known to fail here in some valid scenarios. Try to upgrade FFmpeg?

if str(type(audio_file).__name__) == 'AudioDecoder':
            try:
                if hasattr(audio_file, 'get_all_samples'):
                    decoded_audio = audio_file.get_all_samples()
                    
                    if hasattr(decoded_audio, 'samples'):
                        audio_array = decoded_audio.samples
                    elif hasattr(decoded_audio, 'array'):
                        audio_array = decoded_audio.array
                    elif hasattr(decoded_audio, 'data'):
                        audio_array = decoded_audio.data
                    else:
                        audio_array = decoded_audio
                    
                    if hasattr(audio_array, 'cpu') and hasattr(audio_array, 'numpy'):
                        audio_array = audio_array.cpu().numpy()
                    elif hasattr(audio_array, 'detach'):
                        audio_array = audio_array.detach().cpu().numpy()
                    elif str(type(audio_array).__name__) == 'Tensor':
                        try:
                            audio_array = audio_array.cpu().numpy()
                        except:
                            try:
                                audio_array = audio_array.detach().cpu().numpy()
                            except:
                                audio_array = np.array(audio_array)
                    
                    sampling_rate = 16000  # default
                    if hasattr(decoded_audio, 'sample_rate'):
                        sampling_rate = decoded_audio.sample_rate
                    elif hasattr(decoded_audio, 'sampling_rate'):
                        sampling_rate = decoded_audio.sampling_rate
                    elif hasattr(audio_file, 'metadata') and audio_file.metadata:
                        if hasattr(audio_file.metadata, 'sample_rate'):
                            sampling_rate = audio_file.metadata.sample_rate
                        elif isinstance(audio_file.metadata, dict) and 'sample_rate' in audio_file.metadata:
                            sampling_rate = audio_file.metadata['sample_rate']
                    elif hasattr(audio_file, '_desired_sample_rate') and audio_file._desired_sample_rate:
                        sampling_rate = audio_file._desired_sample_rate
                    
                    audio_dict = {
                        'array': audio_array,
                        'sampling_rate': sampling_rate
                    }
                    return [audio_dict]

请问这个也是用qwen3omini测librispeech的报错吗？

[wuyongtai7]

这个看起来像是加载huggingface上的音频数据时格式没有对齐，huggingface上的音频数据是Audio decoder对象，但是评测框架中的音频输入确实数组的格式，我看到有些同学正在提交一些代码来解决这个问题，但是我在复用这些方案时遇到问题，遇到报错：Error converting AudioDecoder object: The frame has 0 channels, expected 1. If you are hitting this, it may be because you are using a buggy FFmpeg version. FFmpeg4 is known to fail here in some valid scenarios. Try to upgrade FFmpeg?

if str(type(audio_file).__name__) == 'AudioDecoder':
            try:
                if hasattr(audio_file, 'get_all_samples'):
                    decoded_audio = audio_file.get_all_samples()
                    
                    if hasattr(decoded_audio, 'samples'):
                        audio_array = decoded_audio.samples
                    elif hasattr(decoded_audio, 'array'):
                        audio_array = decoded_audio.array
                    elif hasattr(decoded_audio, 'data'):
                        audio_array = decoded_audio.data
                    else:
                        audio_array = decoded_audio
                    
                    if hasattr(audio_array, 'cpu') and hasattr(audio_array, 'numpy'):
                        audio_array = audio_array.cpu().numpy()
                    elif hasattr(audio_array, 'detach'):
                        audio_array = audio_array.detach().cpu().numpy()
                    elif str(type(audio_array).__name__) == 'Tensor':
                        try:
                            audio_array = audio_array.cpu().numpy()
                        except:
                            try:
                                audio_array = audio_array.detach().cpu().numpy()
                            except:
                                audio_array = np.array(audio_array)
                    
                    sampling_rate = 16000  # default
                    if hasattr(decoded_audio, 'sample_rate'):
                        sampling_rate = decoded_audio.sample_rate
                    elif hasattr(decoded_audio, 'sampling_rate'):
                        sampling_rate = decoded_audio.sampling_rate
                    elif hasattr(audio_file, 'metadata') and audio_file.metadata:
                        if hasattr(audio_file.metadata, 'sample_rate'):
                            sampling_rate = audio_file.metadata.sample_rate
                        elif isinstance(audio_file.metadata, dict) and 'sample_rate' in audio_file.metadata:
                            sampling_rate = audio_file.metadata['sample_rate']
                    elif hasattr(audio_file, '_desired_sample_rate') and audio_file._desired_sample_rate:
                        sampling_rate = audio_file._desired_sample_rate
                    
                    audio_dict = {
                        'array': audio_array,
                        'sampling_rate': sampling_rate
                    }
                    return [audio_dict]

请问这个也是用qwen3omini测librispeech的报错吗？

是的

[wuyongtai7]

def librispeech_doc_to_audio(doc): audio_data = doc["audio"] return [{"array": audio_data["array"], "sampling_rate": audio_data["sampling_rate"]}]

像这样改一下数据格式。我这里刚刚实测过了没问题，晚点儿会PR上去。

我在使用这个方法时，依旧是报错：RuntimeError: The frame has 0 channels, expected 1. If you are hitting this, it may be because you are using a buggy FFmpeg version. FFmpeg4 is known to fail here in some valid scenarios. Try to upgrade FFmpeg?

[YichenG170]

def librispeech_doc_to_audio(doc): audio_data = doc["audio"] return [{"array": audio_data["array"], "sampling_rate": audio_data["sampling_rate"]}]
像这样改一下数据格式。我这里刚刚实测过了没问题，晚点儿会PR上去。

我在使用这个方法时，依旧是报错：RuntimeError: The frame has 0 channels, expected 1. If you are hitting this, it may be because you are using a buggy FFmpeg version. FFmpeg4 is known to fail here in some valid scenarios. Try to upgrade FFmpeg?

我刚刚测评的env里没有装torchaudio和ffmpeg
需要我把环境文件发给你吗

[wuyongtai7]

def librispeech_doc_to_audio(doc): audio_data = doc["audio"] return [{"array": audio_data["array"], "sampling_rate": audio_data["sampling_rate"]}]
像这样改一下数据格式。我这里刚刚实测过了没问题，晚点儿会PR上去。

我在使用这个方法时，依旧是报错：RuntimeError: The frame has 0 channels, expected 1. If you are hitting this, it may be because you are using a buggy FFmpeg version. FFmpeg4 is known to fail here in some valid scenarios. Try to upgrade FFmpeg?

我刚刚测评的env里没有装torchaudio和ffmpeg 需要我把环境文件发给你吗

辛苦发一份环境配置吧，感觉应该是某个包版本的问题

[YichenG170]

accelerate==1.10.1
aiohappyeyeballs==2.6.1
aiohttp==3.12.15
aiosignal==1.4.0
annotated-types==0.7.0
antlr4-python3-runtime==4.7.2
anyio==4.11.0
async-timeout==5.0.1
attrs==25.3.0
audioread==3.0.1
av==15.1.0
black==25.9.0
certifi==2025.8.3
cffi==2.0.0
cfgv==3.4.0
chardet==5.2.0
charset-normalizer==3.4.3
click==8.3.0
colorama==0.4.6
dataproperty==1.1.0
datasets==4.1.1
decorator==5.2.1
decord==0.6.0
dill==0.4.0
distlib==0.4.0
distro==1.9.0
editdistance==0.8.1
einops==0.8.1
et-xmlfile==2.0.0
evaluate==0.4.6
exceptiongroup==1.3.0
filelock==3.19.1
frozenlist==1.7.0
fsspec==2025.9.0
ftfy==6.3.1
gitdb==4.0.12
gitpython==3.1.45
h11==0.14.0
hf-transfer==0.1.9
hf-xet==1.1.10
httpcore==0.16.3
httpx==0.23.3
huggingface-hub==0.35.1
identify==2.6.14
idna==3.10
imageio==2.37.0
imageio-ffmpeg==0.6.0
isort==6.0.1
jinja2==3.1.6
jiter==0.11.0
joblib==1.5.2
jsonlines==4.0.0
latex2sympy2==1.9.1
lazy-loader==0.4
librosa==0.11.0
llvmlite==0.45.0
lmms-eval @ git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git@99f93c7081b6d787b723ffe9408b18bf9e3c2b82
loguru==0.7.3
lxml==6.0.2
markupsafe==3.0.2
mbstrdecoder==1.1.4
moviepy==2.2.1
mpmath==1.3.0
msgpack==1.1.1
multidict==6.6.4
multiprocess==0.70.16
mypy-extensions==1.1.0
networkx==3.4.2
nltk==3.9.1
nodeenv==1.9.1
numba==0.62.0
numexpr==2.13.0
numpy==1.26.4
nvidia-cublas-cu12==12.8.4.1
nvidia-cuda-cupti-cu12==12.8.90
nvidia-cuda-nvrtc-cu12==12.8.93
nvidia-cuda-runtime-cu12==12.8.90
nvidia-cudnn-cu12==9.10.2.21
nvidia-cufft-cu12==11.3.3.83
nvidia-cufile-cu12==1.13.1.3
nvidia-curand-cu12==10.3.9.90
nvidia-cusolver-cu12==11.7.3.90
nvidia-cusparse-cu12==12.5.8.93
nvidia-cusparselt-cu12==0.7.1
nvidia-nccl-cu12==2.27.3
nvidia-nvjitlink-cu12==12.8.93
nvidia-nvtx-cu12==12.8.90
openai==1.109.1
opencv-python-headless==4.11.0.86
openpyxl==3.1.5
packaging==25.0
pandas==2.3.2
pathspec==0.12.1
pathvalidate==3.3.1
peft==0.17.1
pillow==11.3.0
platformdirs==4.4.0
pooch==1.8.2
portalocker==3.2.0
pre-commit==4.3.0
proglog==0.1.12
propcache==0.3.2
protobuf==6.32.1
psutil==7.1.0
pyarrow==21.0.0
pybind11==3.0.1
pycocoevalcap==1.2
pycocotools==2.0.10
pycparser==2.23
pydantic==2.11.9
pydantic-core==2.33.2
pytablewriter==1.2.1
python-dateutil==2.9.0.post0
python-dotenv==1.1.1
pytokens==0.1.10
pytz==2025.2
pyyaml==6.0.3
qwen-omni-utils==0.0.8
regex==2025.9.18
requests==2.32.5
rfc3986==1.5.0
sacrebleu==2.5.1
safetensors==0.6.2
scikit-learn==1.7.2
scipy==1.15.3
sentence-transformers==5.1.1
sentencepiece==0.2.1
sentry-sdk==2.39.0
setuptools==80.9.0
six==1.17.0
smmap==5.0.2
sniffio==1.3.1
soundfile==0.13.1
soxr==1.0.0
sqlitedict==2.1.0
sympy==1.14.0
tabledata==1.3.4
tabulate==0.9.0
tcolorpy==0.1.7
tenacity==8.3.0
threadpoolctl==3.6.0
tiktoken==0.11.0
timm==1.0.20
tokenizers==0.22.1
tomli==2.2.1
torch==2.8.0
torchcodec==0.7.0
torchvision==0.23.0
tqdm==4.67.1
tqdm-multiprocess==0.0.11
transformers==4.56.2
transformers-stream-generator==0.0.5
triton==3.4.0
typepy==1.3.4
typing-extensions==4.15.0
typing-inspection==0.4.1
tzdata==2025.2
urllib3==2.5.0
virtualenv==20.34.0
wandb==0.22.0
wcwidth==0.2.14
xxhash==3.5.0
yarl==1.20.1
yt-dlp==2025.9.23
zhconv==1.4.3
zss==1.2.0
zstandard==0.25.0

[YichenG170]

accelerate==1.10.1 aiohappyeyeballs==2.6.1 aiohttp==3.12.15 aiosignal==1.4.0 annotated-types==0.7.0 antlr4-python3-runtime==4.7.2 anyio==4.11.0 async-timeout==5.0.1 attrs==25.3.0 audioread==3.0.1 av==15.1.0 black==25.9.0 certifi==2025.8.3 cffi==2.0.0 cfgv==3.4.0 chardet==5.2.0 charset-normalizer==3.4.3 click==8.3.0 colorama==0.4.6 dataproperty==1.1.0 datasets==4.1.1 decorator==5.2.1 decord==0.6.0 dill==0.4.0 distlib==0.4.0 distro==1.9.0 editdistance==0.8.1 einops==0.8.1 et-xmlfile==2.0.0 evaluate==0.4.6 exceptiongroup==1.3.0 filelock==3.19.1 frozenlist==1.7.0 fsspec==2025.9.0 ftfy==6.3.1 gitdb==4.0.12 gitpython==3.1.45 h11==0.14.0 hf-transfer==0.1.9 hf-xet==1.1.10 httpcore==0.16.3 httpx==0.23.3 huggingface-hub==0.35.1 identify==2.6.14 idna==3.10 imageio==2.37.0 imageio-ffmpeg==0.6.0 isort==6.0.1 jinja2==3.1.6 jiter==0.11.0 joblib==1.5.2 jsonlines==4.0.0 latex2sympy2==1.9.1 lazy-loader==0.4 librosa==0.11.0 llvmlite==0.45.0 lmms-eval @ git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git@99f93c7081b6d787b723ffe9408b18bf9e3c2b82 loguru==0.7.3 lxml==6.0.2 markupsafe==3.0.2 mbstrdecoder==1.1.4 moviepy==2.2.1 mpmath==1.3.0 msgpack==1.1.1 multidict==6.6.4 multiprocess==0.70.16 mypy-extensions==1.1.0 networkx==3.4.2 nltk==3.9.1 nodeenv==1.9.1 numba==0.62.0 numexpr==2.13.0 numpy==1.26.4 nvidia-cublas-cu12==12.8.4.1 nvidia-cuda-cupti-cu12==12.8.90 nvidia-cuda-nvrtc-cu12==12.8.93 nvidia-cuda-runtime-cu12==12.8.90 nvidia-cudnn-cu12==9.10.2.21 nvidia-cufft-cu12==11.3.3.83 nvidia-cufile-cu12==1.13.1.3 nvidia-curand-cu12==10.3.9.90 nvidia-cusolver-cu12==11.7.3.90 nvidia-cusparse-cu12==12.5.8.93 nvidia-cusparselt-cu12==0.7.1 nvidia-nccl-cu12==2.27.3 nvidia-nvjitlink-cu12==12.8.93 nvidia-nvtx-cu12==12.8.90 openai==1.109.1 opencv-python-headless==4.11.0.86 openpyxl==3.1.5 packaging==25.0 pandas==2.3.2 pathspec==0.12.1 pathvalidate==3.3.1 peft==0.17.1 pillow==11.3.0 platformdirs==4.4.0 pooch==1.8.2 portalocker==3.2.0 pre-commit==4.3.0 proglog==0.1.12 propcache==0.3.2 protobuf==6.32.1 psutil==7.1.0 pyarrow==21.0.0 pybind11==3.0.1 pycocoevalcap==1.2 pycocotools==2.0.10 pycparser==2.23 pydantic==2.11.9 pydantic-core==2.33.2 pytablewriter==1.2.1 python-dateutil==2.9.0.post0 python-dotenv==1.1.1 pytokens==0.1.10 pytz==2025.2 pyyaml==6.0.3 qwen-omni-utils==0.0.8 regex==2025.9.18 requests==2.32.5 rfc3986==1.5.0 sacrebleu==2.5.1 safetensors==0.6.2 scikit-learn==1.7.2 scipy==1.15.3 sentence-transformers==5.1.1 sentencepiece==0.2.1 sentry-sdk==2.39.0 setuptools==80.9.0 six==1.17.0 smmap==5.0.2 sniffio==1.3.1 soundfile==0.13.1 soxr==1.0.0 sqlitedict==2.1.0 sympy==1.14.0 tabledata==1.3.4 tabulate==0.9.0 tcolorpy==0.1.7 tenacity==8.3.0 threadpoolctl==3.6.0 tiktoken==0.11.0 timm==1.0.20 tokenizers==0.22.1 tomli==2.2.1 torch==2.8.0 torchcodec==0.7.0 torchvision==0.23.0 tqdm==4.67.1 tqdm-multiprocess==0.0.11 transformers==4.56.2 transformers-stream-generator==0.0.5 triton==3.4.0 typepy==1.3.4 typing-extensions==4.15.0 typing-inspection==0.4.1 tzdata==2025.2 urllib3==2.5.0 virtualenv==20.34.0 wandb==0.22.0 wcwidth==0.2.14 xxhash==3.5.0 yarl==1.20.1 yt-dlp==2025.9.23 zhconv==1.4.3 zss==1.2.0 zstandard==0.25.0

测试libri刚搭建的一个环境，后面有缺的包可以直接install，试一下可不可以。

[kcz358]

Great question! I am answering in English in case any community member also spot this issue. The main problem is actually that the datasets use torchcodec for audio decoding after 4.0.0 version while we developed the audio evaluation toolkit with dataset<=3.6.0. If you are using version before 3.6.0, the loading result should be a dictionary like this

However, after dataset 4.0.0, the package force to use torchcodec as audioencoder which in some case might be unstable. On my settings dependencies provided greatly by the @YichenG170 sadly didn't work and I encountered issue in this

huggingface/datasets#7707

Downgrading torchcodec to 0.5.0 solve my issue and the result for audio is like this and you will have to get the audio by using an 'array' key

For now, I would recommended to downgrade to 3.6.0 for datasets if you want to test on any older integrated datasets. For example, openasr (Yeah I would recommend you to use this task to test ASR tasks because it contains most of the tasks). For newly integrated dataset, there should possibly a simple check for different version so that we can have a fallback solution for older data. CC @YichenG170 for this for the PR changes

Welcome for any design suggestions on audio tasks!