[Bug]: HEVC decoding fails on DG1 when using upstream kernel instead of Intel DKMS

[eero-t]

Which component impacted?

Decode

Is it regression? Good in old configuration?

No response

What happened?

Use-cases

• ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i GTAV_1920x1080_60_yuv420p.h265 -c:v h264_vaapi -f null -
• sample_multi_transcode -i::h265 GTAV_1920x1080_60_yuv420p.h265 -o::h264 /dev/null

Expected outcome

Both of above do transcoding at hundreds of FPS, like is the case with TGL iGPU, with exactly the same setup. Or if I change the input to H.264 one.

Actual outcome

• FFmpeg transcoding happens at 2 FPS
• OneVPL transcoding fails to:

$ sample_multi_transcode -i::h265 /media/GTAV_1920x1080_60_yuv420p.h265 -o::h264 /dev/null
Multi Transcoding Sample Version 8.4.27.0

CONFIGURE LOADER: required implementation: hw 
CONFIGURE LOADER: required implementation mfxAccelerationMode: MFX_ACCEL_MODE_VIA_VAAPI 
libva info: VA-API version 1.14.0
libva info: User environment variable requested driver 'iHD'
libva info: Trying to open /usr/local/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
libva info: VA-API version 1.14.0
libva info: User environment variable requested driver 'iHD'
libva info: Trying to open /usr/local/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
libva info: VA-API version 1.14.0
libva info: User environment variable requested driver 'iHD'
libva info: Trying to open /usr/local/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
libva info: VA-API version 1.14.0
libva info: User environment variable requested driver 'iHD'
libva info: Trying to open /usr/local/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
libva info: VA-API version 1.14.0
libva info: User environment variable requested driver 'iHD'
libva info: Trying to open /usr/local/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
Session 0:
Loaded Library configuration: 
    Version: 2.7 
    ImplName: mfx-gen 
    Adapter number : 0 
    Adapter type: integrated
    DRMRenderNodeNum: 128 
Used implementation number: 0 
Loaded modules:
   0: /usr/local/lib/libmfxhw64.so.1.35 
   1: /usr/local/lib/libmfx-gen.so.1.2.7 

Pipeline surfaces number (DecPool): 10
Input  video: HEVC
Output video: AVC 

Session 0 was NOT joined with other sessions

Transcoding started

[ERROR], sts=MFX_ERR_ABORTED(-12), PutBS, Encode: SyncOperation failed at /home/nobody/source/oneVPL/tools/legacy/sample_multi_transcode/src/pipeline_transcode.cpp:2112

[ERROR], sts=MFX_ERR_ABORTED(-12), Transcode, PutBS failed at /home/nobody/source/oneVPL/tools/legacy/sample_multi_transcode/src/pipeline_transcode.cpp:2068

[ERROR], sts=MFX_ERR_ABORTED(-12), Run, CTranscodingPipeline::Run::Transcode() [0x55ba117e6c90] failed at /home/nobody/source/oneVPL/tools/legacy/sample_multi_transcode/src/pipeline_transcode.cpp:4677


 session 0 [0x55ba117e6c90] failed with status MFX_ERR_ABORTED shutting down the application...

session [0x55ba117e6c90] m_bForceStop is set

Transcoding finished

Common transcoding time is 2.88576 sec
-------------------------------------------------------------------------------
*** session 0 [0x55ba117e6c90] FAILED (MFX_ERR_ABORTED) 2.88555 sec, 4 frames, 1.386 fps
-i::h265 /media/GTAV_1920x1080_60_yuv420p.h265 -o::h264 /dev/null 

-------------------------------------------------------------------------------

The test FAILED

[ERROR], sts=MFX_ERR_ABORTED(-12), main, transcode.ProcessResult failed at /home/nobody/source/oneVPL/tools/legacy/sample_multi_transcode/src/sample_multi_transcode.cpp:1561

I do not know whether this is a regression. There have been too many issues to say for sure whether it's ever worked on 0x4905 device.

What's the usage scenario when you are seeing the problem?

Transcode for media delivery

What impacted?

No response

Debug Information

Setup

• GPU: DG1 (0x4905)
• Ubuntu 20.04.4 distro
• drm-tip 5.18 kernel
• media stack components build from latest release tags (as of today):
  • libva: 2.14.0
  • GMMlib: intel-gmmlib-22.1.3
  • Media: intel-media-22.4.2
  • MediaSDK: intel-mediasdk-22.4.2
  • oneVPL: v2022.1.3
  • VPL-GPU: intel-onevpl-22.4.2
  • FFmpeg: n5.0.1

VA-info

libva info: VA-API version 1.14.0
libva info: User environment variable requested driver 'iHD'
libva info: Trying to open /usr/local/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.14 (libva 2.12.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 22.4.2 (d7a1feb)
vainfo: Supported profile and entrypoints
      VAProfileNone                   :	VAEntrypointVideoProc
      VAProfileNone                   :	VAEntrypointStats
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Simple            :	VAEntrypointEncSlice
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointFEI
      VAProfileH264Main               :	VAEntrypointEncSliceLP
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointFEI
      VAProfileH264High               :	VAEntrypointEncSliceLP
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileJPEGBaseline           :	VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline:	VAEntrypointFEI
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSliceLP
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointEncSlice
      VAProfileHEVCMain               :	VAEntrypointFEI
      VAProfileHEVCMain               :	VAEntrypointEncSliceLP
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointEncSlice
      VAProfileHEVCMain10             :	VAEntrypointEncSliceLP
      VAProfileVP9Profile0            :	VAEntrypointVLD
      VAProfileVP9Profile0            :	VAEntrypointEncSliceLP
      VAProfileVP9Profile1            :	VAEntrypointVLD
      VAProfileVP9Profile1            :	VAEntrypointEncSliceLP
      VAProfileVP9Profile2            :	VAEntrypointVLD
      VAProfileVP9Profile2            :	VAEntrypointEncSliceLP
      VAProfileVP9Profile3            :	VAEntrypointVLD
      VAProfileVP9Profile3            :	VAEntrypointEncSliceLP
      VAProfileHEVCMain12             :	VAEntrypointVLD
      VAProfileHEVCMain12             :	VAEntrypointEncSlice
      VAProfileHEVCMain422_10         :	VAEntrypointVLD
      VAProfileHEVCMain422_10         :	VAEntrypointEncSlice
      VAProfileHEVCMain422_12         :	VAEntrypointVLD
      VAProfileHEVCMain422_12         :	VAEntrypointEncSlice
      VAProfileHEVCMain444            :	VAEntrypointVLD
      VAProfileHEVCMain444            :	VAEntrypointEncSliceLP
      VAProfileHEVCMain444_10         :	VAEntrypointVLD
      VAProfileHEVCMain444_10         :	VAEntrypointEncSliceLP
      VAProfileHEVCMain444_12         :	VAEntrypointVLD
      VAProfileHEVCSccMain            :	VAEntrypointVLD
      VAProfileHEVCSccMain            :	VAEntrypointEncSliceLP
      VAProfileHEVCSccMain10          :	VAEntrypointVLD
      VAProfileHEVCSccMain10          :	VAEntrypointEncSliceLP
      VAProfileHEVCSccMain444         :	VAEntrypointVLD
      VAProfileHEVCSccMain444         :	VAEntrypointEncSliceLP
      VAProfileAV1Profile0            :	VAEntrypointVLD
      VAProfileHEVCSccMain444_10      :	VAEntrypointVLD
      VAProfileHEVCSccMain444_10      :	VAEntrypointEncSliceLP

Notes

There are no GPU hangs. Kernel driver output / settings:

# dmesg |grep i915
[    0.000000] Command line: BOOT_IMAGE=/boot/drm_intel root=/dev/nvme0n1p2 rootwait fsck.repair=yes i915.enable_guc=3 i915.force_probe=4905 ro
[    0.026206] Kernel command line: BOOT_IMAGE=/boot/drm_intel root=/dev/nvme0n1p2 rootwait fsck.repair=yes i915.enable_guc=3 i915.force_probe=4905 ro
[    2.582581] i915 0000:03:00.0: [drm] VT-d active for gfx access
[    2.582586] fb0: switching to i915 from EFI VGA
[    2.582684] i915 0000:03:00.0: vgaarb: deactivate vga console
[    2.582707] i915 0000:03:00.0: [drm] Local memory IO size: 0x00000000fb800000
[    2.582708] i915 0000:03:00.0: [drm] Local memory available: 0x00000000fb800000
[    2.597310] i915 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    2.600584] i915 0000:03:00.0: [drm] Finished loading DMC firmware i915/dg1_dmc_ver2_02.bin (v2.2)
[    2.667756] i915 0000:03:00.0: [drm] GuC firmware i915/dg1_guc_70.1.1.bin version 70.1
[    2.667759] i915 0000:03:00.0: [drm] HuC firmware i915/dg1_huc_7.9.3.bin version 7.9
[    2.673370] i915 0000:03:00.0: [drm] HuC authenticated
[    2.673789] i915 0000:03:00.0: [drm] GuC submission enabled
[    2.673790] i915 0000:03:00.0: [drm] GuC SLPC enabled
[    2.674046] i915 0000:03:00.0: [drm] GuC RC: enabled

Do you want to contribute a patch to fix the issue?

No.

[eero-t]

I've also tested Sysman functionality and simple OpenCL programs. Those work fine, so in general drm-tip kernel seems to work fine.

Btw. media-driver README still states following:

Media-driver requires special i915 kernel mode driver (KMD) version to support the following new platforms since upstream version of i915 KMD does not fully support them (pending patches upstream):

DG1/SG1
Alchemist(DG2)/ATSM

By default, media-driver builds against upstream i915 KMD and will miss support for the platforms listed above. To enable new platforms which require special i915 KMD and specify ENABLE_PRODUCTION_KMD=ON (default: OFF) build configuration option.

Although AFAIK that has not been true for DG1 for over a half a year, since this media-driver commit: db5a870

And DG2 / ATS-M support being already in public kernel (for some of their variants, and requiring force-probing for now).

[XinfengZhang]

from 22'Q1 release, media_driver does not support DG1 with ENABLE_PRODUCTION_KMD=ON anymore
now, this option just support DG2 with suitable kernel support (https://github.com/intel-gpu/intel-gpu-i915-backports/tree/ubuntu/main) will update the document, if you still want DG1 against https://github.com/intel-gpu/kernel, please use 21'Q4 release.

[eero-t]

The note about outdated DG1 info in README was just FYI.

I am using ENABLE_PRODUCTION_KMD=OFF with public kernel, and that fails for me on DG1 with HEVC.

(There may have been 3D + AVC transode running at the same time in the backend while I was running this test-case, but that should not have broken HEVC as dmesg does not show any errors.)

[Xiaogangli-intel]

Hi @eero-t, I noticed i915.enable_guc=3 in your kernel parameters, could you try i915.enable_guc=2? Seems GuC submission doesn't work on DG1.

[Xiaogangli-intel]

Hi @eero-t , media still have some issues on drm-tip KMD for DG1.
Could you please try this KMD at https://github.com/intel-gpu/intel-gpu-i915-backports, and need to build media driver with ENABLE_PRODUCTION_KMD=ON, also i915.enable_guc=2 in kernel boot parameters.

[dvrogozh]

At the moment there are 2 possible ways to setup DG1:

1. Use vanilla kernel (or drm-tip). DG1 support is not still finalized in here, user should use i915.force_probe=* (or specific device id) to enable. You don't need any special options of media driver build for that.
2. Use custom kernel (or rather kernel module) which @Xiaogangli-intel suggests above, https://github.com/intel-gpu/intel-gpu-i915-backports. Use ENABLE_PRODUCTION_KMD=ON to build media-driver.

That's up to the user to decide which kernel to use. However, in both cases, user is NOT supposed to adjust i915.enable_guc option in any way. This is a very risky option and user should clear understand why he is trying to change it.

@eero-t : I strongly suggest to drop i915.enable_guc from cmdline and try again. I vaguely recall you had some issue because of setting this option before. Hope this will help. If not, then you've found LGTM issue for vanilla kernel which media team will need to look at.

[eero-t]

Good catch. I'll test upstream kernel without the GuC option tomorrow, and report back.

(I've intended to clean that out, but had forgotten to do it for all kernel configs on all machines.)

[eero-t]

@dvrogozh GuC scheduling is enabled by public (yesterday) "drm-tip" kernel, even when it's not forced:

# dmesg | grep i915
[    0.000000] Command line: BOOT_IMAGE=/boot/drm_intel root=/dev/nvme0n1p2 rootwait fsck.repair=yes i915.force_probe=4905 ro
[    0.026212] Kernel command line: BOOT_IMAGE=/boot/drm_intel root=/dev/nvme0n1p2 rootwait fsck.repair=yes i915.force_probe=4905 ro
[    2.081413] i915 0000:03:00.0: [drm] VT-d active for gfx access
[    2.081417] fb0: switching to i915 from EFI VGA
[    2.081679] i915 0000:03:00.0: vgaarb: deactivate vga console
[    2.081714] i915 0000:03:00.0: [drm] Local memory IO size: 0x00000000fb800000
[    2.081715] i915 0000:03:00.0: [drm] Local memory available: 0x00000000fb800000
[    2.094978] i915 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    2.099214] i915 0000:03:00.0: [drm] Finished loading DMC firmware i915/dg1_dmc_ver2_02.bin (v2.2)
[    2.165804] i915 0000:03:00.0: [drm] GuC firmware i915/dg1_guc_70.1.1.bin version 70.1
[    2.165807] i915 0000:03:00.0: [drm] HuC firmware i915/dg1_huc_7.9.3.bin version 7.9
[    2.171291] i915 0000:03:00.0: [drm] HuC authenticated
[    2.171568] i915 0000:03:00.0: [drm] GuC submission enabled
[    2.171569] i915 0000:03:00.0: [drm] GuC SLPC enabled
[    2.171828] i915 0000:03:00.0: [drm] GuC RC: enabled
[    2.207306] [drm] Initialized i915 1.6.0 20201103 for 0000:03:00.0 on minor 0

And the same issue persists.

Note: I forgot to mention earlier, but in case it matters, all of these nodes have 2 0x4905 DG1 GPUs. Limiting media-driver devfs visibility just to first one (with Docker) did not change anything though.

[eero-t]

@Xiaogangli-intel even with GuC scheduling explicitly disabled for drm-tip:

# dmesg | grep i915
[    0.000000] Command line: BOOT_IMAGE=/boot/drm_intel root=/dev/nvme0n1p2 rootwait fsck.repair=yes i915.enable_guc=2 i915.force_probe=4905 ro
[    0.026191] Kernel command line: BOOT_IMAGE=/boot/drm_intel root=/dev/nvme0n1p2 rootwait fsck.repair=yes i915.enable_guc=2 i915.force_probe=4905 ro
[    2.084816] i915 0000:03:00.0: [drm] VT-d active for gfx access
[    2.084820] fb0: switching to i915 from EFI VGA
[    2.084891] i915 0000:03:00.0: vgaarb: deactivate vga console
[    2.084913] i915 0000:03:00.0: [drm] Local memory IO size: 0x00000000fb800000
[    2.084914] i915 0000:03:00.0: [drm] Local memory available: 0x00000000fb800000
[    2.096936] i915 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    2.100697] i915 0000:03:00.0: [drm] Finished loading DMC firmware i915/dg1_dmc_ver2_02.bin (v2.2)
[    2.170110] i915 0000:03:00.0: [drm] GuC firmware i915/dg1_guc_70.1.1.bin version 70.1
[    2.170113] i915 0000:03:00.0: [drm] HuC firmware i915/dg1_huc_7.9.3.bin version 7.9
[    2.185450] i915 0000:03:00.0: [drm] HuC authenticated
[    2.185451] i915 0000:03:00.0: [drm] GuC submission disabled
[    2.185452] i915 0000:03:00.0: [drm] GuC SLPC disabled
[    2.220340] [drm] Initialized i915 1.6.0 20201103 for 0000:03:00.0 on minor 0

Media driver fails:

sample_multi_transcode -i::h265 /media/GTAV_1920x1080_60_yuv420p.h265 -o::h264 /dev/null
Multi Transcoding Sample Version 8.4.27.0

CONFIGURE LOADER: required implementation: hw 
CONFIGURE LOADER: required implementation mfxAccelerationMode: MFX_ACCEL_MODE_VIA_VAAPI 
libva info: VA-API version 1.14.0
...
libva info: User environment variable requested driver 'iHD'
libva info: Trying to open /usr/local/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
Session 0:
Loaded Library configuration: 
    Version: 2.7 
    ImplName: mfx-gen 
    Adapter number : 0 
    Adapter type: integrated
    DRMRenderNodeNum: 128 
Used implementation number: 0 
Loaded modules:
   0: /usr/local/lib/libmfxhw64.so.1.35 
   1: /usr/local/lib/libmfx-gen.so.1.2.7 

Pipeline surfaces number (DecPool): 10
Input  video: HEVC
Output video: AVC 

Session 0 was NOT joined with other sessions

Transcoding started

[ERROR], sts=MFX_ERR_ABORTED(-12), PutBS, Encode: SyncOperation failed at /home/nobody/source/oneVPL/tools/legacy/sample_multi_transcode/src/pipeline_transcode.cpp:2112

[ERROR], sts=MFX_ERR_ABORTED(-12), Transcode, PutBS failed at /home/nobody/source/oneVPL/tools/legacy/sample_multi_transcode/src/pipeline_transcode.cpp:2068

[ERROR], sts=MFX_ERR_ABORTED(-12), Run, CTranscodingPipeline::Run::Transcode() [0x55b1e65abc90] failed at /home/nobody/source/oneVPL/tools/legacy/sample_multi_transcode/src/pipeline_transcode.cpp:4677

 session 0 [0x55b1e65abc90] failed with status MFX_ERR_ABORTED shutting down the application...

session [0x55b1e65abc90] m_bForceStop is set

Transcoding finished

[eero-t]

When GuC scheduling is explicitly disabled, there's also a GPU hang:

[   65.151632] i915 0000:03:00.0: [drm] Resetting vcs1 for preemption time out
[   65.151688] i915 0000:03:00.0: [drm] sample_multi_tr[5533] context reset due to GPU hang
[   65.160360] i915 0000:03:00.0: [drm] GPU HANG: ecode 12:4:28fffffd, in sample_multi_tr [5533]

See: gpu-hang.txt

media still have some issues on drm-tip KMD for DG1.

Could you give pointer to more info?

Could you please try this KMD at https://github.com/intel-gpu/intel-gpu-i915-backports, and need to build media driver with ENABLE_PRODUCTION_KMD=ON, also i915.enable_guc=2 in kernel boot parameters.

Sorry, but I'm not interested about public media-driver on backport kernel, only with what's going to upstream.

[eero-t]

Btw. 1-2 months ago when I was testing internal KMD + UMD versions, I was seeing some instances failing with OneVPL HEVC transcode, when trying to do many parallel transcodes on DG2. I did not debug it further (container instances were changing too fast), but I'm now wondering whether it's related to HEVC issues here with public KMD+UMD versions on DG1. Are there known HEVC issues for DG2 too?

[Xiaogangli-intel]

Hi @eero-t, I noticed the hang issue of HEVC decode. If you really mind using backport kernel, maybe we have to sync with KMD to check the progress of DG1 patches upstreaming.