Unable to decode some frames with ESP32-p4 (AUD-6336)

[digitalLumberjack]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate
• Read the documentation to confirm the issue is not addressed there and your configuration is set correctly
• Tested with the latest version to ensure the issue hasn't been fixed

How often does this bug occurs?

always

Expected behavior

Hello. I'm trying to decode H264 video with ESP32-P4.

The H264 decoder should decode.

Actual behavior (suspected bug)

I encode a video this way :

ffmpeg -y -i gundam.original.mp4 -c:v libx264 -profile:v baseline -preset veryslow -tune fastdecode -vf "format=yuv420p, scale=320:240" -x264opts slices=1 -an gundam.h264

Having this ouput:

[libx264 @ 0x5d78f9973a40] profile Constrained Baseline, level 2.2, 4:2:0, 8-bit
Output #0, h264, to 'gundam.h264':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    encoder         : Lavf61.1.100
  Stream #0:0(und): Video: h264, yuv420p(tv, bt709, progressive), 320x240 [SAR 4:3 DAR 16:9], q=2-31, 23.98 fps, 23.98 tbn (default)
      Metadata:
        creation_time   : 2020-02-05T18:42:37.000000Z
        handler_name    : ISO Media file produced by Google Inc. Created on: 02/05/2020.
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.3.100 libx264
      Side data:
        cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
[out#0/h264 @ 0x5d78f9975140] video:2724KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.000000%
frame= 2158 fps=326 q=-1.0 Lsize=    2724KiB time=00:01:30.00 bitrate= 247.9kbits/s speed=13.6x 

We can see the video is Stream #0:0(und): Video: h264, yuv420p(tv, bt709, progressive), 320x240 [SAR 4:3 DAR 16:9], q=2-31, 23.98 fps, 23.98 tbn (default) it seems to be what the H264 decoder is excepting.

When I decode the H264 stream with ESP32P4 on a project that quite simple, based on the testing app :

esp_h264_err_t H264Engine::playVideo(uint8_t * file, uint16_t size) {

    esp_h264_dec_in_frame_t in_frame;
    esp_h264_dec_out_frame_t out_frame;
    esp_h264_err_t ret = ESP_H264_ERR_FAIL;
    esp_h264_dec_handle_t dec = nullptr;
    esp_h264_dec_cfg_sw_t cfg;
    cfg.pic_type = ESP_H264_RAW_FMT_I420;
    int frames_decoded = 0;

    ret = esp_h264_dec_sw_new(&cfg, &dec);
    if (ret != ESP_H264_ERR_OK) {
        printf("new failed. line %d \n", __LINE__);
        goto _single_exit_;
    }

    ret = esp_h264_dec_open(dec);
    ESP_LOGI("H264", "esp_h264_dec_open");

    if (ret != ESP_H264_ERR_OK) {
        printf("open failed .line %d \n", __LINE__);
        goto _single_exit_;
    }
    in_frame.raw_data.buffer = file;
    in_frame.raw_data.len = size;

    esp_h264_dec_param_sw_handle_t dec_param;
    esp_h264_resolution_t res;

    while (1) {
        if (in_frame.raw_data.len <= 0) {
            break;
        }
        ESP_LOGI("H264", "decoding");
        ret = esp_h264_dec_process(dec, &in_frame, &out_frame);

        esp_h264_dec_sw_get_param_hd(dec, &dec_param);
        esp_h264_dec_get_resolution(dec_param, &res);
        ESP_LOGI("H264", "res = %dx%d", res.width, res.height);

        in_frame.raw_data.buffer += in_frame.consume;
        in_frame.raw_data.len -= in_frame.consume;
        ESP_LOGI("H264", "consumed %d, now at %d",  in_frame.consume, in_frame.raw_data.buffer);
        ESP_LOGI("H264", "input frame dts: %d, pts: %d", in_frame.dts, in_frame.pts);
        ESP_LOGI("H264", "decoded frames %d", frames_decoded++);

        if (ret == ESP_H264_ERR_OK) {
            ESP_LOGI("H264", "frame decoded size: %d, dts: %d, pts: %d", out_frame.out_size, out_frame.dts, out_frame.pts);
        } else {
            printf("process failed. ret %d line %d \n", ret, __LINE__);
            goto _single_exit_;
        }
        vTaskDelay(1);
    }
    _single_exit_:
        ESP_LOGI("H264", "exiting");

        esp_h264_dec_close(dec);
        esp_h264_dec_del(dec);
    return ret;

  }

Error logs or terminal output

I can decode up to 40 frames but I then I have:

I (2596) H264: decoded 38
I (2598) H264: frame decoded size: 115200, dts: 1341241836, pts: 1073952316
I (2607) H264: decoding
I (2619) H264: res = 320x240
I (2619) H264: consumed 551, now at 1074014023
I (2619) H264: input frame dts: 1341241836, pts: 1073952316
I (2620) H264: decoded 39
I (2623) H264: frame decoded size: 115200, dts: 1341241836, pts: 1073952316
I (2631) H264: decoding

E (2640) H264_DEC: Decode macroblock layer error(210).
E (2640) H264_DEC: Decode slice data error.
E (2643) H264_DEC.SW: Error in decoding

I (2646) H264: res = 320x240
I (2649) H264: consumed 377, now at 1074014400
I (2653) H264: input frame dts: 1341241836, pts: 1073952316
I (2659) H264: decoded 40
process failed. ret -1 line 63 
I (2664) H264: exiting

Steps to reproduce the behavior

I tried with many different videos, many encoder options, but it seems I cannot achieve a entire decoding.

Is there any standard ffmpeg configuration you can provide if it's encoding configuration causing the issue ?
Any other ideas ?

Project release version

component 1.1.1 on idf master 1c468f68259065ef51afd114605d9122f13d9d72

System architecture

Intel/AMD 64-bit (modern PC, older Mac)

Operating system

Linux

Operating system version

Linux ubuntu 24.10

Shell

ZSH

Additional context

No response

[PumpkinLin314]

Thank you for the feedback.
If possible, please provide this video file to help us reproduce the issue.
Also, please try using version v1.0.4 to see if the issue can be reproduced.

[digitalLumberjack]

Hello, thank you for your reply.
I've got some good news, and maybe it can be useful for the documentation :)

The error I raised above was due to the signature of my method which took a uint16 instead of a uint32 for the size of the video.

Once this problem was corrected, I had a hard time understanding how to generate videos with ffmpeg that were compatible with the component and esp32-p4.

Here's the ffmpeg configuration I use, which allows me to have compatible videos:

ffmpeg -i '201411_blender_big_buck_bunny_60fps_720p_h264-high.mp4' -c:v libx264 -profile:v baseline -preset veryslow -tune fastdecode -vf "format=yuv420p, fps=fps=60000/1001" -colorspace:v 'bt709' -color_primaries:v 'bt709' -color_trc:v 'bt709' -color_range:v 'tv' -x264opts slices=1 -an output.h264

I struggled on getting a FPS option that create compatible, and the XX000/1001 always works, for 25, 30, 60 fps.

It's running good :

1024x600px  -> 16.9fps
640x480px    -> 35.7fps
320x240px    -> 151.6fps

If you don't mind i'll have other questions :

1. Do you know if it is possible to get dts and pts on h264 raw file ? I though this was a feature managed by the container

ESP_LOGI("H264", "input frame dts: %d, pts: %d", in_frame.dts, in_frame.pts);
ESP_LOGI("H264", "frame decoded size: %d, dts: %d, pts: %d", out_frame.out_size, out_frame.dts, out_frame.pts);

Returns always the same dts and pts for all frames:
I (32825) H264: input frame dts: 1341264796, pts: 1208170572
I (32825) H264: frame decoded size: 115200, dts: 1341264796, pts: 1208170572

2. Do you have a yuv420p -> rgb565 and yuv420p > rgb888 efficient converter ? It wanted to use https://github.com/espressif/esp32-camera/blob/master/conversions/yuv.c but it's for yuv422

I'm using https://github.com/yangfuyuan/yuv2rgb/blob/master/v0.03/yuv420rgb565c.c implementation that is great, but maybe we could use esp32-p4 specific instructions for that ?

3. Do you have advices about performance tweaks ? I use the defconfig of the test suite of this project.

I'll be happy to make a PR on the documentation once I understand every details about how it can be the most effective :)

[houhaiyan]

Q1：
You're right. Currently, the decoding support is baseline. Its PTS and DTS depend on the container.
Q2:
please use PPA or esp-image-effects
Q3:

1. The preferred encoding and decoding chip esp32p4. You have already made your choice.
2. Decoding select dual-task decoding (menuconfig: CONFIG_ESP_H264_DUAL_TASK)
3. For color conversion, PPA(hardware color conversion, not occupying CPU) is preferred. If PPA cannot be used, please select esp-image-effects(the SIMD-optimized version will be released around July, with good performance for CPU).

[digitalLumberjack]

Hello @houhaiyan thank you for the answer.
PPA seems not having color conversion features, but i'll try esp-image-effects and compare its performances to https://github.com/yangfuyuan/yuv2rgb/blob/master/v0.03/yuv420rgb565c.c
I'll check again in July too !
Thank you for the tip !

Have a good day :)

[houhaiyan]

@digitalLumberjack you are welcome. Thank you for your feedback
please refer gmf_video. There are use ppa and esp-image-effects to do color convert