linux-v4l2: Fix H.264 decoding artifacts at high resolution

[JiangXsong]

Description

This PR addresses severe artifacts (mosaic/smearing) when capturing high-resolution (e.g., 4K) H.264 streams via V4L2 on Linux.

Motivation and Context

On Linux, capturing 4K H.264 via V4L2 was practically unusable due to single-threaded decoding bottlenecks, resulting in corrupted keyframes (mosaic artifacts). Standard tools like ffplay worked fine because they default to multi-threaded software decoding, whereas OBS was restricted to a single thread.

This PR aligns the V4L2 source performance with other platforms (Windows) and standard players (ffplay), making 4K H.264 capture viable on Linux.

How Has This Been Tested?

• Platform: Ubuntu 24.04
• Device: UVC device (bulk mode) supporting H.264
• Resolution: 3840x2160 @ 60fps and 4096x2160 @ 60fps
• Verified that the preview is smooth and free of mosaic artifacts at 4K resolution.
• Verified that lower resolutions (1080p) and other formats (MJPEG) still function correctly.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[stephematician]

I see you are trying to prevent obs_source_output_video from being called if we encounter the EAGAIN error.

This solution works.

Another option would be to handle the error inside v4l2_thread rather than inside the v4l2_decode_frame, i.e. something like:

// v4l2_decoder.c
// remove #include <libavutil/error.h>

// v4l2_decode_frame()
    r = avcodec_receive_frame(decoder->context, decoder->frame);
    if (r < 0) {
        blog(LOG_ERROR, "failed to receive frame from codec");
        return r;
    }
// don't return 1 at the end, just 0 as before

And

// v4l2-input.c
#include <libavutil/error.h>

// v4l2_thread()
        if (data->pixfmt == V4L2_PIX_FMT_MJPEG || data->pixfmt == V4L2_PIX_FMT_H264) {
            r = v4l2_decode_frame(&out, start, buf.bytesused, &data->decoder);
            if (r == AVERROR(EAGAIN)) {
                blog(DEBUG, "decoder encountered EAGAIN, so continue with next frame");
                goto continue_queue_buffer;
            } else if (r < 0) {
                blog(LOG_ERROR, "failed to unpack jpeg or h264");
                break;
            }
        } else {
            for (uint_fast32_t i = 0; i < MAX_AV_PLANES; ++i)
                out.data[i] = start + plane_offsets[i];
        }
        obs_source_output_video(data->source, &out);

I have no preference! Well done on finding a fix 🎉

[JiangXsong]

Thanks for the detailed feedback and the alternative approach!

You're right — handling AVERROR(EAGAIN) in v4l2_thread instead of inside v4l2_decode_frame is also a clean solution, and it keeps the decoder helper function closer to the original behavior and avoids returning special values from the decode function.

I don't have a strong preference either, and I'm happy to adjust the patch to the approach you outlined if you think it fits better with the existing design.
Please let me know which direction you'd prefer, and I'll update the PR
accordingly.

[stephematician]

Thanks for reading my feedback! I also don't have a strong preference; the solution you provided works, too. Will wait and see what the maintainers say.