Issues and Features: What's happening

[winlinvip]

The current version has some known issues that we plan to fix in the future. You are welcome to help us fix them by sending a pull request.

• Basic framework for WHIP, ICE, DTLS, SRTP. FFmpeg: Add Whip Muxer support for subsecond latency streaming #1 Feature: refactor DTLS to merge it into tls_openssl.c #18 WHIP: Fix bugs #19
• NACK support. Implement NACK+RTX suppport #45
• OpenSSL DTLS active mode.
• Other SSL libraries.
• GCC for network congestion control.
• FEC for packet loss recovery.

The current known issues include:

• RTCP SR is needed to sync audio and video streams when converting RTC stream to RTMP or HLS. It helps recreate the timestamp. See WHIP: Convert OBS WHIP to RTMP failed for no RTCP SR packets. srs#3582
• Long First Screen Wait Time: Respond to the PLI request for quick decoding and rendering; otherwise, it takes a GOP's duration to display the first decoded image. A RTC encoder ought to have the capability to refresh the IDR frame when a new player joins or there is packet loss, as requested by the player through a PLI. When working within the ffmpeg framework, it's not possible to control the encoder in muxer. This means we cannot force an encoder to generate an IDR frame right away. One possible solution could be to cache the IDR frame or a GOP of frames in the RTC muxer and send it to the server upon receiving a PLI request. The player can then handle any latency issues that may arise.
• Multiple Slices Issue: If the -tune zerolatency option is enabled and threads>1, FFmpeg will encode the frame in multiple slices. This process can cause stuttering in Chrome's decoding, with only the IDR being able to be decoded. Both the IDR and P frames may be encoded to multiple slices and sent by multiple RTP packets with the same timestamp. RTC players, such as Chrome, may have issues decoding frames that have been encoded using multiple slices. As a result, Chrome may only decode some of the slices of the IDR frame and drop the P frames. If this issue occurs, it may appear as if the video player is stuttering and the decoded framerate will be equivalent to the GOP size.
• There is an issue with the timestamp of OPUS audio, especially when the input is an MP4 file. The timestamp deviates from the expected value of 960, causing Chrome to play the audio stream with noise. This problem can be replicated by transcoding a specific file into MP4 format and publishing it using the WHIP muxer. However, when directly transcoding and publishing through the WHIP muxer, the issue is not present, and the audio timestamp remains consistent. See comment for detail.
• The SDP functionality already exists in FFmpeg, so it is advisable to utilize it instead of generating SDP manually. The av_sdp_create function is employed to produce SDP for RTSP. Upon testing av_sdp_create and comparing its output with the WHIP muxer, we observed a considerable difference. It is strongly recommended to enhance the SDP functionality in the future once this patch is integrated. See here for detail.
• In addition to OpenSSL, it is necessary to provide support for alternative cryptographic libraries like GnuTLS, mbedtls, and others. To accomplish this, a dtls.c file should be created, analogous to tls.c, to improve utilization. Presently, only OpenSSL is supported, but after merging this patch, the goal is to extend support to other SSL libraries. This approach is taken because incorporating support for all SSL libraries in a single patch would result in excessive complexity and hinder comprehension for others. See here for detail.
• Currently, only the DTLS server role is supported for DTLS, as the DTLS server is much simpler than the DTLS client, and there is no need to handle the ARQ since the DTLS client will take care of it. All WHIP servers will support both DTLS server and client roles, making it safe and acceptable for FFmpeg to use the DTLS server role.
• Features such as congestion control, NACK, and FEC are not yet implemented. While NACK is essential, FEC remains optional.
• Currently, only the first candidate (UDP and host type) is utilized. A mechanism to determine the fastest candidate, select the best one, or switch during publishing is needed.
• While capturing the screen using FFmpeg, the h.264 profile may be negative (e.g. -99), in which case we set the profile to 0x42 (baseline). However, in such a situation, it may be necessary to parse from SPS/PPS.

Below are the issues that have already been fixed:

• Need to improve the end-to-end latency to 150ms from 800ms. Fixed by @T-bagwell @mypopydev @winlinvip
• For SRS to convert RTC to RTMP using time information, it is essential to send an RTCP SR report. Fixed by @duiniuluantanqin
• Only send binding requests to the server and cannot receive binding requests from the server. Fixed by @cloudwebrtc. In addition, he assists in enabling ICE use-candidate for Janus or Pion to finalize the ICE handshake.
• WHIP currently supports only HTTP, with HTTPS yet to be implemented. However, incorporating HTTPS should be straightforward, as ffmpeg already contains the necessary code. Note: The ffmpeg HTTP library has the functionality to support both the HTTP and HTTPS protocols.
• Support baseline/main/high profile without B frames. Even though WebRTC has the capability to support x264 main and high profiles without B frames, it is advisable to use the baseline profile for better compatibility. See bd9f7d1 @duiniuluantanqin
• During the DTLS handshake, when FFmpeg receives a ServerHello from the server, it generates a ClientHello for retransmission with a Certificate. This causes the server to retransmit the ServerHello, which still works but is not efficient. We should try to eliminate the unnecessary retransmission of the ClientHello. See 3b3b17a @winlinvip
• Compatibility is limited to OpenSSL 1.0.1k or newer versions. To prevent build failures, it may be necessary to ensure compatibility with older versions. See d68c259 @winlinvip
• When remuxing WebRTC to RTMP or HTTP-FLV, stuttering can occur every N seconds, which may be the same as the GOP size. Note that this bug occurs during screen streaming, but there is no issues when converting a file to WHIP streaming. After using the h264_mp4toannexb BSF, the RTC2RTMP works better, without stuttering issues anymore. See winlinvip@6598ffc @winlinvip
• Use h264_mp4toannexb to convert MP4/ISOM to annexb. Since the h264_mp4toannexb filter only processes the MP4 ISOM format and bypasses the annexb format, it is necessary to manually insert encoder metadata before each IDR when dealing with annexb format packets. For instance, in the case of H.264, we must insert SPS and PPS before the IDR frame. See here for detail. @winlinvip
• When duplicating a stream, the demuxer already sets the extradata, profile, and level of the par, so this function won't be invoked. When using an encoder like libx264, the extradata in par->extradata contains the SPS, including profile and level information, but the par's profile and level are not specified. It is essential to extract the profile and level data from the extradata and assign it to the par's profile and level. Remember to enable AVFMT_GLOBALHEADER, or the extradata will be empty. See eb0d0c0 @mypopydev @winlinvip