[Question]: (H.264) How to enforce limit of maximum encoded frame size in bits?

[samisallinen]

Our streaming application requires minimum latency, so we'd like to enforce a maximum for encoded frame size in bits to have an upper bound for the network transfer delay for the encoded data.

We're initializing the encoder like this:

  SETPROPERTY(pEncoder, AMF_VIDEO_ENCODER_USAGE, AMF_VIDEO_ENCODER_USAGE_ULTRA_LOW_LATENCY);          
  pEncoder->SetProperty(AMF_VIDEO_ENCODER_B_PIC_PATTERN, 0);
  SETPROPERTY(pEncoder, AMF_VIDEO_ENCODER_TARGET_BITRATE, bitRate);
  SETPROPERTY(pEncoder, AMF_VIDEO_ENCODER_FRAMERATE, ::AMFConstructRate(frameRate, 1));

  SETPROPERTY(pEncoder, AMF_VIDEO_ENCODER_RATE_CONTROL_METHOD,    AMF_VIDEO_ENCODER_RATE_CONTROL_METHOD_LATENCY_CONSTRAINED_VBR);     
  SETPROPERTY(pEncoder, AMF_VIDEO_ENCODER_ENFORCE_HRD, true);
  SETPROPERTY(pEncoder, AMF_VIDEO_ENCODER_FILLER_DATA_ENABLE, false);
  SETPROPERTY(pEncoder, AMF_VIDEO_ENCODER_VBV_BUFFER_SIZE, static_cast<amf_int64>((1.3 * bitRate / (frameRate)))); 

Then before submitting every frame, we've tried various combinations of the following (including none of them):

  SETPROPERTY_BOOLRET(AMF_VIDEO_ENCODER_RATE_CONTROL_METHOD, AMF_VIDEO_ENCODER_RATE_CONTROL_METHOD_LATENCY_CONSTRAINED_VBR); 
  SETPROPERTY_BOOLRET(AMF_VIDEO_ENCODER_TARGET_BITRATE, static_cast<amf_int64>(0.75*m_bitRate * m_currBandwidthAdjustment));      
  SETPROPERTY_BOOLRET(AMF_VIDEO_ENCODER_PEAK_BITRATE, static_cast<amf_int64>(0.85*m_bitRate * m_currBandwidthAdjustment));
  SETPROPERTY_BOOLRET(AMF_VIDEO_ENCODER_VBV_BUFFER_SIZE, static_cast<amf_int64>((1.3*m_bitRate * m_currBandwidthAdjustment / (m_frameRate))));      
  SETPROPERTY_BOOLRET(AMF_VIDEO_ENCODER_INITIAL_VBV_BUFFER_FULLNESS, static_cast<amf_int64>(32));

Being desperate enough, we've tried various fudge factors on the target and peak bitrates and the vbv buffer size.

However, when there are step changes in the input video data, we get a lot larger frames output than what we are asking for:

(Target bitrate 8Mbit/s, individual frame sizes (purple graph) hit ~19Mbit/s. The average bitrate seems to be somewhat close to what we're asking.

We have tried a lot of different combinations and parameters, including AMF_VIDEO_ENCODER_RATE_CONTROL_SKIP_FRAME_ENABLE, but so far haven't found the solution.

Any advise on what we're doing wrong would be greatly appreciated.

[rhutsAMD]

Please use AMF_VIDEO_ENCODER_MAX_AU_SIZE to enforce a hard-limit on frame size.

[samisallinen]

Thanks for the tip and the quick reply! I must confess I had overlooked that option.

AMF_VIDEO_ENCODER_MAX_AU_SIZE also seems to have an effect on the average bitrate, but when there are abrupt scene changes in the video, the individual frame size still shoots up to 9.5 Mbits per frame with a 8Mbit/s bitrate target and a 300KBit AU size limit (25Hz frame rate).

The setup we have is a Ryzen AI MAX+ Pro 395 with Radeon 8060s, using Windows 11 and driver 32.0.22021.1009.

Here is the AMF log of one such session in case it shows me doing something stupid...

amflog.txt

Here is the maximum output frame size of each 20 frame period logged while recording the above amf log.

[rhutsAMD]

Please consider the following suggestions for your encoding settings:

1. Your current VBVBufferSize is set to less than or equal to the average frame size, which is not quite reasonable. It would be more effective to set the VBVBufferSize to be the same as the TargetBitrate
2. The MaxAUSize is also set to the average frame size, which is unexpected. Typically the MaxAUSize should be larger than the average frame size. A more reasonable value would be something like 8 times the average frame size or perhaps 5 times the average frame size.
3. For a 1280x720p@25fps stream, a target bitrate of 8192000 seems a bit excessive. It may be more efficient to set it to less than 2500000 to achieve better results.

Additionally, you can try the CBR rate control method with ENFORCE_HRD / FILLER_DATA enabled as a trade-off between higher latency but more constant bitrate.

One more thing to keep in mind is that the actual bitrate fluctuates up and down around the TargetBitrate. It may be even higher than the PeakBitrate in specific short instances. This is normal and represents a balance between video quality and bitrate smoothness.

In regards to your current set of parameters, the particular AU value doesn’t enforce a strict restriction because of the nature of the rate control algorithm. This is because rate control is heuristic by nature and tailors to a need to preserve video quality. For instance, if rate control predicts a significant drop in quality due to this restriction, then it will relax the limit internally. In the current rate control design, video quality takes priority over other aspects such as bitrate accuracy and frame size. We understand that this aligns better with our typical use cases and current industry trends.

[samisallinen]

Hi Roman,
Thank you for your quick response again & sorry for being a pain.

The VBV Buffer size value was just a result of a lot of exploration trying to find some way to cap the max frame size.
The bitrate of 8Mbit/s - hoping I am counting the zeros correctly ;-) - was also just a test case, I did test other values in the 2-4 Mbit/s range too.

The issue with out use case (cloud gaming) is that we measure the bitrate we have available to the client, and then try to utilize that as well as we can. However, if we have let's say a 8Mbit/s pipe available to the user and the encoder sometimes produces 5-8x the amount of bits per frame, then it will also take 5-8 frame times to transmit those bits to the user, which is then visible as highly stuttering motion to the user. Thus the desire for a relatively strict bitrate cap.

I took the bitrate control algorithm selection from here:
https://github.com/GPUOpen-LibrariesAndSDKs/AMF/wiki/Rate-Control-Methods

The explanation of Latency Constraint Variable bitrate (LCVBR) would sound like it should more or less perfectly fit our purpose.

Best Regards,
-Sami

[samisallinen]

We've been unable to figure out any method of properly capping the frame size so far.
AMF_Video_Encode_API.pdf seems to hint that this should be possible:

Any clues on if this is really possible or not? The peak frame size of 5-8x average or target bitrate is really a killer for us.

Best Regards,
-Sami

[MikhailAMD]

Please provide full set of parameters based on Roman's suggestions. Also try CBR.