Asymmetric Quantization Offset Overflow Issue with Large Activation Ranges

[mbdibd]

ssue Summary
When using asymmetric quantization, I encountered an overflow issue with the activation offset (zero point). This leads to abnormal inference results or significant accuracy drops. The problem typically occurs when the activation distribution has a large range or contains extreme outliers, causing the computed offset to exceed the supported numerical range for quantization.

Request for Help
Are there recommended quantization algorithms or parameter settings to effectively avoid offset overflow?
Does Qualcomm's quantization toolchain support automatic clipping, dynamic quantization, or other mechanisms to mitigate this issue?
Are there best practices or example cases you can share?

[quic-hitameht]

Hi mbdibd Thanks for reporting this issue. In AIMET when computing offset, we ensure that that the offset stay within the valid bounds but offset clamping can cause accuracy degradation if the activation contains extreme outliers.

Recommended strategies to mitigate offset overflow issue:

1. Use Tf-Enhanced calibration - optimizes the encoding range by minimizing the quantization noise, which handles the outliers better.
2. Use higher bitwidth for problematic layers - For layers with extreme activation ranges, consider using higher bitwidth

Also, could you share what offset/encodings value you’re seeing? A minimal repro would be very helpful.

[quic-bhushans]

mbdibd could you please share more info with Hitarth Mehta (@quic-hitameht)