Questions Regarding Tequila Implementation and Quantization Details

[tuzhijun]

Hi, thanks for open-sourcing this interesting work!

I have several questions about the implementation of Tequila that I couldn't fully clarify from the paper or code:

1. Quantization Methods in models/utils_quant.py

I noticed multiple quantization methods such as ultraquant, ultraquantv2, ultraquantv3, and ultraquantv4, but only ultraquantv2 and ultraquantv3 seem to have actual implementations. Could you clarify which version is the final one used in the paper? The paper doesn't explicitly specify this.

2. Differentiable Reactivation (Bypassing STE)

The paper highlights "Bypassing STE through Differentiable Reactivation" as a key contribution, but I couldn't locate the corresponding code. Could you point me to where this is implemented?

3. Repurposing Dead Weights as Biases

The mechanism for "Repurposing Dead Weights as Biases" is a bit unclear. Is the bias applied per-channel or per-group? It would be helpful to understand how the bias is structured and initialized.

4. Inference-Time Ternary Weight Conversion

Could you provide or clarify the inference procedure for converting quantized weights into ternary weights (e.g., {-1, 0, +1}) using the learned bias? Even a fake-quant implementation would be very helpful for understanding the deployment behavior.

5. Training Configuration and Efficiency

• What batch size was used in the experiments?
• How does the overall training cost (in terms of time and GPU memory) compare to ParetoQ?
• Specifically, in UltraQuant V2, each layer introduces an additional learnable parameter of the same size as the weight matrix—this seems non-trivial in terms of memory and optimization overhead. Was this mitigated in practice?

Thanks in advance for your clarification!

[Little0o0]

Thank you for your interest and your thoughtful questions! Please find my detailed replies below.

Q1: what are ultraquant, ultraquantv2, ultraquantv3, ultraquantv4
"UltraQuant" was our project's previous name, which is now "Tequila." The versions correspond to different stages of development:

1. v1 refers to the "Minima Reactivation" method described in the paper.
2. v2 is the reported "Tequila" method.
3. v3 and v4 were experimental attempts at enhancement that are not part of the final paper.

We recognize this is confusing and will submit a PR to the repository to correct, clarify, and remove these misleading names and codes.

Q2: Bypassing STE
Bypassing the STE means that the bias term in Tequila receives pure gradients, compared with Minima Reactivation. While the STE is still used for gradients from the ternary matrix multiplication, the backward pass in ultraquantv2 shows that the gradients for the bias are used directly without passing through an STE.

Q3: Repurposing Dead Weights as Biases, Is the bias applied per-channel or per-group?
The bias is applied per-channel. During inference, the "dead weights" are fixed, allowing us to precompute the bias term (as in Equation 7 of the paper) offline.

Q4. Inference-Time Ternary Weight Conversion
After training, we convert the weights for inference as follows:

1. The weight matrix is split into a ternary weight matrix A (values {-1, 0, +1}) and dead weight matrix B, with the ternary methods like Absmean. B contains the BF values that correspond to the '0' positions in A.
2. The values in B are summed per channel and multiplied by λ to form a bias vector, b.
3. The inference operation then becomes: X * A * α + b.

Q5. Training Configuration and Efficiency

1. Batch Size: 256.
2. Training Cost: While a detailed comparison with ParetoQ was not conducted, training a 1B model on 10B tokens takes approximately 20 hours on 32 GPU.
3. Theoretically, the extra learnable parameters are not necessary. As shown in Equation 7, the gradients for the dead weights in Tequila are a biased version of the standard ternary gradients, offset by a lambda (λ) term. This means the dead weights do not need to be stored as extra trainable parameters. In our code, however, we implemented them as a separate matrix (B) to make the procedure more straightforward and easier to understand.