Skip to content

IMPLEMENTATION_SPEC.md

Latest commit

 

History

History
74 lines (48 loc) · 2.45 KB

IMPLEMENTATION_SPEC.md

File metadata and controls

74 lines (48 loc) · 2.45 KB
Raw

Implementation specification: TurboAngle and TurboQuant-style KV compression

This repository keeps its algorithm notes in-tree so it can be published independently of the parent Gemma workspace.

Primary references

These paper links are kept here directly so the repository does not need a separate paper-notes page for core references.

Empirical angle sanity check

The repository now includes a simple post-FWHT angle histogram tool for model-derived key vectors:

  • plot script: bench/plot_post_fwht_angle_histogram.py
  • example output: docs/assets/post-fwht-angle-histogram.png

On a tiny-random/gemma-2 layer-0 run, the current checked-in stats were:

  • KL divergence to a uniform 256-bin angle distribution: 0.0456
  • max bin probability deviation: 0.0039

This is not a proof of exact uniformity, but it is useful supporting evidence that uniform angle bins are not grossly mismatched after the randomized transform.

Core implementation contract

  • Input vectors are rows of shape [B, d]
  • d must be a power of two for the canonical FWHT path in v1
  • K and V may use different layerwise policies and different codecs

TurboAngle reference path

  1. Random diagonal sign rotation D
  2. Orthonormal Walsh-Hadamard transform H
  3. Pairwise polar decomposition of transformed coordinates
  4. Uniform angle quantization
  5. Quantized norm storage

Required invariants

  • FWHT preserves vector norm within float tolerance
  • encode/decode is deterministic for fixed seed + config
  • measured error decreases as angle bins increase

TurboQuant path (clean-room target)

The repository should move from the current TurboQuant-inspired scalar-codebook baseline toward a fuller TurboQuant implementation that includes:

  • random rotation
  • scalar/codebook quantization on rotated coordinates
  • norm handling for non-unit vectors
  • residual / QJL-style correction path for the inner-product-aware variant

Storage reporting

Every codec must report:

  • estimated bits per element
  • compression ratio vs fp16 KV
  • reconstruction metrics
  • later: attention distortion and model quality metrics

Benchmark layers

  1. Reconstruction-only
  2. Attention distortion
  3. Model quality
  4. Runtime and memory

Claims discipline

Use lossless only for explicitly stated metrics and workloads. Do not generalize synthetic reconstruction numbers into full-model quality claims.