Google Research introduced the TurboQuant quantization method for large language models

Google Research has published a quantization method called TurboQuant, which significantly reduces the cost of running large language models. The main issue with long AI conversations is the rapid growth of the KV cache (key-value cache — the memory that stores context). This research primarily addresses that bottleneck when handling long contexts. TurboQuant compresses KV cache data down to 3 bits per element (compared to the original 16 or 32 bits) with minimal accuracy loss. The proposed algorithm also works without data‑specific calibration, unlike many other quantization methods that require dataset runs for tuning.

The approach combines two key techniques: 🔵PolarQuant converts the Cartesian coordinates of KV cache vectors into polar form. This preserves critical angular information and removes the need for normalization, which typically distorts data during compression. 🔵QJL (Quantized Johnson‑Lindenstrauss) corrects compression errors after PolarQuant using 1‑bit projections, ensuring high response accuracy.

The researchers tested the approach on Gemma and Mistral models and achieved impressive results: 🔵memory usage dropped sixfold; 🔵attention kernel computations ran up to 8× faster; 🔵the model maintained baseline accuracy even on ultra‑long‑context tasks (LongBench benchmark).

TurboQuant makes it possible to run heavyweight models on standard hardware and dramatically cut cloud compute costs.

💬 Discuss