Study finds weight order matters more than value in LLM compression
Researchers compress models to 16–64 shared values and find rank is king
The facts
- Researchers Borja Aizpurua, Sukhbinder Singh and Román Orús applied scalar K-means clustering to replace each weight matrix with just 16–64 shared values
- Tested on Llama 3.1-8B-Instruct and SmolLM2-135M, the method preserves strong accuracy without retraining
- Fine-tuning only the cluster centroids recovers 30–40% of the remaining accuracy loss, the paper reports
- Breaking the relative rank ordering of weights sharply increases perplexity, while changing values but keeping ordering causes little loss in many middle and late layers; variance and spatial geometry of cluster assignments also matter, and early layers stay more sensitive, a pattern observed across four LLMs from two architecture families
A new paper posted to arXiv compresses large language models by collapsing each weight matrix down to between 16 and 64 shared values. It uses scalar K-means clustering to do this. The authors report that the resulting models keep strong accuracy even without retraining.
The authors are Borja Aizpurua, Sukhbinder Singh and Román Orús. They tested the technique on Llama 3.1-8B-Instruct and SmolLM2-135M. Fine-tuning only the cluster centroids, they say, recovers 30 to 40 percent of whatever accuracy the clustering loses. That is a simple route to shrinking a model's footprint on disk, since a weight matrix that once needed billions of distinct floating-point values can be described with a handful of shared ones plus an index.
The more striking part of the paper is what it does with clustering as a diagnostic rather than a compression trick. By holding the cluster assignments fixed and perturbing only the shared values, the authors isolate which property of a weight actually carries information. They find that preserving relative rank — whether one connection is stronger or weaker than its neighbours — matters far more than preserving the exact magnitude. Scrambling that ordering sends perplexity sharply higher. Changing the underlying values but keeping their rank intact causes little degradation in many middle and late layers.
Rank alone is not the whole story. The paper also finds that performance depends on keeping the variance of the reconstructed weights close to the original. Early layers remain more sensitive to perturbation than later ones. Perturbing the cluster assignments themselves shows the spatial geometry of weights is load-bearing too. The authors say they observed the same pattern across four LLMs drawn from two architecture families, which they take as evidence the finding is not an artifact of one model's training run.
No code, model weights or benchmark release accompanies the abstract. The paper does not state inference speedups or wall-clock compression ratios beyond the 16–64 value count per matrix.
Why it matters
If ordering rather than precision carries the signal, quantisation and pruning schemes that protect rank could compress further than magnitude-preserving methods allow. Engineers building compression pipelines now have a concrete lever — rank and variance, not raw values — worth testing before assuming aggressive quantisation will hurt accuracy.
Primary source
arXiv cs.CL — arxiv.orgWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.