Knowledge distillation
2 stories. An explainer is being written.
Knowledge distillation in the news
·arXiv cs.CL
Study finds weight order matters more than value in LLM compression
Researchers compress models to 16–64 shared values and find rank is king
·OpenAI
OpenAI says it disrupted a Moonshot AI-linked reasoning extraction campaign
A cluster of over 15,000 accounts tried to copy hidden chain-of-thought from OpenAI models before being shut down.