Trend Tuesday: Small Models, Big Brains: AM-Thinking-v1 and the Rise of Tiny Titans

Art by @basilonmypizza: https://lnkd.in/eF8FkWzN - https://basilhefti.ch/

The trend towards small large language models continues (or should we just call them small language models now?). One interesting example: AM-Thinking-v1, published this May.

🖥️ It’s compact enough to run on a MacBook Air in quantized form.

🏘️ It wasn’t built by the usual suspects, but by an internal R&D team at real estate platform Beike (ke.com).

🧠 And it achieves remarkable reasoning capabilities through supervised finetuning and reinforcement learning, leveraging well-curated tasks from math, code, science, and general chat.

Once again, we’re seeing that learning is compression (see earlier post). These models are essentially highly efficient databases. Just not the SQL kind. Instead of tables and queries, they store knowledge in weights and retrieve it through attention mechanisms and prompts.

But how does this compression work? Some clues come from recent research:

🔍 Hong et al. show that stronger models form specialized weight clusters, like little islands of factual knowledge, ready to be accessed.

🧩 Yao et al. map out “knowledge circuits”: coordinated groups of attention heads and MLPs that store specific facts, acting like distributed memory cells.

Why care?

Because we need smaller models for phones, drones, or energy-sensitive deployments. And because understanding compression will help us better mix pre-trained and contextual knowledge. Bonus: it might even teach us something about our own brains.

Exciting times.

👇 What’s the smallest model you’ve run locally? And what did you use it for?

• Learning is Compression 📚✨ https://lnkd.in/ebbq-TtY

• https://lnkd.in/e58ZE-az: AM-Thinking-v1: Advancing the Frontier of Reasoning at 32B Scale

• https://lnkd.in/eWke_Jrt: The Rise of Parameter Specialization for Knowledge Storage in Large Language Models

• https://lnkd.in/epwTc78a: Knowledge Circuits in Pretrained Transformers

You may find my earlier posts also as PDF here:

• https://lnkd.in/eBYXUpqB Collected Posts Volume 1 as pdf

• https://lnkd.in/et7THkXs Collected Posts Volume 2 as pdf

Big thanks to Christian Erni, Lukas Hefti, and Basil for the design, booklets, and site.

Art: https://lnkd.in/eF8FkWzN https://basilhefti.ch/