Theory Thursday: Energy Efficiency as an Innovation Driver ✅

Art by @basilonmypizza: https://lnkd.in/eF8FkWzN - https://basilhefti.ch/

Today’s AI is energy-intensive. I get many questions about it. And it is true: today’s AI is largely a brute-force approach. Clumsy in a way. We are very happy to (finally) have working systems with astonishing capabilities. As long as we have no better options, we feed energy into the systems we have. A bit like the first cars with incredibly low miles per gallon. And as with these cars, the energy efficiency of AI will eventually improve. And likely unlock additional capabilities along the way.

When energy becomes a constraint, architecture matters.

That’s why a 2022 DeepMind paper on RETRO is back in focus. The approach retrieves external text for a conversation, a common approach for well-grounded chat systems (retrieval-augmented generation, or RAG). Unlike most RAG systems, RETRO uses cross-attention: the retrieved information is integrated without making the prompt longer. This matters because prompt length is one of the dominant drivers of energy consumption in autoregressive LLM inference.

In other words: people are actively searching for and experimenting with ideas that achieve the same results with less effort. Be it fewer parameters or shorter prompts. And this is not just theoretical. Microsoft shows that cost-aware training of their Search-R1 algorithm improves exact match by about 5% on average while using roughly 20% fewer inference resources.

AI is developing at high speed. I’m curious to see which additional architectural ideas we will find next. Ideas that improve AI both in terms of energy efficiency and in how we design these systems in the first place.

• Helia Hashemi et al, https://lnkd.in/e_f9FvUG, Cost-Aware Retrieval-Augmentation Reasoning Models with Adaptive Retrieval Depth (Oct 25)

• Art: https://lnkd.in/eF8FkWzN https://basilhefti.ch/