Theory Thursday: From Noise to Choice

Art by @basilonmypizza: https://lnkd.in/eF8FkWzN - https://basilhefti.ch/

Prompt Sandwich, Context Engineering, and Cleaner AI Answers

The office corridors are getting quieter by the day. Summer vacations. Is it for the heat that conversations with LLMs can be weird? After a handful of exchanges the answers spin in circles. We see the models hallucinate or repeat itself.

Researchers call this the “lost-in-the-middle” effect: the LLM pays close attention to start and end of the prompt, ignoring the middle (hence the «prompt sandwich»).

Why?

Picture the model as a zipped encyclopaedia: billions of weights that compress its entire training corpus.

Your prompt is the decompression key. It tells the model which page to unzip.

Noise (stray JSON, meeting jokes, or half-cooked ideas) blurs the key. The model inflates the wrong paragraph.

That’s where context engineering enters. Classic machine learning and data-science tricks help to keep the context clean:

Ranking & re-ranking (such as maximal marginal relevance, reciprocal rank function, or ranked list fusion) keep the most important knowledge on top.

Graphs (such as GraphRAG and cousins) follow entity links to pull only the facts that matter.

Pruning (such as learned filters and reinforcement) knock out passages the model is unlikely to need.

And Philipp Thomann’s approach to pre-compute questions a user is likely to ask.

It’s an old insight: things work better if you trim the noise.

Meanwhile, here are a few practical tricks on how to avoid the «prompt sandwich»:

🔄 Context-Reset Reflex: when you sense looping or drift, open a fresh chat (or clear history).

🎭 Role-Switch: start with “You are an AI compliance officer reviewing …”; resetting the role refocuses attention and sidelines irrelevant crumbs from earlier.

🧐 “Is this sane?” Check: before trusting the output, pause and ask whether it could possibly be true. The gaps reveal themselves fast.

A disciplined context costs nothing and beats letting random noise creep in while everyone else is at the beach. 🌊

What tricks keep your LLMs on track? I’d love to hear.

• Happy happy Birthday, Hanspeter Högger, Xavier Rouault and Simon Scheurer

• Apple paper: https://lnkd.in/e6sSvm7y

• Learning is compression: https://lnkd.in/ebbq-TtY

• How GPT searches for words: https://lnkd.in/eN5sDx5t

• Simon Willison on Context Engineering https://lnkd.in/ekkYYebJ

• Art by @basilonmypizza: https://lnkd.in/eF8FkWzN https://basilhefti.ch/. Stay tuned for a making off of these cool visualisations.

• Earlier posts as PDF: https://lnkd.in/eBYXUpqB and https://lnkd.in/et7THkXs

• As many of you, this post series takes a summer break - looking forward to see you all again in August