2026-06-09 Workflow Wednesday: Whose Turn Is It to Talk?

Art by @basilonmypizza: https://lnkd.in/eF8FkWzN - https://basilhefti.ch/

As kids, we built our own telephone: two tins and a cord were enough for a remote communication device. We loved it. And tested its limits.

And limits it had. Around corners it failed. In fact, anything touching the cord broke the connection. Communication was not fluent either. You either talked or listened, to the point that often nobody knew whose turn it was.

Granted, voice bots have come a long way since. Still, when you build audio conversation applications, the challenge remains the same: fluency. Fast responses. Smooth turn-taking. Easy interruptions. Natural flow.

To get there, the system must work close to the audio itself. That is why realtime speech-to-speech systems matter. OpenAI's Realtime API, for example, processes and generates audio directly, instead of chaining speech-to-text and text-to-speech, the old-school approach.

How does it stay close to the audio? Text systems work with text tokens. Voice systems work with audio tokens. And tokens are processed with the centerpiece of modern AI: embeddings.

An embedding maps a token into a high-dimensional vector space. The better that mapping, the better the system connects what was said, what was meant, and what to say next. Modern embedders such as Gemini Embedding 2 can even bring text, images, video, and audio tokens into one shared semantic space. 

For voice systems, the practical test is simple: can it listen, respond, pause, recover, and be interrupted without breaking the flow? If it breaks, you know something is still "touching the cord".

If you build voice systems: what input trips them up? If you don't yet: what are you waiting for?

—

• Art: https://lnkd.in/eF8FkWzN https://basilhefti.ch/

• Thanks to Philipp Thomann for bringing this to my attention.

• As a curiosity: the common embedding size of 768 has roots in graphics processing; to process images, they were cut into patches of 16x16 pixels with 3 color channels, exactly 768 numbers.