Theory Thursday: From Video to Hidden Find: How AI Searches

Art by @basilonmypizza: https://lnkd.in/eF8FkWzN - https://basilhefti.ch/

Recently, I wanted to buy a second wooden chess set. The problem: it is no longer manufactured. After searching stores in different countries for years, I tried a different approach:

Upload a video of the product and ask AI to find shops that still carry it.

It worked. And it felt a little like magic.

The underlying method is surprisingly simple.

A multimodal AI system can split the video into representative frames and turn these images into embeddings: numerical vectors that capture what is shown.

Think of two hands on a watch. If they point in almost the same direction, the angle between them is small. At a quarter past 12, the angle is 90 degrees.

Image embeddings work similarly: the smaller the angle between two image vectors, the more similar they are.

The hard part is learning an embedding space in which the “right” things actually end up close together. But this is another story.

Combined with web search, the system can then find candidate listings, compare them with the object in the video, and rank the best matches.

Images, video, audio and documents contain vast amounts of information. Multimodal AI makes much of it accessible.

For companies, this can mean finding a spare part from a photo, finding similar defects in maintenance images, or retrieving a past case.

So, the next time you search for something unusual, try the assisted approach and let me know how well it worked.

PS: the story is not quite over yet. The shop has apparently misplaced the wooden chess set in its storage and is now trying to find it. Fingers crossed.

• Art: https://lnkd.in/eF8FkWzN https://basilhefti.ch/