Workflow Wednesday: A Fireworks Way to Understand Data

Art by @basilonmypizza: https://lnkd.in/eF8FkWzN - https://basilhefti.ch/

Remember when you were a child on Silvester and were allowed to light a small firework for the first time? Chances are you were in awe, a little scared, and fascinated by the light at the same time. You kept your distance, careful about the splinters - and drawn to it.

Working with data often feels similar. You are excited to get access and eager to start, but you do not yet know where to begin. Everything is dark: large schemas, cryptic names, outdated or missing documentation. You proceed carefully, trying a few queries, hoping something lights up. Sometimes it does. Often it just fizzles.

So what is a good way to understand the data at hand? Here is my process.

I start by extracting the raw structure: system → table (or folder) → column, plus types, keys, and basic constraints if available. This is the orientation step. Like a small firework on the ground, it gives you an initial overview: where things stand, how they are grouped, and what might connect to what.

Next, I focus on meaning. Using an LLM, each data product and data element gets a short semantic description of what it likely represents. I prefer to keep these under 10 words, but choose the setup that works best for you. This is like adding colour to the fireworks: it enriches the picture.

All of this runs locally, for example with Ollama. No schemas, no samples, and no data are sent outside your system.

Then I look for connections in the data. Candidates are identified through shared semantics. I often add a few sample values, for example 10 random rows, which contain hints about how data needs to be processed to become connectable. This is when individual fireworks synchronise into larger patterns and figures.

At this point, we are no longer in the dark. Structure, meaning, and connections are visible. Now I can ask the LLM for what I actually need: given this setup and my information need, produce the queries.

When everything aligns, the data comes together in a clean, satisfying finale. A fitting way to end the year. 🎆

Thank you for following my work this year. Wishing you a Happy New Year.

Art: https://lnkd.in/eF8FkWzN https://basilhefti.ch/