đźš› Workflow Wednesday: Truck, Check, Ship
Art by @basilonmypizza: https://lnkd.in/eF8FkWzN - https://basilhefti.ch/
Turning raw bytes into gold - without a magic wand.
Data ingestion is no alchemy. It’s the loading dock of your analytics platform. When a truck backs up, you check the pallet, sign the delivery note, and roll it inside. Straightforward, right?
Then why the headaches?
Because the source changes without notice. Deliveries arrive via too many routes. Source systems groan under complex queries. And ingestion tries to do everything at once.
The fix? Spot the drift, standardise the on-ramps, ask for raw tables, and defer polishing. You don’t need sorcery. Just better routines.
Here is a simple recipe:
Step 1: Explore 🔍
Pull a one-off extract. Explore it. Is the data complete, decipherable, and - most importantly - valuable?
Followed by a decision gate: if the data provides no value, there is no need to do a proper onboarding.
Step 2: Contract đź“ś
Define data content, cadence, and structure. Document everything: data dictionary, lineage, tests. And agree on it. If it’s not written down, it won’t be enforced (machine-readable data contract).
Step 3: Operate ⚙️
Build the proper data ingestion pipeline. Use the contract for the automated “goods-in check”- exactly like signing for a delivery.
We typically do step 1 in the lab, while the operation (step 3) runs in the fab.
Behind the routine sit a few helpful principles:
Pull data directly from the source whenever you can (middlemen, like in real life, tend to complicate things). Go to full granularity. Be tolerant on input. If a change does not hurt you, accept it. And take data in 1:1, no renames etc. (copy first, enrich later).
When you redesign and simplify the data ingestion based on these patterns, you will save time, reduce errors, and unlock the real value of your data 🔓
👉 Which ingestion pain point trips you up the most-and how are you tackling it?
• Happy Birthday, Dominique Im Obersteg
• Art by @basilonmypizza: https://lnkd.in/eF8FkWzN https://basilhefti.ch/
• Earlier posts as PDF: Collected Posts Volume 1 https://lnkd.in/eBYXUpqB and Collected Posts Volume 2 https://lnkd.in/et7THkXs
