💡Workflow Wednesday: From ETL to AITL - When AI Joins the Pipeline

Art by @basilonmypizza: https://lnkd.in/eF8FkWzN - https://basilhefti.ch/

Not long ago, AI lived on the sidelines.

The typical pattern? Data gets processed, copied into a data science system, analyzed using fragile scripts and models - then merged back. It was clunky, isolated, and clearly separated from the “real” IT stack.

That’s changing fast with AITL (AI-in-the-Loop ETL).

Today, AI and GenAI are being embedded directly into the ETL pipeline. Why? Because traditional tools like SQL or regex are not sufficient.

Think about:

🧬 Fixing misspelled names

🗂️ Categorizing customer intent

🏥 Extracting diagnoses from clinical notes

Trying to tackle such tasks with rules is not working. And GenAI provides the needed capabilities, working with normal language, not special syntax.

🚀 The shift brings three big wins:

No hand-offs. AI is part of the flow. No detours or brittle integrations.

Consistent data. Processing is synchronized across steps, improving quality (consistency is one of the 6 C of data quality).

Unified ops. Same orchestration tools, same dependency graph, same observability.

In short: While AI was for nerds, GenAI is for the cool cats. And is making your data pipelines simpler and more powerful. Because this isn’t “experimental” anymore. It’s necessary.

What’s the first step you’d promote from script to AITL?

Art: https://lnkd.in/eF8FkWzN https://basilhefti.ch/

Booklets:

• https://lnkd.in/eBYXUpqB Collected Posts Volume 1

• https://lnkd.in/et7THkXs Collected Posts Volume 2