Will the Agent Take the Jobs of Our Data Engineers? Two Execution Models for One Data Lake and What Changes in 2029

In three years, AI agents will write your Airflow DAGs. But who will build the pipelines your auditors need to see?

This session confronts the question every data leader is asking: As agentic AI automates data pipeline creation, what happens to our Data Engineering teams? Drawing from real-world experience managing global manufacturing IT infrastructure, I'll show why the answer is neither "yes, they're replaced" nor "no, nothing changes", but something more nuanced and immediately actionable.

The core thesis: Agentic AI doesn't just automate pipelines, it fundamentally challenges the Medallion Architecture itself. If an agent can transform Bronze data on-demand for any question, why pre-curate Silver and Gold layers? Yet this only works in one context. Modern enterprises actually need two execution models for the same data lake:

Model A (Code-First + Agentic / Bronze-Only Architecture): AI agents query raw Bronze data directly, generate Airflow DAGs on-demand, and create ephemeral transformations. No pre-built Silver/Gold layers—the agent does it just-in-time. Fast, iterative, perfect for operations and predictive analytics. CI/CD-governed, but machine-written.

Model B (Visual + Governed / Full Medallion: Bronze → Silver → Gold): Human-defined workflows in platforms like Alteryx, building curated Silver and Gold layers. Why? Because auditors, regulators, and compliance officers need persistent, traceable, human-readable data pipelines. Increasingly augmented by MCP (Model Context Protocol) servers for coding agents, but fundamentally governed by visual audit trails.

Both require rigorous CI/CD practices, but they represent fundamentally different data architecture philosophies and require different human skills.

Key Takeaways:

• Why agentic AI makes Silver/Gold layers optional in operational contexts, but mandatory in compliance contexts
• Which pipelines can (and should) be agent-built vs. human-controlled
• How CI/CD for data pipelines becomes the governance layer for both models
• What "Data Engineer" means when agents write the code and curate on-demand
• Practical examples from automotive manufacturing: OT data, MES integration, multi-country compliance