Tags
Share
Data engineering is entering one of the fastest periods of change in its history.
The fundamentals have not disappeared. Enterprises still need reliable ingestion, transformation, quality, governance, storage, and access. But AI is changing both what data infrastructure must support and how that infrastructure gets built and operated.
The change is already visible. dbt Labs’ 2026 State of Analytics Engineering Report found that 72% of respondents prioritize AI-assisted coding, while only 24% prioritize AI-assisted pipeline management, including testing, quality, and observability. The imbalance is revealing: AI is accelerating the creation of data assets faster than many organizations are strengthening the systems required to trust and operate them.
Over the next three years, the biggest change will not be that AI simply writes more SQL.
Data engineering is moving from a discipline centered heavily on constructing pipelines toward one centered on orchestrating increasingly automated, context-aware, real-time, and governed data systems.
That does not make data engineers less important. It changes where their value concentrates.
dbt Labs has described the effect through Jevons paradox: when automation makes individual data-engineering tasks faster and cheaper, demand for data capability may actually increase because organizations can attempt more use cases. The result is likely to be less repetitive pipeline work but more demand for architecture, platform engineering, automation, domain understanding, governance, and reliability.
For enterprise data leaders, five trends are particularly important to plan for between now and 2029.
The next phase of data engineering will be shaped less by one new tool than by five connected shifts in how data is built, understood, governed, and delivered to AI.
Your AI Partner
Get Ahead of the Data & AI Curve.
Exadel helps modernize data foundations for AI, automation, and real-time operations.
Trend 1: Agentic Data Pipelines — AI That Builds and Fixes Itself
The most forward-looking change is the move from AI-assisted pipeline development toward agentic pipeline operations.
An agentic data pipeline is not simply a pipeline containing an AI model. It is a data workflow in which an AI agent can reason about the engineering environment, use tools, inspect context, take a sequence of actions, and validate the result within defined controls.
Consider a failed production pipeline.
Today, an engineer may:
- Receive an alert.
- Open logs.
- Inspect recent code changes.
- Check schemas and lineage.
- Identify the likely cause.
- Write a fix.
- Run tests.
- Deploy the change.
An increasingly agentic workflow can compress that sequence:
Detect failure → inspect logs and lineage → diagnose probable cause → propose remediation → run tests → present validated change → engineer approval / controlled deployment
Parts of this are already emerging.
In May 2026, dbt introduced its Developer Agent in preview, designed to work with the context of an actual dbt project—including lineage, tests, contracts, and governed definitions—rather than producing generic SQL without understanding downstream consequences.
Confluent has likewise introduced agent-oriented capabilities that allow AI to interact with streaming operations through Model Context Protocol infrastructure and agent skills.
But there is an important difference between agentic and uncontrolled.
The near-term enterprise model is unlikely to be:
AI detects a problem → AI rewrites production → nobody checks.
A more credible progression is bounded autonomy. Organizations define which changes an agent may propose, which tests it must pass, which environments it can access, what identity it operates under, and when a human must approve the action.
Our view at Exadel is that this shift will also change data architecture itself. Agents need governed identities, traceable access, reliable semantic context, observability, and auditable records of what they read and changed.
In other words, self-managing pipelines require more engineering discipline, not less.
The move toward agentic pipelines is really a move from manual incident response toward a controlled loop of detection, diagnosis, validation, action, and learning.
Suggested flow:
Monitor → Detect → Diagnose → Propose → Test → Approve/Act → Observe outcome → Learn
Cross-cutting controls:
Agent identity | Lineage | Access | Policy | Auditability | Human escalation
Trend 2: The Rise of the AI Data Engineer as a Distinct Role
The second trend is organizational.
AI is creating a broader data-engineering role.
Traditional data engineers already work across databases, pipelines, transformation, cloud infrastructure, orchestration, and reliability. AI workloads add responsibilities around:
- Unstructured data
- Embeddings and vector retrieval
- Feature pipelines
- Model data
- RAG infrastructure
- Streaming
- Semantic context
- AI evaluation data
- Model and agent access
- AI-specific observability
At the same time, engineers are increasingly using AI to generate code, tests, documentation, and initial incident diagnoses.
The result is not necessarily a universal new job title called AI Data Engineer. Organizations will structure teams differently.
But the capability itself is becoming distinct.
As we explain in our deeper guide to AI data engineering, the engineer’s role increasingly shifts from building every pipeline manually toward architecture, platform reliability, governance, automation, and enabling AI workloads.
dbt’s analysis of the profession points toward three broad directions: deeper platform engineering, greater automation engineering, and stronger business/domain involvement. That matters for enterprise workforce planning.
Hiring solely for the ability to write pipeline code may become less useful than building teams that combine:
data engineering + platform thinking + AI context + domain knowledge + governance
The engineer of 2029 may write less boilerplate code than the engineer of 2026.
But they are likely to own a wider and more consequential system.
Trend 3: Real-Time Everything — The End of Batch Analytics
The heading is intentionally provocative.
Batch processing is not disappearing.
Payroll does not need to be recomputed every millisecond. Many regulatory reports, historical models, finance processes, and analytical workloads will continue to run perfectly well on scheduled pipelines.
What is changing is that organizations will increasingly have to justify why a workload is batch rather than simply accept batch as the default.
AI applications often operate inside much shorter decision windows:
- Fraud detection
- Dynamic pricing
- Recommendations
- Equipment monitoring
- Supply-chain interventions
- Real-time customer service
- Cybersecurity
- Agentic workflows
Confluent’s 2026 Data Streaming Report, based on 4,625 IT leaders, found that 72% reported insufficient real-time data-processing infrastructure as a challenge when scaling AI.
Closing that gap requires more than making nightly ETL run faster. Architectures may need:
- Change Data Capture
- Event-driven ingestion
- Kafka or Kinesis
- Flink or other stream-processing frameworks
- Incremental transformation
- Low-latency serving
- Materialized views
- Real-time quality monitoring
- Event-aware machine learning and agent workflows
The architectural principle is simpler than the tooling:
Data needs to arrive within the decision window of the system consuming it.
That may mean milliseconds.
It may mean five minutes.
It may still mean tomorrow morning.
The future of data engineering is not “real time everywhere.” It is latency designed deliberately around business and AI requirements.
Trend 4: The Semantic Layer Becomes the Standard
AI has exposed a problem that dashboards have struggled with for years:
What does the data actually mean?
Suppose a CEO asks an AI analytics assistant:
What was customer revenue last quarter?
The model may know SQL perfectly.
But does “revenue” mean booked revenue, recognized revenue, gross sales, net revenue, or revenue after returns?
Which customer definition should it use?
Which currency treatment?
Which business unit?
Those are not model problems.
They are semantic problems.
A semantic layer creates a governed representation of business meaning above raw databases: shared metrics, dimensions, entities, relationships, and definitions that applications can use consistently.
dbt describes the semantic layer as a way of defining business logic centrally so dashboards, applications, and AI systems can query the same governed definitions rather than recreating them independently.
Its importance increases substantially in an AI environment. A human analyst seeing a suspicious figure may recognize that something is wrong and investigate it.
An AI system can return an incorrect answer confidently and at enormous scale.
Current dbt guidance therefore frames AI-ready data as more than clean data: systems also need context about ownership, definitions, lineage, and business meaning.
Over the next three years, we expect this concept to expand beyond BI metrics.
Enterprise semantic infrastructure will increasingly include:
- Shared business vocabularies
- Semantic models
- Data contracts
- Entity definitions
- Ontologies
- Knowledge graphs
- Governed APIs
- Metadata
- Permissions
- Relationships understood by both humans and agents
For agentic AI particularly, the semantic layer may become the mechanism that prevents ten different agents from constructing ten different interpretations of the same customer, product, order, or business metric.
The future data platform therefore needs to answer two questions:
Can AI access the data?
and:
Does AI understand what that data means?
The second may prove harder.
Trend 5: Data Governance Becomes an AI Compliance Function
Historically, data governance was often associated with catalogs, ownership, access controls, retention policies, and data quality.
Those remain important.
But AI is moving governance closer to operational and regulatory risk.
For high-risk systems under the EU AI Act, Article 10 directly addresses training, validation, and testing datasets and their governance. It covers areas such as data origin, collection, preparation, assumptions, suitability, representativeness, gaps, and bias detection and mitigation. AI Act Service Desk
That means questions that once looked like internal data-management concerns can become part of AI governance:
- Where did the data originate?
- Who owns it?
- Was its intended use documented?
- Which transformations were applied?
- Is it representative?
- What biases were tested?
- Which model or agent used it?
- Who had access?
- Can the resulting output be traced?
Sector oversight reinforces the same direction.
The SEC’s 2026 examination priorities include scrutiny of firms’ representations about AI and whether policies and procedures adequately monitor or supervise AI use in areas such as fraud, AML, trading, and back-office operations.
Healthcare provides another example. FDA-recognized guidance and standards for AI/ML-enabled medical technology increasingly address lifecycle risks associated with data management, training, evaluation, cybersecurity, and monitoring.The direction is clear even though the specific obligations vary considerably by jurisdiction and use case:
AI governance increasingly reaches back into the data pipeline.
For engineering teams, this means governance cannot remain a review performed after a system is built.
Expect more policy enforcement to move into ingestion, transformation, access, deployment, and monitoring itself:
PII detection → lineage tagging → quality controls → permission enforcement → model/agent identity → logging → audit trail
The future of data governance is increasingly governance as code and governance in motion.
This is not legal or regulatory advice. Organizations should determine the requirements applicable to individual systems with their legal, risk, security, privacy, and compliance teams.
What These Trends Mean for Enterprise Data Teams
Taken together, the five trends point toward a fundamental change in the operating model.
Data engineering will become more automated but less forgiving.
AI can generate pipelines faster.
Agents may diagnose incidents faster.
Streaming can deliver information faster.
Natural-language interfaces can put analytics into many more hands.
But every increase in speed increases the consequences of weak semantics, ownership, governance, or quality.
dbt’s 2026 research captures that tension clearly: while AI-assisted development has become embedded, trust in data and data teams has also risen sharply as a strategic priority, and 71% of respondents remain concerned about incorrect or hallucinated information reaching stakeholders.
The future-ready data team will therefore need to optimize for reliable acceleration, not acceleration alone.
Engineers author pipelines
Engineers supervise increasingly automated pipeline creation
Incident alerts
Contextual diagnosis and agent-assisted remediation
Batch by default
Latency matched deliberately to the decision
Metrics scattered across tools
Governed semantic context shared by humans and AI
Governance documented separately
Governance embedded in pipelines and access
Human users consume data
Humans, applications, models, and agents consume data
Tool expertise
Architecture + domain + automation + governance expertise
Project handover
Continuous enablement and internal ownership
This also changes how organizations should think about talent. The goal should not be to make every existing engineer an expert in every emerging AI framework.
It should be to build a team with enough combined capability to architect, govern, automate, validate, and own the platform.
Build vs. Buy: How to Respond to the AI-Driven Data Engineering Shift
Most enterprises will not respond successfully by choosing entirely between internal capability and external providers.
They need to decide what must remain organizational knowledge and where specialist expertise can accelerate change.
Capabilities enterprises should usually own
Internal teams should retain authority over:
- Business meaning and semantic definitions
- Data ownership
- Critical domain knowledge
- Risk tolerance
- Decision rights
- Regulatory interpretation
- Access policy
- AI-use priorities
- Success measures
A consultancy should not become the only organization that knows what “customer,” “revenue,” or “eligible transaction” means.
Capabilities where specialist partners can accelerate delivery
External engineering can be valuable for:
- Platform modernization
- Streaming architecture
- Complex migration
- Semantic-layer implementation
- Agentic pipeline design
- Data observability
- Governance automation
- Cloud/data-platform engineering
- ML and AI pipeline integration
The strongest model is often build with enablement.
A specialist team accelerates difficult architecture and implementation work while deliberately transferring patterns, tooling knowledge, documentation, governance practices, and operational ownership into the client organization.
That is also the distinction between a one-time modernization project and building durable capability.
Organizations still determining what architecture and modernization work their AI workloads require can begin with AI data consulting before committing to a major platform change.
Prepare your data platform for what comes next.
Exadel helps modernize data foundations for AI, automation, and real-time operations.
How Exadel Helps Organizations Navigate the Future of Data Engineering
At Exadel, we do not see these trends as five separate technology purchases.
Agentic pipelines depend on observability and governance. Real-time systems need reliable architecture. AI interfaces need semantic context. Governance depends on lineage and ownership. All of them depend on engineering teams capable of operating the resulting environment.
Our Data Engineering & Analytics services cover data assessment, ingestion, transformation, cloud and lakehouse architecture, batch and streaming pipelines, data quality, governance, analytics infrastructure, and AI-ready foundations.
Exadel publicly brings certified engineering expertise across AWS, Azure, GCP, Databricks, Snowflake, and other data ecosystems, alongside architectures designed for machine learning and generative AI.
Our wider partnerships with Cursor, Anthropic, and OpenAI, together with certified practitioners, add expertise around the AI systems increasingly consuming those data platforms.
The value is not the partner logos themselves.
It is the ability to understand both ends of the architecture:
what emerging AI workloads require—and what enterprise data systems must provide to support them safely and reliably.
Technology selection remains requirement-driven. We expect the future data estate to remain heterogeneous, because workload, scale, latency, security, governance, cost, skills, and existing systems differ from organization to organization.
The future is therefore unlikely to be one universal “AI data stack.”
It is more likely to be a set of governed architectural patterns that can evolve as tools change.
That is also why our model combines consulting, engineering services, and enablement.
Our consultants remain connected to the engineers responsible for implementation. And our aim is not merely to leave behind a modernized platform, but to strengthen the client's ability to operate, govern, and extend it.
Our architectural view is similarly evolutionary rather than replacement-led.
Most enterprises already have ingestion tools, data platforms, catalogs, APIs, warehouses, and security infrastructure. Preparing them for agentic AI is often less about starting over than extending those foundations with stronger semantics, vector and retrieval capabilities, AI-aware identity, orchestration, observability, and automatic governance.
Our AI Engineering services provide the complementary production layer where those data foundations connect to model pipelines, MLOps, applications, and governed AI workflows.
The organizations best positioned for the next three years will not be those that predict every technology correctly.
They will be those that build a data foundation capable of changing safely when the prediction is wrong.
Frequently Asked Questions
Will AI Replace Data Engineers?
No. AI is likely to automate a growing number of data-engineering tasks, but it is simultaneously increasing demand for reliable data architecture, governance, semantics, platform engineering, and AI integration.
Engineers may spend less time writing repetitive transformations, documentation, and boilerplate pipeline code. Their work is likely to shift toward designing platforms, validating AI-generated artifacts, managing reliability, automating operations, and translating business meaning into systems that AI can use safely. The role is changing, but the underlying need for engineering judgment is increasing rather than disappearing.
What Is an Agentic Data Pipeline?
An agentic data pipeline is a data workflow in which an AI agent can reason about pipeline state, use engineering tools, inspect context, and perform or recommend multi-step actions.
For example, an agent might detect a failed job, inspect logs and lineage, identify a likely root cause, generate a proposed correction, run tests, and escalate the validated change for approval. The appropriate level of autonomy depends on the risk. Enterprise implementations need clear agent identity, permissions, testing, auditability, monitoring, and human escalation rather than unrestricted production access.
What Is a Semantic Layer in Data Engineering?
A semantic layer is a governed representation of business meaning that sits between raw data and the people or systems consuming it.
It centrally defines concepts such as revenue, customer, order, margin, or active user so that dashboards, applications, and AI systems use consistent definitions. Semantic layers are becoming more important as organizations introduce natural-language analytics and AI agents because models cannot infer an enterprise's internal business meaning reliably from raw tables alone.
Build for the next era of data engineering.
Exadel helps create reliable, adaptable data foundations for production AI.
