What is Data Mesh for Enterprises and: How It Works & Why You Need It

Data is the lifeblood of decision-making — but how well does your organization utilize data?

The traditional centralized approach to data management and analytics is proving to be increasingly inadequate in today’s vastly digital business environment. Created in 2019, the data mesh architecture breathes new life into how organizations manage, share, and leverage data.

But what, exactly, is data mesh?

As businesses strive to improve agility, scalability, and improved data-driven decision-making, understanding the nuances of data mesh becomes imperative.

Join us today as we unravel the intricacies of the data mesh architecture, shedding light on its origin, benefits, and guiding principles. Together, we will discover why data mesh is emerging as the latest beacon of innovation and a key enabler of data democratization.

What is Data Mesh?

Data mesh is an architectural approach that aims to make data easier to find, access, and use.

A data mesh architecture works by decoupling data from traditional data silos and centralized data lakes. Instead, data is organized into small, decentralized data sets called “data assets” that are owned and managed by specific business domains or teams.

Zhamak Dehghani, a prominent figure in the technology and software development fields, first created the concept of the data mesh architecture in 2019. In Dehghani’s book Data Mesh: Delivering Data-Driven Value at Scale, Dehghani explains what is data mesh as a “decentralized sociotechnical approach to share, access, and manage analytical data in complex and large-scale environments within or across organizations.”

Analytical data is the main type of data utilized within the data mesh architecture. Due to the predictive and diagnostic capabilities of analytical data, this data forms a crucial foundation for improved visualizations, reports, and insights.

Additionally, analytical data is commonly used to train machine learning algorithms. According to Dehghani, analytical data fuels an organization’s artificial intelligence (AI) and analytics.

What are the Four Key Data Mesh Principles?

When creating the data mesh architecture, Dehghani designed the architecture around four main principles. These principles determine the architecture’s logical capabilities and operating model, striving to increase data value, sustain agility, and ensure scalability.

Four key data mesh principles and what they entail:

1. Principle of Domain Ownership

The Principle of Domain Ownership dictates that each domain or team is responsible for managing its own data assets and ensuring those assets are accurate, up-to-date, and accessible to other teams that need them. This data mesh principle aims to decentralize ownership of analytical data and enable independent management of domain-oriented data.

Moreover, the Principle of Domain Ownership strives to align business objectives, technology, and analytical data on both the architectural and organizational levels.

The goals and motivations of this data mesh principle include:

• Scalability: Data sharing should be able to scale alongside the organization as the number of data sources, data consumers, and diversity of data use cases increase.
• Continuous Change: Data must be optimized for continuous change by allowing change to be localized within specific business domains.
• Heightened Agility: Domain ownership enables greater business agility by eliminating centralized data bottlenecks.
• Increased Truth: Through domain ownership, businesses can bridge the gap more tightly between data origination and data use in analytical use cases.
• Enhanced Resilience: Domain ownership increases resiliency in analytical and machine learning models thanks to the elimination of complex intermediary data pipelines.

2. Principle of Data as a Product

The Principle of Data as a Product dictates that analytical data provided by the domains should be treated as a product, and the consumers of that data should be treated as customers. In turn, domain-oriented data can be shared directly as a product with different users.

Data as a product applies product thinking to domain-oriented data to truly delight the experience of the data users—data scientists, data analysts, data explorers, and anyone in between. Furthermore, data as a product underpins the use case for data mesh, unlocking the value of an organization’s data by dramatically increasing the potential for serendipitous and intentional use.

Dehghani identifies eight data usability characteristics that this data mesh principle adheres to:

To align with the Principle of Domain Ownership, the Principle of Data as a Product recognizes each data product as autonomous within its domain to be managed independently.

The goals and motivations of this data mesh principle include:

3. Principle of Self-Serve Data Platform

The Principle of the Self-Serve Data Platform outlines that data should be made available to other teams through self-service APIs and tools. These tools should allow easy discovery, queries, and integrations of data into domain-specific applications without requiring the source domain’s involvement, empowering cross-functional teams to share data more freely.

A self-serve data platform can manage the full lifecycle of independent data products, ensuring the mesh of interconnected data products is reliable and easy to access.

The goals and motivations of this data mesh principle include:

• Reduced Costs: The use of a self-serve data platform can reduce the total costs of decentralized data ownership by enabling simplified data sharing between domains.
• Simplified Lifecycle Management: A self-serve data platform reduces the complexity of data management, as well as the overall cognitive load for domain teams when managing the lifecycles of domain-specific data products.
• Reduced Specialization: Self-serve data platforms enable businesses to mobilize more in-house developers for data product development rather than relying on technology specialists. This can also contribute to cost savings in the long run.
• Automated Governance: Through self-serve data platform services, businesses can automate governance, security, and compliance standards for data products. As a result, the operational and financial burden of compliance can be reduced.

4. Principle of Federated Computational Governance

The Principle of Federated Computational Governance establishes that, rather than having a strict centralized authority governing all data across the organization, data mesh domains have autonomy and responsibility for the majority of policies within their sphere of influence.

Through this data mesh principle, Dehghani creates a data governance operating model that is based on a “federated decision-making and accountability structure.”

Each team should be composed of domain representatives and subject-matter experts to balance the autonomy of the domains with the interconnected data mesh. According to Dehghani:

The goals and motivations of this data mesh principle include:

The Fifth Data Mesh Principle: Building a Collaborative Data Culture

Though not one of Dehghani’s official four data mesh principles, the value of a collaborative data culture can be loosely defined as the fifth principle of the data mesh architecture.

The data mesh approach encourages collaboration and cross-functional communication between different domains. Moreover, this approach also places greater focus and emphasis on reusable data assets that can be shared across different domains within an organization.

Overall, data mesh makes it easier for organizations to access and share data more efficiently and effectively, while simultaneously promoting a more decentralized and collaborative approach for managing and governing analytical data.

What are the Benefits of Data Mesh for Enterprises?

As the world continues to evolve digitally, businesses need a modern approach to data that addresses the challenges of scalability and data complexity.

Dehghani defines an organization’s approach to data management as the “inflection point” that can determine whether that organization can achieve agility in response to change. From Dehghani’s perspective, data mesh represents a new evolution in data management that can help businesses take their data usage to new heights.

Examining the data mesh architecture from a business perspective reveals several key reasons why the adoption of data mesh is vital:

Ultimately, data mesh presents organizations with the opportunity to shift their data architecture and social structure to make better use of analytical data.

What are the Implementation Obstacles of Data Mesh?

While data mesh presents many advantages for businesses to consider — and, potentially, the key to achieving true data agility — the data mesh architecture still comes with implementation challenges.

To enable a data mesh implementation, many of the following capabilities are still needed:

• Analytical data storage in the form of a lake, warehouse, or lakehouse
• Data processing frameworks and computation engines to process data in batch and streaming modes
• Data querying languages, based on two modes of computational data flow programming or algebraic SQL-like statements
• Data catalog solutions to enable data governance, as well as discovery of all data across an organization
• Pipeline workflow management for orchestrating complex data pipeline tasks or machine learning model deployment workflows

Additionally, integrating diverse data technologies across different domains can be complex. Ensuring compatibility and seamless communication between decentralized data products may pose technical challenges, while incompatibility can result in data silos.

Overcoming the data mesh implementation challenges of data mesh requires a diverse approach that involves both cultural shifts within an organization that focus on increased collaboration and communication and a high level of technical expertise.

Implementing Data Mesh Architecture at Your Organization in 2024

If your organization is ready to take your analytical data capabilities to the next level, adopting the data mesh architecture is a necessity.

Implementing data mesh requires a mix of leveraging technical expertise and fostering a culture of data ownership and collaboration within your organization. By investing in the cross-functional capabilities of your teams, you can establish a robust ecosystem of interoperable data.

While Dehghani’s four data mesh principles lay the foundation for the data mesh architecture, it is up to your team to find the right technologies and technology providers to facilitate this modern approach to data at your organization.