Data Mesh or Data Fabric? Do Better with Both

While certain proponents of these new approaches may be passionate about outlining what "counts" as employing data mesh or data fabric, it is our experience that in the instances where they are, in fact, applied, they are rarely employed in their "purest" form. Every organization has its own ingrained culture and constraints. The choice of how best to implement these approaches for a federal agency depends on factors such as data governance and metadata maturity, organizational buy-in and structure, and specific operational needs regarding data access, security, and agility. By keeping the outcomes an agency seeks with their data as the North Star of their efforts, we can help them design a hybrid approach that serves as a comprehensive solution to optimize data sharing while accommodating their respective organizational realities. Platforms that incorporate both data fabric and data mesh can, in many cases, be key to providing the foundation for flexibility and scale to meet organizational needs at the enterprise level.

By leveraging data mesh principles within a data fabric architecture, agencies can maintain centralized control and security standards while empowering domain-specific teams to manage their data autonomously. This hybrid approach allows agencies to capitalize on the agility and accountability promoted by data mesh, ensuring that data producers are closest to the data they understand best. Simultaneously, the centralized data fabric layer provides standardized integration, governance, and security protocols across the organization, ensuring consistency and compliance with regulatory requirements.

This combination creates flexibility in data management and sharing, accommodating diverse needs across different departments or domains while safeguarding data integrity and security. Moreover, it fosters a culture of collaboration and innovation by enabling cross-functional teams to share insights and collaborate effectively, thereby enhancing the agency's overall data-driven, decision-making capabilities. By implementing the best aspects of decentralization and centralization, federal agencies can take a balanced approach to data sharing that optimizes both data agility and governance, ultimately driving improved operational efficiency and mission success.

Imagine if a national health agency took such a hybrid approach. Data fabric could integrate data from clinical trials, genomic studies, and patient records to provide researchers with a comprehensive view of health-related data, while also exposing metadata to authorized users to contextualize the data and make it discoverable and accessible. By also incorporating data mesh principles, the agency could empower clinicians and researchers to manage and analyze specialized datasets independently, ensuring that insights into personalized treatment strategies are derived efficiently while maintaining data integrity and compliance with regulatory standards.

Or, consider how a federal financial agency could centralize financial data from various sources, ensuring a unified and consistent view of their data that enhances the accuracy of financial assessments and audits through an enterprise data platform and advanced analytics platform. Integrating data mesh principles within this fabric would empower mission or business-line teams that are responsible for specific financial domains to manage their datasets in a federated manner, while maintaining alignment to enterprise standards. This decentralized approach allows true domain experts to ensure both that data quality standards are met, and that AI models or analytic reports are making use of the best available data. For example, domain experts could assess payment and account data integrity and fidelity at the source, enhancing the agency’s ability to share high-quality data with relevant stakeholders across the organization, and ultimately advance financial fraud detection rates across the organization.

The rise of the semantic layer: As we see more and more U.S. agencies considering a blend of data approaches and leaning into such "what if" scenarios, we believe that the semantic layer will continue to emerge as the linchpin that harmonizes the decentralized principles of a data mesh with the integrated framework of a data fabric. A semantic layer sits between raw data sources and end users (typically business intelligence tools or applications).

By providing a unified abstraction of data semantics, it empowers organizations to decentralize data ownership while maintaining consistency and accessibility across diverse data sources. A semantic layer fosters agility by enabling self-service analytics and data governance, essential in navigating the complexities of modern data ecosystems. This convergence not only enhances operational efficiency but also accelerates innovation by democratizing data insights across the enterprise. Unfortunately, that doesn’t mean that standing up and maintaining a semantic layer is easy. Taking this route to standardization can, in fact, be quite challenging due to the need to accommodate diverse data sources, each with its own structure and format. As domains grow, the need for buy-in and standardization grows as well.

The need for organizational dedication: Regardless of the design decisions they make, when organizations fail to reach their goals for their data program, it’s frequently due to a lack of commitment to change across the enterprise. Re-drawing the lines of data ownership requires more dedication to robust governance and buy-in from business units than leaders often anticipate. The age-old questions of "who decides" and "who owns what data" will emerge as a thorny challenge. One reliable route out of the weeds is to bring on a partner to support and guide early steps like facilitating initial assessments and running buy-in sessions, as well as later steps like tooling selection and setting up specific roles and required controls. That said, it doesn’t need to be all or nothing—plotting out the future with a trusted partner often wins friends across the organization and sets a foundation for future progress and success.

The taming of technology trendsetters: Integrating data mesh and data fabric requires leveraging cutting-edge technologies and robust tools. Some of them may already be in use, while others may need to be added to the stack. Vital to both data mesh and fabric are advanced cataloging tools like Collibra and Alation, which streamline metadata management, enhance data discoverability across decentralized environments, and enable the semantic layer. Open table formats such as Delta and Iceberg have more robust, and more easily activated metadata, along with transactional consistency that accelerates virtualization and, therefore, data discoverability and accessibility. Equally critical are security and privacy tools such as Immuta that focus on access to reduce risk in data sharing. Privacy-enhancing technologies, such as Differential Privacy, that ensure data privacy while enabling seamless data integration and collaborative insights will become increasingly essential for navigating regulatory complexities and driving mission innovation in modern data ecosystems. And, data platforms that power modern data ecosystems, such as Databricks, Palantir, and Snowflake, are essential for ingesting, processing, transforming, sharing, and making data available across the organization.

Finally, relevant new technologies seemingly emerge daily. For example, technical metadata catalogs are being developed and released at a steady clip. They integrate essential data management features—such as data cataloging, data discoverability, metadata tagging, and attribute-based access controls—into a single product. This, combined with the rise in generative AI-enhanced data management, governance, and cataloging, further simplifies and enhances the mixed implementation of mesh and fabric.

Of course, there are tradeoffs when choosing the right tooling, and the technology landscape is rapidly advancing. At Booz Allen, we stay apprised of emerging technology through our tech scouting team, invest in promising new ideas through our corporate venture team, and continually examine tooling potential for clients with innovation teams localized by sector. This means that we also understand where a given client’s current tooling can be leveraged and where new technologies will be needed. Our clients find that having us as an honest broker to cut through the technology noise makes the difference.

Dan Tucker is the senior vice president and leader of cloud and data solutions in our civil business.

Katie-Coral Sicora leads strategy and capability incubation for emerging data solutions in our civil business.