Level 2 - Domain-Level (Solution) Architecture and Patterns

Return to home

This section shows how the Enterprise Data Platform Reference Architecture as well as standards are applied within individual domains to create solutions.

Why It Matters

Using common patterns and standards at the domain level keeps solutions consistent and compatible. This speeds up delivery, avoids rework, and ensures every solution contributes to and strengthens the overall enterprise “town plan.”

Example Platform and Pipeline Reference Architecture

Business architecture

Use cases and business requirements

Business requirements, use cases, and use case roadmaps together define what a solution must achieve, how it will be applied, and when capabilities will be delivered according to agreed priorities. These priorities may sit at the enterprise, domain level or both.

Understanding them is necessary to understand:

the outcomes the solution must enable or improve
the measures of success, constraints, and scope that shape design
how the solution will be used to meet specific business needs
the sequence and priority of delivering capabilities to maximise business value

Business processes

Business processes are the activities and tasks undertaken to achieve business goals. Understanding them allows the data architecture to uncover:

the context in which information is captured and used
key concepts and entities relevant to the domain or use case
relationships between processes and data
the basis for defining measures and metrics

Business glossary

A business glossary is a curated list of terms and definitions relevant to the business (at both the Enterprise and Domain levels). Understanding these terms and how they map across the organisation by stakeholders and systems is critical to consistent understanding and usage of concepts.

It is a core part of Enterprise Metadata Architecture.

Example glossary from Intuitas’ Glossary builder tool

Business metrics

Measures are raw, quantifiable values that capture facts about business activities (e.g., total sales, number of customers).

Metrics are calculated or derived indicators that assess the performance of business processes, often using one or more measures (e.g., sales growth rate, customer churn rate).

Both require capture, at a minimum, of:

name
definition
formula (with reference to concepts and terms as defined in the business glossary)
associated dimensions
source(s)
metric owner
frequency

Where metrics are implemented on the lakehouse, Unity Catalog metric views (GA, and open-sourced) provide a governed physical home for these definitions — see the semantic layer discussion in Level 1.

Data Architecture

Data Architecture defines how data is structured, stored, and accessed. It spans storage design, data models, integration, and governance to ensure trusted, reusable, and consistent data.

Data and information models

Conceptual Models – capture high-level business concepts and rules in plain language, creating a shared understanding between business and IT.
Logical Models – refine concepts into entities, attributes, and relationships, ensuring clarity and consistency across domains while remaining technology-agnostic.
Physical Models – implement the design in databases and systems, optimised for performance, scalability, and integration.

Domain-level models often align more closely to real-world business semantics and rules than the enterprise-wide model. While they may not map one-to-one with enterprise or other domain models, cross-mapping is essential to identify dependencies, ensure conformance (e.g., shared dimensions, master data), and support integration across the organisation.

Example of modelling a Domain-scoped Conceptual Information Model

See Bounded context

Domain glossary

The Domain Glossary complements the Domain Model by describing concepts in clear business language. It defines domain-specific terms, synonyms, and the context in which they are used, along with properties such as attributes, keys, measures, and metrics.

A well-curated Domain Glossary:

Ensures clarity, reduces ambiguity, and strengthens alignment between business understanding and technical implementation.
Builds on the Enterprise Glossary: Extend the enterprise glossary with domain-specific definitions.
References when aligned: Where a domain term is synonymous with an enterprise definition, the enterprise glossary should be referenced rather than duplicated.
Resolves when conflicting: Where definitions diverge or conflict, governance processes must be applied to reconcile differences and ensure consistency.

Example of authoring a Domain-scoped Glossary aligned to the CIM and Enterprise Glossary

Example of a Domain-scoped Glossary mapped to an Enterprise Glossary

Example of syncing the Glossary term to real-life Data and Products

Screenshot from an earlier product version; to be re-captured.

Domain data and warehouse models

Domain-level data and warehouse models reflect domain-specific scope, requirements and semantics as expressed in models and glossaries.
Conformed dimensions may serve as a bridge between domains for common entities.

Data zones and layers

Data and analytics pipelines flow through data zones and layers. Conventions vary across organisations, however the following is an effective approach:

Within each zone, data flows through a series of processing layers.
Top level zones follow the Medallion architecture.
- Raw (Bronze): Data according to source.
- EDW (Silver): Data according to business. (see Data and information models)
- Infomart (Gold): Data according to requirements.

Data zones and layers

The following table summarises the zones, layers and their naming patterns. Schema examples use the {domain}__{zone}__{env} catalog granularity; alternative granularities are described in Catalog naming and conventions.

Zone	Layer	Schema (illustrative)	Object patterns (illustrative)
Raw (Bronze)	Landing	`landing`	Source-named files and tables, partitioned by load date (`YYYY/MM/DD/HH`)
Raw (Bronze)	ODS	`ods`	Current-state source tables, e.g. `patient`, `encounter`
Raw (Bronze)	PDS	`pds`	Historised change tables carrying standard SCD columns (`effective_from_datetime`, `effective_to_datetime`, `updated_datetime`)
EDW (Silver)	Cleansed	`edw__cleansed`	`{source_system}__{object}` — typed, deduplicated, 1:1 with source
EDW (Silver)	Standardised	`edw__standardised`	`{source_system}__{entity}` — glossary-mapped, source-separated
EDW (Silver)	Integrated	`edw__integrated`	Unprefixed entities (`employee`), keysets `keys__{entity}`, reference `ref_{concept}`; workbench `stg__` micro-steps `00_keyed` … `07_desensitised`
EDW (Silver)	Data quality	`edw__dq`	Test results and shaped data quality reports
Infomart (Gold)	Workbench	`im__stg`	`stg__` shaping models (ephemeral by default, as in EDW)
Infomart (Gold)	Mart (product)	`im`	Requirement-specific `fact_`, `dim_`, `obt_` views over integrated entities (materialised where serving demands) and `mv_{subject}` metric views
Cross-cutting	Metadata	`meta`	Audit and event logs, engineering metadata (see Audit and Enterprise Metadata Architecture)

These map to naming standards and conventions for Catalog, Schemas and dbt.

The EDW (Silver) zone is organised into cleansed, standardised and integrated sub-layers — described in the EDW (Silver) zone section below.

Note on separators: double underscore (__) separates all structural name components; single underscore appears only within a token (e.g. ambo_sim). This discipline is applied consistently across the examples below — see Standards and Conventions for the rationale.

Raw (Bronze) zone: Data according to source

The Raw (Bronze) zone stores raw, immutable data as it is ingested from source systems. The choice of persistence level will depend on requirements.

(Persistent) Landing

Initial storage area for raw data from source systems.
Stores raw events as JSON or CDC/tabular change records.
Data is maintained in a primarily raw format, with the possibility of adding extra fields that might be useful later, such as for identifying duplicates. These fields could include the source file name and the load date.
- Partitioned by load date (YYYY/MM/DD/HH)
- Raw data preserved in original format
- Append-only immutable data
- Schema changes tracked but not enforced

Where a Lakeflow Connect managed connector or Zerobus Ingest writes directly to Delta tables (see Ingestion), the landing layer may be bypassed for those sources; the connector’s target tables then act as the ODS, with change history retained via table history and/or PDS.

ODS (Operational Data Store)

Current state of source system data with latest changes applied.
- Maintains latest version of each record
- Supports merge operations for change data capture (CDC) / De-duplicated
- Preserves source system relationships

PDS (Persistent Data Store)

Historical storage of all changes over time.
- Append-only for all changes
- De-duplicated
- Supports point-in-time analysis
- Configurable retention periods
- As these may be available in landing - may be realised through views over landing *subject to deduplication
- Historisation is implemented with dbt snapshots configured to emit the standard SCD columns (see SCD conventions)

EDW (Silver) zone: Data according to business

The EDW (Silver) zone transforms raw data into the enterprise’s business entities through three sub-layers, each a distinct stage of transformation (the examples through this section follow one scenario: the customer entity sourced from crm and billing in the corporate domain):

Cleansed (edw__cleansed) — 1:1 with source; typed, deduplicated, technically clean, at source grain.
- e.g: central_hub__prod.edw__cleansed.crm__customers — the CRM customer table, typed and deduplicated, columns still CRM-named
- dbt: models/edw/cleansed/crm/cln__crm__customers.sql
Standardised (edw__standardised) — still source-separated, with column names and code values mapped to the central glossary vocabulary: the physical home of semantic mapping.
- e.g: corporate__prod.edw__standardised.crm__customer and billing__customer — the same glossary columns (customer_name, customer_email), one object per source
- dbt: models/edw/standardised/crm/std__crm__customer.sql (and billing/std__billing__customer.sql)
Integrated (edw__integrated, unprefixed entities) — the definitive business entities: unified keys resolved through keysets (keys__{entity}), survivorship applied per glossary-defined rules, and business history materialised.
- e.g: corporate__prod.edw__integrated.customer — one row per customer, unioned from the two standardised objects via the keyset, with SCD2 history
- dbt: models/edw/integrated/customer/customer.sql

flowchart LR B["Raw (Bronze) landing · ods · pds"]:::raw C["edw__cleansed crm__deals"]:::hub S["edw__standardised crm__deal"]:::dom I["edw__integrated deal · sales_rep · keys__sales_rep"]:::dom G["Infomart (Gold) — optional dim_sales_rep · fact_sales (views)"]:::gold B --> C --> S --> I --> G classDef raw fill:#F7F6F2,stroke:#9A9A94 classDef hub fill:#E6F2F5,stroke:#4CAAC0 classDef dom fill:#FFFFFF,stroke:#2E7F93 classDef gold fill:#FFF4B8,stroke:#1A1A1A

Why Silver is staged this way

Bridging “what the system said” to “what the business means” involves genuinely different jobs; tangled into one hop, pipelines become unreadable and errors untraceable. Each job gets a stage:

First make it clean. Cast, deduplicate, flatten — change nothing else. A pristine 1:1 copy means downstream work re-runs from here instead of re-parsing raw files.
Then make it speak the business language — one source at a time. Mapping to the glossary vocabulary is a job for the people who know that source; keeping sources separate keeps every quality issue traceable to one system and makes integration a simple union. Skip this stage only where a single, well-designed source already speaks the glossary vocabulary.
Only then make it one truth. Sources meet for the first time in the integrated stage — the only place cross-system identity and survivorship can be solved, so the only place we solve them.
Pay for history once, where it is true. Raw keeps the immutable record of what arrived; business history is materialised once, in the integrated entities.
Serve, don’t copy. Gold products need not duplicate integrated entities — a dimensional view is a shape, not another dataset to keep in sync.

Models and metadata here reflect Enterprise as well as Domain-specific definitions (with Domains recognised as the owner of definitions within their boundary). Upstream sub-layers are source-centric; the integrated sub-layer is entity-centric.

The workbench

Each sub-layer carries a stg__ workbench (ephemeral by default; keysets and audit-boundary models are materialised). The canonical transformation micro-steps apply within the workbench:

00_keyed
01_renamed_and_typed
02_deduped
03_cleaned
04_filtered / 04_split
05_column_selected
06_business_validated
07_desensitised

Not every step is required for every source; steps that are used retain their number so that lineage remains comparable across pipelines.

Keysets

A keyset is a simple key mapping table: it resolves each system’s keys to the standardised key for the universal entity, so that sources conform to one identity. One per entity (keys__{entity}), always materialised in edw__integrated and effective-dated — if a mapping needs to change, the old row is closed and a new one opened; where matching is more than a shared natural key, record how the match was made so stewards can inspect and correct it as data rather than code. The pattern is the Data Vault Same-As Link adopted on its own; naming detail in Standards and Conventions.

e.g: corporate__prod.edw__integrated.keys__customer — maps crm C-1042 and billing 88213 to the same customer_key
dbt: models/edw/integrated/customer/keys__customer.sql

Worked dbt folder layouts for the sub-layers — workbench placement, keysets and the full example project tree — are in Standards and Conventions.

Data Quality

Data quality test results from checks applied to source data land in edw__dq. Further transformation of these results may be applied to shape them into data quality reports. e.g: the failing rows from an email-format test on edw__standardised.crm__customer, traceable to the glossary rule it enforces.

Reference Data

Reference data are integrated entities: common assets provided for broad consumption, aligned to standards and downstream needs. Historical versioning requirements of reference data may need to be considered. Reference objects carry the ref_ prefix with singular nouns in edw__integrated.

e.g: corporate__prod.edw__integrated.ref_country_code — the conformed country list that both crm and billing address codes map to during standardisation
dbt: models/edw/integrated/ref/country_code/ref_country_code.sql

History

Raw (Bronze) keeps the immutable arrival history; the integrated sub-layer is the only place business history is materialised. SCD2 is default-on for core entities; only attributes whose changes matter to reporting trigger new history rows — the rest are simply overwritten. Transactional events are immutable and never carry SCD tracking.

In some scenarios, it may be beneficial to retain standardised (source-separated) objects for consumption alongside the integrated entity — all conforming to a common logical model. This supports decoupled pipelining across multiple sources, improving modularity and maintainability.

Infomart (Gold) zone: Data according to requirements

The Infomart (Gold) zone focuses on delivering business-ready datasets, including aggregations and reporting structures that directly reflect specific business requirements.

In some instances, Infomart assets may be reused across multiple use cases or domains—blurring the line with EDW. While this is not inherently problematic, it is important to consider supportability and scalability to ensure these assets remain trustworthy, maintainable, and accessible over time.

Consider shifting logic left into the EDW (Silver) zone—such as common aggregations, reusable business rules, or conformed dimensions. This improves consistency, reduces duplication, and enables faster development of Infomart zone assets by building on stronger, more standardised foundations.

Mart Staging

The Infomart carries its own workbench (im__stg), following the same rule as the EDW workbenches: stg__ shaping models, ephemeral by default, materialised only where an inspectable object is required (07_desensitised remains an audit boundary; a shaped intermediate reused by several products may warrant materialisation).

Examples of business-specific transformations include:
- Pivoting
- Aggregation
- Joining
- Conformance
- Desensitisation
While dbt best practices use the term ‘Intermediates’ as reusable building blocks for marts, this is considered a form of staging and these are hence optional under this blueprint. https://docs.getdbt.com/best-practices/how-we-structure/3-intermediate

Product (Requirement Specific) Marts

These marts are designed for a defined requirement. e.g. sales fact aggregated by region.

These marts may be Kimball or denormalised flat tables depending on requirements; although Kimball dimensional models are encouraged. Objects always carry their object-type prefix (fact_, dim_, obt_).

A solution served to the consumption layer is drawn entirely from the Infomart interface — consumers are not granted directly on edw__integrated. Where a product needs a conformed entity as-is, it is surfaced as a thin 1:1 consumption view over the integrated entity; the view is nearly free, carries the consumption-facing name and keeps the contract stable while the integrated layer refactors freely. e.g:

clinical__prod.im.dim_customer (1:1 view over edw__integrated.customer)
clinical__prod.im.fact_sales_aggregated_by_region (product-shaped)

Direct reads of edw__integrated are reserved for intra-domain engineering and advanced analytical users.

Lakehouse Catalog to Storage Mapping

Unity Catalog objects (catalogs, schemas, objects) are mapped to storage as follows:

Managed tables are the default. Managed storage is assigned at the catalog level (the continuing Unity Catalog best-practice recommendation), giving each domain catalog its own storage root, e.g. .../{domain}__{env}_catalog.
Environments map to storage containers using the canonical environment tokens (dev, test, pat, prod; non-promoted: sandbox, poc).
Where external or managed-location choices are made below catalog level, the explicit zone/layer folder mapping applies, with folder names matching schema names — e.g. edw/cleansed, edw/standardised, edw/integrated, infomart/stg, infomart/mart. This is an option, not the default; use it only where isolation, migration or lifecycle requirements demand it.

Illustrative example of Catalog to storage mapping in the Intuitas Demo Environment:

flowchart TB subgraph MS[" Unity Catalog metastore — intuitas (one per region) "] direction TB subgraph CE[" central_hub domain "] CE1["central_hub__dev · central_hub__test"] CE3["central_hub__prod schemas: landing · ods · edw__cleansed · meta"] end subgraph CC[" clinical domain "] CC1["clinical__dev · clinical__test"] CC3["clinical__prod schemas: stg · ref · mart (emergency_department objects)"] end CO["corporate__dev · corporate__test · corporate__prod"] end subgraph NET[" vn_lakehouse__hub "] SA1["dlintuitascentralhub containers: dev · test · pat · prod managed root: central_hub__prod_catalog"] SA2["dlintuitasclinical containers: dev · test · pat · prod optional external folders: edw/cleansed · edw/standardised · edw/integrated"] SA3["dlintuitascorporate containers: dev · test · pat · prod"] end CE -- "managed storage (default)" --> SA1 CC -- "managed storage / external locations (option)" --> SA2 CO -- "managed storage (default)" --> SA3 style MS fill:#FFFFFF,stroke:#1A1A1A style NET fill:#F7F6F2,stroke:#9A9A94 style CE fill:#E6F2F5,stroke:#4CAAC0 style CC fill:#E6F2F5,stroke:#4CAAC0

External locations and storage credentials follow the naming standards exloc-{storage_account}-{container} and stcred-{storage_account} respectively — see Standards and Conventions.

Data Engineering

Ingestion

Note: These patterns and findings reflect GA functionality as at July 2026 unless labelled otherwise. Refer to respective product roadmaps and documentation for the latest guidance on functionality.

Ingestion is the process of acquiring data from external sources and landing it in the platform Raw zone.

It should be:

Scalable, Resilient, Maintainable, Governed
Pattern-based, automated and Metadata-driven where possible
Batch and stream-based

Following the general availability of Lakeflow (June 2025), managed connectors are the first-choice pattern for supported sources, and ADF is retained only as a hybrid-connectivity exception.

Example batch ingestion options:

flowchart LR SRC["Source systems SaaS apps · databases · event producers · files"]:::src LFC["Pattern 1 — Lakeflow Connect managed connector (GA)"]:::rec O1["ODS — managed Delta"]:::ods ZB["Pattern 2 — Zerobus Ingest direct serverless write (GA)"]:::rec O2["ODS / streaming tables"]:::ods KFK["Kafka / event bus (where one already exists)"]:::rec LN1["Landing"]:::rec AL1["Pattern 3 — Autoloader"]:::rec O3["ODS — managed Delta"]:::ods LN2["Landing — file drops"]:::rec AL2["Pattern 4 — Autoloader + merge"]:::rec O4["ODS — managed Delta"]:::ods ADF["Pattern 5 — ADF land-and-orchestrate hybrid-connectivity exception (self-hosted IR)"]:::alt PY["Pattern 6 — custom Python polars / delta-rs (niche)"]:::alt X1["Bronze — external Delta"]:::alt FT["Fivetran — uncovered sources only"]:::alt X2["Bronze — managed"]:::alt SRC --> LFC --> O1 SRC --> ZB --> O2 SRC --> KFK --> LN1 --> AL1 --> O3 SRC --> LN2 --> AL2 --> O4 SRC --> ADF --> LN2 SRC --> PY --> X1 SRC --> FT --> X2 classDef src fill:#E6F2F5,stroke:#4CAAC0,stroke-width:1.5px classDef rec fill:#FFFFFF,stroke:#2E7F93,stroke-width:1.5px classDef ods fill:#E6F2F5,stroke:#2E7F93,stroke-width:1.5px classDef alt fill:#F7F6F2,stroke:#9A9A94,color:#6B6B66,stroke-dasharray:4 3

Ingestion patterns and notes:

Pattern 1: managed connector: source -> Lakeflow Connect -> ods (first choice for supported sources)
- Lakeflow Connect provides fully managed, serverless, Unity Catalog-governed connectors. GA connectors as at July 2026 include SQL Server (with built-in CDC and Change Tracking), ServiceNow, Google Analytics 4, Salesforce, Workday, SharePoint, PostgreSQL and Oracle NetSuite, with further connectors (SFTP, MySQL, DB2, MongoDB, DynamoDB) in development.
- A free ingestion tier applies to managed connectors (100 free DBUs per day, up to approximately 100 million records per day), which materially changes the cost case against third-party ingestion tools.
- The connector’s target tables act as the ODS; the landing layer may be bypassed for these sources (see Data zones and layers).
Pattern 2: streaming: producer -> Zerobus Ingest -> ods / streaming tables (streaming without a bus)
- Zerobus Ingest (GA; gRPC API GA, REST API Beta) writes events directly into Delta tables at up to 100 MB/s per connection with sub-5-second latency — removing the need for an intermediate Kafka bus where the organisation controls the producers. Existing Kafka producers can be repointed without code changes.
Pattern 3: streaming: kafka -> landing -> databricks autoloader -> ods (where an event bus already exists)
- see repo Bronze Landing to ODS Project
- Retained for estates where Kafka (or an equivalent broker) is already established as the enterprise event backbone, or where multiple consumers subscribe to the same streams.
Pattern 4: batch: file drops / unsupported sources -> landing -> databricks autoloader merge to ods
- see repo Bronze landing SQL Server to ODS Project
- The default pattern for file-based interfaces and any source not yet covered by a managed connector.
Pattern 5: batch: source -> adf -> landing -> databricks autoloader merge to ods (hybrid-connectivity exception only)
- see repo External Database to ODS Project
- Use ADF only where a self-hosted integration runtime is required to reach on-premises or network-isolated sources that Lakeflow Connect gateways cannot reach, or where substantial existing ADF investment must be preserved.
Pattern 6: batch/streaming: source -> custom python (polars / delta-rs) -> deltalake -> external table (niche)
- Reserved for lightweight or unusual sources where the engineering overhead of a managed service is not justified.
Fivetran: only for sources not covered by Lakeflow Connect
- see repo fivetran
- SharePoint ingestion is served by the native SharePoint connector; retain Fivetran only where its connector catalogue exceeds Lakeflow Connect for a given source.

Rejected patterns:

batch: adf -> deltalake -> ods (does not support unity catalog, requires target tables to be pre-initialised)
batch: adf -> databricks sql endpoint -> ods (no linked service for databricks)
batch: adf + databricks notebook -> landing, ods, pds (more undesirable coupling of adf and databricks and associated risks)

With ADF scoped to a hybrid-connectivity exception, these rejected patterns are retained here for teams maintaining existing ADF-based estates.

Transformation

Transformation steps are organised into sub-layer workbenches — see the EDW (Silver) zone.

Batch and Micro-batch SQL transformation

dbt is our lead SQL transformation framework. Its testing, documentation, lineage and packaging ecosystem, together with the staging micro-step conventions defined in this blueprint, make it the default for batch and micro-batch transformation across the EDW and Infomart zones. See dbt standards.

Streaming SQL transformation

Lakeflow Declarative Pipelines (formerly Delta Live Tables) is a first-class transformation option alongside dbt. It is GA as part of the Lakeflow suite and has been contributed to Apache Spark as Spark Declarative Pipelines. Key capabilities:

Streaming tables — incremental ingestion and transformation over streaming sources with exactly-once processing.
Materialised views — declaratively maintained aggregates and joins, recomputed incrementally where possible.
Expectations — in-pipeline data quality constraints with configurable warn/drop/fail actions (see Observability).

Choosing between dbt and Lakeflow Declarative Pipelines

Consideration	Favours dbt	Favours Lakeflow Declarative Pipelines
Latency	Batch and micro-batch	Streaming and continuous
Team skills	SQL/analytics engineering	SQL or Python data engineering
Testing and documentation	Mature test/doc/lineage ecosystem, Elementary observability	Expectations for in-pipeline gates
Portability	Warehouse-agnostic projects	Databricks/Spark-native
Orchestration	dbt task in Lakeflow Jobs	Pipeline task in Lakeflow Jobs; managed by the pipeline itself

A pragmatic split we commonly apply: Lakeflow Declarative Pipelines for the Raw (Bronze) zone and streaming paths (ingestion, ODS/PDS maintenance, streaming tables); dbt for the EDW (Silver) and Infomart (Gold) zones where modelling rigour, testing and documented lineage dominate. Both are orchestrated by Lakeflow Jobs (see Orchestration), so a mixed estate remains operable from a single control plane.

Non SQL transformation

Under development. (Contact us to know more).

Data can be shared and delivered to consumers through various channels, each differing in:

Cost
Functionality
Scalability
Security
Maintainability

The following subsections offer more details about the channels depicted below.

Sharing and delivery visualisation channels

flowchart LR GOLD["Lakehouse — Unity Catalog governed EDW / Infomart assets: tables · views · MVs · streaming tables · metric views · volumes"]:::lh subgraph NATIVE[" Databricks-native surfaces "] DASH["AI/BI Dashboards (external embedding GA)"]:::rec GEN["Genie / Genie One"]:::rec APPS["Databricks Apps"]:::rec end subgraph FAB[" Microsoft Fabric / Power BI "] MIR["Mirrored UC catalog + Direct Lake on OneLake"]:::rec IMP["Import via Publish-to-Power-BI"]:::rec DQY["DirectQuery with OAuth SSO (per-user UC RLS/masks)"]:::rec end subgraph EXTC[" External / cross-organisation "] OS["OpenSharing recipients (OIDC federation)"]:::rec CR["Clean Rooms"]:::alt ICE["Iceberg REST Catalog engines DuckDB · Snowflake · Trino"]:::rec end subgraph OPS[" Operational / programmatic "] LB["Lakebase synced tables (reverse ETL)"]:::rec SQLW["SQL warehouse / JDBC-ODBC"]:::rec API["SQL Statement Execution API"]:::alt end GOLD --> DASH GOLD --> GEN GOLD --> APPS GOLD --> MIR GOLD --> IMP GOLD --> DQY GOLD --> OS GOLD --> CR GOLD --> ICE GOLD --> LB GOLD --> SQLW GOLD --> API classDef lh fill:#E6F2F5,stroke:#2E7F93,stroke-width:1.5px classDef rec fill:#FFFFFF,stroke:#2E7F93,stroke-width:1.5px classDef alt fill:#F7F6F2,stroke:#9A9A94,color:#6B6B66 style NATIVE fill:#F7F6F2,stroke:#9A9A94 style FAB fill:#F7F6F2,stroke:#9A9A94 style EXTC fill:#F7F6F2,stroke:#9A9A94 style OPS fill:#F7F6F2,stroke:#9A9A94

Pull / direct access

Databricks OpenSharing practices

OpenSharing (formerly Delta Sharing) allows read-only access directly to governed data in the lakehouse without the recipient needing a Databricks cluster or SQL endpoint. Two protocols are available: Databricks-to-Databricks and Databricks-to-Open.

Shareable assets: Databricks-to-Databricks shares can include tables, change feeds, views, dynamic views, materialised views, streaming tables, volumes, AI models, notebooks and managed Iceberg tables. Foreign-table sharing is in Beta (Preview).
Permissions: OpenSharing is a feature of Databricks Unity Catalog that requires enablement and authorised user/group permissions for the feature as well as the shared object.
Recipient authentication: for open (non-Databricks) recipients, we recommend OIDC token federation — short-lived OAuth tokens exchanged against the recipient’s identity provider (Entra ID, Okta) — over long-lived bearer tokens.
Costs: compute cost is borne by the consumer only for direct table shares. Where recipients lack direct access — open connectors, or cross-account classic compute — shares of views, materialised views and streaming tables are materialised by the provider’s serverless compute, and the provider pays; usage is trackable via the materialisation history system table. Other costs include storage API calls and data transfer (egress options include Cloudflare R2 and Databricks-billed SecureConnect).
Naming standards and conventions see naming standards
Tightly scope the share as per the principle of least privilege:
- Share only the necessary data
- Single purpose, single recipient
- Granular access control
- Prefer OIDC federation; where bearer tokens are unavoidable, set an expiry

Use audit logging to track access and usage:

SELECT *
FROM system.access.audit
WHERE
action_name LIKE 'deltaSharing%'
ORDER BY event_time DESC
LIMIT 100;

Limitations:
- Tables carrying row filters or column masks cannot be shared — recipient-aware filtering requires dynamic views using CURRENT_RECIPIENT(). Note: tables protected by ABAC policies can be shared if the share owner is exempt from the policy — the policy does not travel with the share (see Privacy Preservation).
Reference: OpenSharing documentation

Clean Rooms. For privacy-preserving multi-party collaboration — where counterparties compute over combined data without either side exposing raw records — Databricks Clean Rooms (GA, built on OpenSharing) belongs in the sharing decision tree alongside conventional shares.

ADLSGen2 Access to Data

Provide direct ADLSGen2 access via Managed Identity, SAS or Account Key
Note: While technically possible, ADLSGen2 access is not generally recommended for end user consumption as it bypasses the Unity Catalog and associated governance and observability controls.
Example Scenarios: Direct ADLS file sharing might be preferable in certain cases, even when OpenSharing is available:
- Unstructured data
- Large non-delta file transfer
- Consumers that don’t support OpenSharing

Open engine access via the Iceberg REST Catalog (DuckDB and others)

External engines — including DuckDB, Trino and Snowflake — can reach Unity Catalog tables directly via UC’s Iceberg REST Catalog APIs, reading managed Iceberg tables and Delta tables exposed through UniForm.
Costs: open-engine data access incurs the costs of the underlying compute, storage access and data transfer, similar to OpenSharing.
Example Opportunities/Uses:
- Last mile analysis
- SQL interface to Delta, Iceberg, Parquet, CSV, etc.
- dbt compatibility
- Local execution and storage of queries and data
- Use to feed visualisation tools, e.g., Apache Superset
See repo DuckDB

SQL Access

SQL Access is provided by the Databricks SQL (serverless) endpoint.

API Access

The Databricks SQL Statement Execution API can be used to execute SQL statements on a SQL warehouse and fetch the result.

References:

Snowflake Access

Snowflake access is provided by Databricks OpenSharing.
Snowflake can also read Unity Catalog managed Iceberg tables via the Iceberg REST Catalog, or Delta Lake external tables over ADLSGen2 see external tables

Push

Operational serving / reverse ETL: Lakebase synced tables. Lakebase (managed Postgres, GA on Azure) provides synced tables that continuously serve lakehouse tables into Postgres for low-latency operational application access — our recommended pattern for reverse ETL and operational data products, replacing hand-built export pipelines. Pair with Databricks Apps where the consuming application can live inside the governed perimeter.

Other push channels (custom jobs pushing to APIs or queues) remain available where the consumer cannot pull; implement them as Lakeflow Jobs so that delivery is observable and attributable.

Databricks-native consumption surfaces

Before evaluating any Fabric/Power BI pattern, apply the decision gate: does this audience need Power BI at all? A substantial class of consumers — operational dashboard users, natural-language explorers, internal tool users — can be served natively within the governed Databricks perimeter, avoiding Fabric capacity cost, data egress and dual-governance overhead entirely.

AI/BI Dashboards (GA) — governed BI on serverless SQL, consuming Unity Catalog tables and metric views directly, with UC permissions enforced natively. Dashboard embedding for external (non-Databricks) users went GA in March 2026, extending reach beyond workspace users.
Genie (GA) — natural-language Q&A over governed data — and Genie One (formerly Databricks One), the simplified business-user interface combining dashboards, Genie chat and Databricks Apps under a consumer-access entitlement. Curate Genie Agents (formerly Genie Spaces) over certified metric views rather than raw tables.
Databricks Apps (GA) — serverless hosting of Python/JavaScript applications (Dash, Streamlit, Gradio, Flask, React) inside the workspace, for bespoke data products and internal tools.

Where Power BI is required — finance packs, paginated reporting, deep Office-ecosystem integration, established Power BI estates — proceed to the patterns below.

Microsoft Fabric and Power BI access

The following describes options for providing access to Microsoft Fabric / Power BI, in order of consideration.

Security boundary — read this first

There are two fundamentally different enforcement paths across the Databricks-to-Fabric boundary:

Governed enforcement path — DirectQuery (or Import refresh) via the Databricks connector with OAuth SSO, and Publish-to-Power-BI from Unity Catalog: UC row filters and column masks are enforced per end user.
Re-secure-in-Fabric path — mirroring, shortcuts and Direct Lake all read under a fixed credential: per-user UC enforcement does not carry through, and security must be re-applied Fabric-side (semantic-model RLS on a fixed-identity connection, or OneLake security — noting OneLake security remains Preview).

Documented trap: Direct Lake on OneLake ignores SQL-endpoint RLS entirely — queries succeed and SQL-based RLS is simply not applied. Only Direct Lake on SQL honours SQL-endpoint RLS, via DirectQuery fallback.

Option 1. Mirrored Azure Databricks Unity Catalog + Direct Lake on OneLake (default zero-copy path)

Steps:

Within a Fabric Workspace, create a new item Mirrored Azure Databricks Catalog (documentation)
Build a Direct Lake on OneLake semantic model over the mirrored item

Evaluation:

Pros:
- GA since July 2025; zero-copy and metadata-only — catalog structure is mirrored, data is accessed through shortcuts with no replication or mirrored-storage cost
- Private-endpoint workspaces supported (GA) via the VNet data gateway
- Automatic metadata sync; Direct Lake on OneLake models are multi-source and support composite Direct Lake + Import tables (GA; calculated columns remain Preview)
Cons:
- Excluded table types: tables with UC RLS/column-mask policies, views, materialised views, streaming tables and federated tables are not mirrored — this pushes teams to materialise gold-layer assets as plain Delta tables
- Per-user UC enforcement does not carry through (fixed-credential shortcuts) — re-secure Fabric-side per the security boundary note above
- The underlying ADLS Gen2 account must also be accessible to Fabric
- Requires Fabric capacity (F SKU)

Option 2. Power BI Import via SQL endpoint — automated with Publish to Power BI from Unity Catalog

Steps:

Use Publish to Power BI Service from Unity Catalog (GA since October 2024) to publish schemas — including PK/FK relationships — straight to a Power BI workspace as a semantic model
Automate publish and refresh with the Power BI task in Lakeflow Jobs

Evaluation:

Pros:
- The most feature-complete and predictable pattern; burst DBU cost at refresh time only, schedulable with serverless auto-stop
- Works with Pro licences alone — no Fabric capacity required
- OAuth SSO honours UC permissions at publish and refresh
Cons:
- Data duplicated into the semantic model; freshness bounded by refresh cadence
- Per-user UC RLS/masking is not enforced at query time (the refresh identity reads the data) — model-level RLS required where needed

Option 3. Power BI DirectQuery via SQL endpoint with OAuth SSO

Steps:

Databricks documentation

Evaluation:

Pros:
- The only pattern in which Unity Catalog row filters and column masks are enforced per end user (via the native connector’s OAuth SSO), without re-implementing security in Power BI
- No duplication; truly live data
- Photon accelerates the aggregation-heavy queries Power BI generates
Cons:
- Compute cost on Databricks for every report interaction; interactive workloads keep warehouses warm (mitigate with serverless auto-stop, statement timeouts and query budgets)
- Less feature rich than import mode

DirectQuery is not deprecated — Microsoft continues to position it for freshness and scale scenarios, and Databricks actively invests in it. The standard cost mitigation is a composite model: Import (or Direct Lake) dimensions and aggregations with DirectQuery fact tables.

Steps:

Create an OpenSharing share
Use the Delta Sharing connector to import from within Power BI

Evaluation:

Pros:
- No duplication at source; centralised control over access policies; works across tenants, clouds and organisations
Cons:
- The connector remains Import-only with an in-memory row limit (1 million rows by default, configurable)
- Row-level security and masking via dynamic views only
- Within a single Azure tenant, Option 1 (mirroring) is superior — reserve this pattern for external counterparties

Option 5. Raw ADLS Gen2 shortcut (niche — non-UC data)

Steps:

Create a OneLake shortcut to ADLS Gen2 using a workspace identity or service principal; trusted workspace access (GA) reaches firewalled storage accounts
Build Direct Lake or SQL-endpoint consumption over the shortcut

Evaluation:

Pros:
- Zero-copy for data that is not in Unity Catalog, or in cross-tenant storage
Cons:
- Delegated, storage-level authorisation — Unity Catalog is bypassed entirely; use Option 1 wherever the data is UC-governed
- Fabric-side re-securing required (OneLake security RLS/CLS is Preview)

Option 6. Replicate into Fabric (not recommended)

Superseded for read serving by Option 1 (mirroring removes its rationale) and, for single-copy architectures, by Unity Catalog managed tables stored directly in OneLake combined with OneLake catalog federation (Databricks mounting Fabric items as foreign catalogs, GA).
Retain only for edge cases: Fabric-side write workloads over the data, or regulatory separation requirements.

Recommended decision tree

flowchart TD START["Who consumes, and where?"] --> Q0{"Audience can be served within Databricks-native surfaces?"} Q0 -- "Yes" --> NAT["AI/BI Dashboards + Genie / Genie One No Fabric capacity · single governance plane · native UC RLS"] Q0 -- "No — Power BI / Fabric required" --> QX{"External organisation / different tenant or cloud?"} QX -- "Yes" --> OSH["OpenSharing → Power BI Import (Import-only connector, row-limit caveat)"] QX -- "No" --> Q1{"Must UC RLS / column masks be enforced per end user?"} Q1 -- "Yes" --> DQY["DirectQuery (or composite model) via SQL warehouse with OAuth SSO Photon + serverless auto-stop"] Q1 -- "No" --> Q2{"Fabric capacity owned or planned?"} Q2 -- "No — Pro licences only" --> IMP["Import via SQL endpoint automated with Publish-to-Power-BI + Power BI job task"] Q2 -- "Yes" --> MIR["Mirrored Azure Databricks UC catalog + Direct Lake on OneLake re-secure Fabric-side"] MIR --> Q3{"Tables excluded from mirroring? (views · MVs · streaming · RLS-policied)"} Q3 -- "Yes" --> MAT["Materialise as plain Delta gold tables, or fall back to Import / DirectQuery for those tables"] MIR --> Q4{"Data not in UC / cross-tenant storage?"} Q4 -- "Yes" --> SHC["Raw ADLS Gen2 shortcut with workspace identity + trusted workspace access"] Q2 -- "Fabric-side WRITE workloads" --> ONE["UC managed tables stored in OneLake + OneLake catalog federation (before building a replication pipeline)"] classDef leaf fill:#E6F2F5,stroke:#2E7F93,stroke-width:1.5px class NAT,OSH,DQY,IMP,MIR,MAT,SHC,ONE leaf

Cost model per pattern

Pattern	Databricks cost driver	Fabric CU driver	Licensing prerequisite
1. Mirrored UC catalog + Direct Lake on OneLake	None at rest (metadata-only mirroring)	SQL endpoint queries, Direct Lake framing/transcoding, OneLake transactions	Fabric F SKU (capacity-only — no Pro-only path)
2. Import via Publish-to-Power-BI	Serverless SQL warehouse burst at refresh	None if hosted in a Pro workspace; refresh CU if on capacity	Power BI Pro
3. DirectQuery with OAuth SSO	DBUs per report interaction; warm-warehouse pressure	Minimal	Pro or F SKU
4. OpenSharing import (cross-org)	None for direct table shares; provider-side serverless materialisation cost for view/MV/streaming-table shares to open recipients	Refresh CU	Pro
5. Raw ADLS Gen2 shortcut	Storage transactions only (UC bypassed)	As pattern 1	F SKU
6. UC managed tables in OneLake	Write-path compute	Full Fabric workload CU	F SKU

Licensing threshold to note: free viewers of Power BI content require an F64 or larger capacity; below F64, every consumer needs a Pro licence (Fabric pricing). If no Fabric capacity exists and audiences are modest, Option 2 (Pro licences only) or staying Databricks-native is cheaper than buying an F SKU.

Visualisation

Options in order of consideration:

Databricks AI/BI Dashboards — first choice for governed consumers inside the perimeter, including externally embedded audiences (see Databricks-native consumption surfaces).
Power BI — where the audience, licensing estate or reporting requirements demand it (see Microsoft Fabric and Power BI access).
Databricks Apps — bespoke visual data products (Dash, Streamlit, Gradio) hosted in the workspace.
Open-source visual options (e.g. Apache Superset) — served via the SQL warehouse or the Iceberg REST Catalog.

AI/ML

Under development. (Contact us to know more).

Areas for consideration include:

MLOps
Training
Databricks (including Agent Bricks and agent governance — a watch item for 2.x)
Azure ML

Data governance

This section describes how Enterprise-level governance will be implemented through solutions at the domain level.

Data lifecycle and asset management

Under development. (Contact us to know more).

Areas for consideration include:

data contracts and policy
data asset tagging — anchored on governed tags (GA): account-level tag definitions with enforced allowed values and assignment permissions, shared across catalogs, bundles and Azure resources (see the tag registry in Standards and Conventions)

Data access management

Under development. (Contact us to know more).

RBAC — privilege grants to Entra groups per the group naming standard at catalog, schema and object level — remains the foundation: it decides who can reach an object at all. ABAC complements it, targeting what a permitted user may see within an object: row filtering, column masking and PII controls at scale, driven by governed tags rather than per-object logic. See ABAC, Governed Tags and Data Classification in Level 1 for the full write-up, including its known limitations.

Areas for consideration include:

data access request management
data contracts
access audit
activity audit

Data quality

Three complementary tiers apply; they observe different scopes and are not competing:

Tier	Scope	Status
Elementary	The dbt DAG — freshness, volume anomalies, schema changes, test results	Actively maintained
Unity Catalog Data Quality Monitoring (formerly Lakehouse Monitoring)	The whole catalog, including non-dbt tables — profiling, freshness and completeness without per-model instrumentation	Data profiling GA; anomaly detection Public Preview
Lakeflow Declarative Pipelines expectations	In-pipeline enforcement gates (warn/drop/fail) at the point of transformation	GA

Guidance:

Use Elementary where dbt is the transformation framework (see Observability); use Data Quality Monitoring for catalog-wide coverage including tables dbt never touches; use expectations to stop bad data at the pipeline boundary rather than reporting on it afterwards.
Data quality test results land in the edw__dq layer of the EDW (Silver) zone for shaping into data quality reports (see Data zones and layers).
Data standards and quality business rule management remain governance-owned; implement rules as dbt tests or expectations depending on the executing framework.
Quality should trace to the standards it enforces. Data standards and rules are defined in the data modelling repository and/or glossary — the governance metadata is the master — and the tests that implement them should reference those definitions, so a failing test points back to the standard it protects. Tests implement standards; they do not define them.

Data quality in governance platforms

Governance platforms such as Microsoft Purview (Data Quality), Collibra and Atlan increasingly bundle their own data quality capabilities — profiling, rule authoring, scorecards. Our position: we do not use these yet. The capability is already covered by the tiers above, where rules live in code beside the transformations they test, version-controlled and deployed with them. Revisit if consolidating quality into the governance platform would genuinely retire one of the tiers rather than add a fourth.

Data understandability

Under development. (Contact us to know more).

Areas for consideration include:

data lineage
data object metadata

Privacy Preservation

Privacy preservation is built on policy-based controls that scale, rather than per-table mechanisms. ABAC row-filtering and column-masking policies, governed tags and data classification are all GA (May 2026). Dynamic views are retained — but scoped to where they remain necessary: share-scoped filtering.

The recommended mechanism stack:

Governed tags (GA) — define the classification vocabulary (e.g. data_class: pii, data_class: sensitive) once at account level, with enforced allowed values and controlled assignment permissions. Tag keys follow snake_case per the tag registry.
Data classification (GA) — agentic classification automatically detects and tags PII, feeding the governed-tag vocabulary rather than relying on manual tagging sweeps.
ABAC policies (GA May 2026) — row-filter and column-mask policies attached at catalog, schema or table level, evaluated dynamically against governed tags. One policy protects every current and future column carrying the tag — this is the scaling mechanism that replaces hand-applied per-table row filters and masks. Masking functions follow the mask_{data_class} naming standard.
Dynamic views — retained for share-scoped filtering. Tables carrying row filters or column masks cannot be shared via OpenSharing; recipient-aware filtering on shares still requires dynamic views using CURRENT_RECIPIENT(). Note the ABAC nuance: an ABAC-protected table can be shared if the share owner is exempt from the policy — the policy does not travel with the share, so treat shared copies as unprotected unless a dynamic view intervenes.

Techniques covered by the above: row level security, data masking, column level security, data anonymisation and de-identification (the 07_desensitised staging step applies desensitisation ahead of exposure — see Data zones and layers).

Reference: row and column filter limitations — “If you want to filter data when you share it using Delta Sharing, you must use dynamic views.”

Row Level Security

Within the platform: prefer ABAC row-filter policies driven by governed tags; fall back to per-table row filters only for genuinely one-off cases.
Across shares: dynamic views (as above).
Across the Fabric boundary: only DirectQuery with OAuth SSO enforces UC RLS per end user; all OneLake-side paths require Fabric-side re-securing (see Microsoft Fabric and Power BI access).
Consider the costs and overheads of each pattern: policy evaluation is near-free; dynamic views add a view layer to maintain; provider-side materialisation of view shares carries serverless compute cost (see OpenSharing practices).

Audit

Audit queries run against the system.access.audit schema — see the share-audit query under OpenSharing practices.

Typical observability requirements by role

As a Domain (workspace) Admin

1. Where are there misconfigured catalogs / schemas / objects?
2. Who is sharing what to who and is that permitted (as per access approvals?)
3. Who is accessing data and are they permitted (as per access approvals?)

Infrastructure

Environments, Workspaces and Storage

Workspaces, Environments and Storage

flowchart LR CAT["Data catalog — Unity Catalog (one metastore per region)"]:::gov subgraph SRC[" Sources "] direction TB S1["Custom / Python"] S2["Kafka / Zerobus"] S3["Databases — Lakeflow Connect"] S4["SharePoint — Lakeflow Connect"] end subgraph ENT[" Central hub "] direction TB EP["PROD workspace enterprise Raw + EDW"] ET["TEST workspace enterprise Raw + EDW"] ED["DEV workspace enterprise Raw + EDW"] end subgraph DOM[" Domain (n) — single-domain lake, workspace and repo "] direction TB DW["Domain workspace"] DP["PROD — local Raw · EDW · Infomart"] DT["TEST — local Raw · EDW · Infomart"] DD["DEV — local Raw · EDW · Infomart"] end SRC -- "feed" --> ENT EP -- "share" --> DW DW --> DP DW --> DT DW --> DD CAT -. "governs" .-> ENT CAT -. "governs" .-> DOM classDef gov fill:#FFF4B8,stroke:#1A1A1A,stroke-width:1px style SRC fill:#F7F6F2,stroke:#9A9A94 style ENT fill:#E6F2F5,stroke:#4CAAC0 style DOM fill:#FFFFFF,stroke:#4CAAC0

This diagram illustrates a data lakehouse architecture with the following components and flow:

Data Sources

Data originates from multiple sources such as:

Databases
Kafka, Zerobus or event streaming
APIs or Python scripts
SharePoint (or similar sources)

Central Hub Layer

Centralised enterprise workspaces are managed here with multiple environments.
While work can be achieved within a single workspace and lakehouse storage account, decoupling the workspaces and storage accounts allows for more isolated change at the infrastructure level — in line with engineering requirements.
Data from prod catalogs can be shared to other domains.

Domain-Specific Layer

Each domain (e.g., business units or specific applications) operates independently within a single workspace that houses multiple environments.
PROD, TEST, and DEV storage containers within a single lakehouse storage account for domain-specific data management (environment tokens per the canonical taxonomy — see Standards and Conventions).
Local Raw (Bronze) for domain-specific engineering of domain-local data (not managed by the central hub)
Data from prod catalogs can be shared to other domains.

Data Catalog

A centralised data catalog (Unity Catalog) serves as a metadata repository for the entire architecture:
Enables discovery and governance of data.
Optional external catalog storage.

Serverless workspaces — workspaces with no classic compute plane and Databricks-managed default storage — went GA in January 2026. They are a sound default for new domains with no classic-compute or custom-networking requirements, and reduce the infrastructure surface this section describes to catalogs, storage and identity.

Secrets

Under development. (Contact us to know more).

Areas for consideration include:

Management
Areas of use
Handling practices
Secret scope naming follows {domain}-{source_system}-{env} — see Standards and Conventions

Storage

Lakehouse storage

Lakehouse data for all environments and zones, by default, share a single storage account with LRS or GRS redundancy. This can then be modified according to costs, requirements, policies, projected workload and resource limits from both Azure and Databricks.

Resource: ADLSGen2
Tier: Standard/Premium (depends on workload)
Redundancy:
- Minimum ZRS or GRS for prod
- Minimum LRS for poc, dev, test and pat

Generic Blob storage

Generic Blob storage can be used for all non-lakehouse data; or alternatively within the lakehouse storage account in the appropriate container and folder.

Resource: ADLSGen2
Naming: storage accounts are subject to the 24-character Azure limit — see the storage account naming standard and abbreviation registry in Standards and Conventions
Tier: Standard/Premium (depends on workload)
Redundancy:
- Minimum ZRS or GRS for prod
- Minimum LRS for poc, dev, test and pat

CICD and Repository

CICD is built around Declarative Automation Bundles (formerly Databricks Asset Bundles) — GA, and Databricks’ recommended approach to CI/CD. A bundle (databricks.yml plus resource YAML) is the standard deployment unit for jobs, pipelines, dashboards, metric views and model endpoints.

Prescriptions:

One bundle per domain/project, versioned in that domain’s repository.
Deployment modes map to the environment taxonomy: development mode (pauses schedules and triggers, prefixes resources, permits concurrent runs) for dev; production mode (branch validation, enforced run_as) for pat and prod.
Production deploys to a shared workspace path (/Workspace/Shared/.bundle/{bundle}/{target}) and runs as a service principal (sp-{org}-{domain}-{purpose}-{env}), never a personal user.
Pipeline authentication uses workload identity federation from GitHub Actions or Azure DevOps — no Databricks secrets or personal access tokens stored in pipelines.

Tools

Github / Azure Devops (pipelines authenticated via workload identity federation)
Declarative Automation Bundles (deployment unit)

Repositories

Infrastructure IAC repos (Terraform)
dbt projects (separate for each domain)
- potential for enterprise level stitching of lineage
Data engineering code (separate for each domain) using Declarative Automation Bundles

Observability

Tools included in reference architecture:

dbt observability — Elementary (actively maintained)
Databricks observability — Unity Catalog Data Quality Monitoring and pipeline expectations
ADF — native adf monitoring (only where the ADF exception pattern is in use — see Orchestration)

dbt observability - Elementary

Elementary is a dbt observability tool available in both Open Source and Cloud Service forms. For more information, visit: Elementary Documentation

dbt warehouse observability

Example observability dashboard for Intuitas Engineering Domain

Elementary acts as a health monitor and quality checker for dbt projects by automatically tracking, alerting, and reporting on:

Data freshness: Ensures your data is up to date.
Volume anomalies: Detects unexpected changes in row counts.
Schema changes: Monitors additions, deletions, or modifications of columns.
Test results: Checks if your dbt tests are passing or failing.
Custom metrics: Allows you to define your own checks.

It leverages dbt artifacts (such as run results and sources) to send alerts to Slack, email, or other tools. Additionally, it can automatically generate reports after dbt runs, enabling early detection of issues without manual intervention.

Databricks-native observability — Data Quality Monitoring

Unity Catalog Data Quality Monitoring (formerly Lakehouse Monitoring) provides:

Data profiling (GA) — statistical profiles of tables over time.
Anomaly detection (Public Preview) — schema-level freshness and completeness monitoring without per-model instrumentation, with table health indicators surfaced in Catalog Explorer.

Positioning: complementary, not competing — Elementary observes the dbt DAG; Data Quality Monitoring observes the whole catalog, including tables no dbt project touches; Lakeflow Declarative Pipelines expectations enforce quality gates in-pipeline (see Data quality).

Networking

Areas for consideration include:

By default - all resources reside within the same VNet with private endpoints.
Service endpoints and policies can be enabled - however consider impacts on private endpoints.

Orchestration

Lakeflow Jobs (formerly Databricks Workflows) is the primary orchestrator for all lakehouse workloads; ADF is an explicitly-scoped exception, not a default. Orchestration should live where execution lives.

Primary: Lakeflow Jobs

Orchestrates notebooks, pipelines, dbt tasks and the Power BI task (publish and refresh — see Microsoft Fabric and Power BI access) from a single control plane.
Mature DAG features: conditional execution, loops, repair-and-rerun, and table-arrival and file-arrival triggers in addition to schedules.
Runs on serverless compute by default, with cost attribution via serverless usage policies (Public Preview) and the system.lakeflow job tables (GA January 2026) — see Level 1.
Job naming follows {domain}__orchestrate__{workflow_name}{optional: __schedule}{optional: __version}__{env} — see Standards and Conventions.

Exception: Azure Data Factory

Retain ADF only for:

On-premises extraction via self-hosted integration runtime where Lakeflow Connect gateways cannot reach the source.
Enterprise-wide cross-system choreography where a non-Databricks orchestrator is already established.

Security

Tools included in reference architecture

Azure Entra (group standard grp-{org}-{domain}-{plat}-{scope}-{role}-{env} — see Standards and Conventions)
Azure Key Vault
Unity Catalog
- System access reports
- ABAC policies, governed tags and data classification (see Privacy Preservation)

Watch item: Lakewatch, Databricks’ agentic SIEM announced June 2026, is a lakehouse-native security-analytics option for security teams to evaluate; it is not yet part of this reference architecture.

Level 2 - Domain-Level (Solution) Architecture and Patterns

Why It Matters

Table of Contents

Business architecture

Use cases and business requirements

Business processes

Business glossary

Business metrics

Data Architecture

Data and information models

Domain glossary

Domain data and warehouse models

Data zones and layers

Raw (Bronze) zone: Data according to source

EDW (Silver) zone: Data according to business

Infomart (Gold) zone: Data according to requirements

Lakehouse Catalog to Storage Mapping

Data Engineering

Ingestion

Ingestion patterns and notes:

Transformation

Batch and Micro-batch SQL transformation

Streaming SQL transformation

Choosing between dbt and Lakeflow Declarative Pipelines

Non SQL transformation

Data sharing and delivery patterns

Pull / direct access

Databricks OpenSharing practices

ADLSGen2 Access to Data

Open engine access via the Iceberg REST Catalog (DuckDB and others)

SQL Access

API Access

Snowflake Access

Push

Databricks-native consumption surfaces

Microsoft Fabric and Power BI access

Security boundary — read this first

Option 1. Mirrored Azure Databricks Unity Catalog + Direct Lake on OneLake (default zero-copy path)

Option 2. Power BI Import via SQL endpoint — automated with Publish to Power BI from Unity Catalog

Option 3. Power BI DirectQuery via SQL endpoint with OAuth SSO

Option 4. Share via OpenSharing (cross-tenant / cross-organisation niche)

Option 5. Raw ADLS Gen2 shortcut (niche — non-UC data)

Option 6. Replicate into Fabric (not recommended)

Recommended decision tree

Cost model per pattern

Visualisation

AI/ML

Data governance

Data lifecycle and asset management

Data access management

Data quality

Data understandability

Privacy Preservation

Row Level Security

Audit

Typical observability requirements by role

Infrastructure

Environments, Workspaces and Storage

Secrets

Storage

Lakehouse storage

Generic Blob storage

CICD and Repository

Tools

Repositories

Observability

dbt observability - Elementary

Databricks-native observability — Data Quality Monitoring

Networking

Orchestration

Primary: Lakeflow Jobs

Exception: Azure Data Factory

Security

Tools included in reference architecture