Change History

Return to home

This page records the substantive changes between major releases of the blueprint. The blueprint documents themselves describe the current position only; the release-to-release story lives here.

Release 3.0.0 (July 2026)

A restructure of the EDW (Silver) zone; all other zones, levels and standards carry forward from 2.0.

• EDW (Silver) sub-layer model adopted. The Silver zone is reorganised from transformation-step staging (edw_stg micro-steps feeding base marts in edw) into three sub-layers representing distinct stages of transformation: edw__cleansed (1:1 with source, owned by the central hub), edw__standardised (glossary-mapped and source-separated, domain-owned) and edw__integrated (unprefixed business entities). Transformation micro-steps become the workbench vocabulary within each sub-layer, ephemeral by default with materialised exceptions (keysets, business-validation and desensitisation boundaries). The model is documented in Level 2 and the Standards.
• Keysets given a full contract. Grain, standard columns (match method, rule reference, confidence, effective dating), resolution rules and Data Vault Same-As Link semantics; keysets move to edw__integrated.
• Business history materialised once. Raw keeps arrival history; SCD2 is default-on for core entities in the integrated sub-layer, tracking history only for attributes whose changes matter to reporting. dim_/fact_ become Infomart consumption terms; base marts in edw are retired.
• Infomart repositioned as optional and views-by-default over integrated entities, materialised only where a serving pattern demands it (Fabric mirroring, open sharing, performance). The im_stg schema is renamed im__stg under the separator discipline, as are the optional vault schemas (edw__rv, edw__bv).
• Central hub naming. The central platform domain is named central_hub across examples and diagrams.

Release 2.0 (July 2026)

Release 2.0 is a fact-refresh and consistency pass, not a restructure. The architecture, zone model and level structure are unchanged from 1.x; positions and product references have been brought up to the July 2026 Databricks and Microsoft Fabric landscape, with inline links to official sources wherever a fact has changed.

Highlights

• Fabric and Power BI serving guidance re-ranked. Mirrored Azure Databricks Unity Catalog (GA) combined with Direct Lake on OneLake (GA) is now the default zero-copy path for Fabric-first consumers, with the serving options in Level 2 re-ordered and their security caveats restated accordingly.
• Ingestion and orchestration modernised around Lakeflow. Lakeflow Connect managed connectors are the first-choice ingestion pattern for supported sources; Lakeflow Declarative Pipelines (formerly Delta Live Tables) joins dbt as a first-class transformation option; and Lakeflow Jobs (formerly Databricks Workflows) is the primary orchestrator, with ADF repositioned as a hybrid-connectivity exception.
• Sharing updated for OpenSharing (formerly Delta Sharing). Guidance now covers the expanded set of shareable assets (views, materialised views, streaming tables, volumes, models and managed Iceberg tables), OIDC token federation in preference to long-lived bearer tokens, and the cost implications of view-based shares.
• Naming standards unified. A single separator discipline now applies across the estate — __ (double underscore) separates all structural name components — together with one canonical environment taxonomy, one transformation-step vocabulary and one SCD column set. See Standards & Conventions, including a consolidated old-to-new product name mapping.
• Naming coverage extended to the post-2025 platform surface, including service principals, external locations, storage credentials, SQL warehouses, serverless usage policies, Databricks Apps, Genie Agents, metric views and Fabric workspaces.
• Cost governance updated to the two-track model — cluster policies with enforced tags for classic compute, and serverless usage policies (Public Preview) with budgets for serverless compute.
• Diagrams progressively redrawn as maintainable, text-defined (Mermaid) figures; flow and topology diagrams reflect the updated 2.0 positions.

Level 0: Enterprise Context

Largely timeless; refreshed rather than restructured. The prose has been lightly tightened, and Funding and costing structures now reflects the two-track cost governance model for classic and serverless compute.

Level 1: Enterprise Architecture

The enterprise architecture reasoning of 1.0 — domain-centric design, the hybrid federated mesh topology, the logical data warehouse and the metadata architecture principles — remains sound and is retained. The substantive changes:

• Added ABAC, Governed Tags and Data Classification — the now-GA mechanism for scaling row-level security, masking and PII controls beyond per-table approaches. The PII tagging story is upgraded: in 1.0, PII tagging was a manual, user-driven activity; in 2.0, governed tags provide the controlled vocabulary, agentic classification proposes or applies the tags, and human stewards verify.
• Added Table Formats and Interoperability — Delta remains the default; managed Apache Iceberg and the Iceberg REST Catalog are now GA and change the interoperability picture. The Iceberg REST Catalog retires the 1.0-era limitation that external engines could not reach Unity Catalog governed tables.
• Added Metric Views and Business Semantics — Unity Catalog metric views join the Snappy / dbt / DataHub semantic-lineage architecture as the governed runtime semantic layer.
• Restructured Enterprise Billing around a two-track cost governance model: cluster policies for classic compute; serverless usage policies and budgets for serverless. Since 1.0, the platform’s centre of gravity has shifted towards serverless compute, where cluster policies do not apply. Job-level cost attribution now uses the system.lakeflow tables (GA January 2026), removing the dependency on job clusters. The system-table estate has expanded materially since 1.0.
• Added a security watch-list note covering Lakewatch (announced June 2026; not yet evaluated).
• Updated product terminology throughout (e.g. Lakeflow Jobs, formerly Databricks Workflows).

Level 2: Domain Architecture

The most substantially revised document in the 2.0 release. The architecture reasoning of 1.x remains sound; the product landscape beneath it has moved considerably.

• Ingestion has been rebuilt around Lakeflow Connect managed connectors as the first-choice pattern, with Zerobus Ingest added for streaming without a bus. Following the general availability of Lakeflow (June 2025), ADF moves from default batch mover to a hybrid-connectivity exception, and the native SharePoint connector displaces the Fivetran SharePoint use case.
• Transformation, previously under development, is now written: dbt remains the lead SQL framework, with Lakeflow Declarative Pipelines added as a first-class option.
• Orchestration flips the default: Lakeflow Jobs is the primary orchestrator, including for dbt.
• Data sharing and delivery is updated for OpenSharing: shareable assets expand beyond tables and change feeds; OIDC token federation is now the recommended recipient authentication; and the cost model is refreshed — consumer-borne compute applies to direct table shares, with provider-side serverless materialisation cost for view/MV/streaming-table shares to open recipients.
• A new section, Databricks-native consumption surfaces, introduces AI/BI Dashboards, Genie and Databricks Apps — and the decision gate they create ahead of any Power BI investment.
• Microsoft Fabric and Power BI access is re-ranked around the now-GA Mirrored Azure Databricks Unity Catalog and the two Direct Lake flavours, with a decision tree and per-pattern cost table. Private-endpoint workspaces are now supported for mirroring (GA via the VNet data gateway), and DirectQuery is re-affirmed as the per-user Unity Catalog enforcement path.
• CICD is now written around Declarative Automation Bundles (formerly Databricks Asset Bundles), deployment modes and workload identity federation.
• Observability retains Elementary (verified actively maintained) and adds Unity Catalog Data Quality Monitoring and pipeline expectations as complementary native tiers.
• Privacy preservation upgrades from per-table mechanisms to ABAC policies with governed tags, retaining dynamic views for share-scoped filtering.

Mapping from the 1.x Fabric/Power BI option list:

1.x option	2026 position
1. Share via Delta Sharing	Still valid — now scoped to the cross-org niche (Option 4, as OpenSharing)
2. Directlake via ADLSGen2	Valid but superseded as default by mirroring (now Option 5, niche)
3. Fabric mirrored unity catalog	GA July 2025 — promoted to Option 1, the default zero-copy path
4. PowerBI Import via SQL Endpoint	Still sound — now automatable via Publish-to-Power-BI (Option 2)
5. PowerBI DirectQuery via SQL Endpoint	Re-affirmed as the per-user UC enforcement path (Option 3)
6. Replicate into Fabric	Superseded — prefer UC managed tables in OneLake + OneLake catalog federation (Option 6)

Standards and Conventions

The 2.0 naming standards unify the conventions into single canonical rulings and track platform renames through to July 2026:

• Separator discipline tightened. Double underscore (__) now separates all structural components of a name; single underscore (_) appears only within a token (e.g. ambo_sim). Catalog granularity patterns are therefore corporate__raw__dev, machine-parseable even where domain names themselves contain single underscores. The discipline flows through consistently: keys__{entity} keysets, the stg__ staging prefix in both EDW and Infomart zones, and schedule suffixes (__hourly/__daily/__weekly).
• Single environment taxonomy. One canonical environment table (dev/test/pat/prod promoted; sandbox/poc non-promoted) now governs every section — catalogs, containers, jobs, groups, bundles and Key Vaults all draw from the same vocabulary, with pat as the sanctioned production-acceptance token.
• One ruling per concern. A single transformation-step vocabulary; one Entra group standard (grp-, carrying platform, scope, role and environment); one physical SCD column set (effective_from_datetime / effective_to_datetime / updated_datetime); the orchestrate job token; the environment token now mandatory in Key Vault names; lowercase, space-free cluster and workflow names; job/pipeline event logging defaulting to the meta schema of the domain catalog (with a dedicated audit catalog retained as a stricter-isolation option); the automation-safe owning_domain tag key; singular reference objects; and dbt sources located top-level (models/sources/).
• Product renames adopted. Delta Sharing → OpenSharing; Delta Live Tables → Lakeflow Declarative Pipelines; Databricks Workflows → Lakeflow Jobs; Databricks Asset Bundles → Declarative Automation Bundles; Genie Spaces → Genie Agents. See Appendix A for the full mapping with sources.
• New naming coverage. Service principals, Unity Catalog infrastructure objects (external locations, storage credentials, connections, secret scopes), SQL warehouses, cluster policies and serverless usage policies, Databricks Apps, Genie Agents, metric views, streaming tables, sharing recipients, and Microsoft Fabric / Power BI objects.
• Platform constraints made explicit. A consolidated limits table (Appendix B) with per-pattern length budgets and an abbreviation-registry rule; the Key Vault pattern gains an environment token; storage account patterns gain abbreviation and length rules.
• Production deployment hardened. Declarative Automation Bundles now deploy production targets to a shared workspace path and run as a service principal, authenticated via workload identity federation — 1.x deployed both dev and prod under a personal user’s workspace folder and ran production as a named user.

Modelling Framework

• Semantic layers updated for the governed-runtime era. Unity Catalog metric views — GA and open-sourced as at July 2026 — are added as the governed runtime implementation option alongside the dbt Semantic Layer and BI-tool semantic models, with new guidance on where to define measures.
• Consistency pass only. The modelling framework itself — concepts, perspectives, levels, lifecycle, domain and canonical models, warehouse model types, hierarchies — is unchanged from 1.x; this release aligns terminology and examples with the 2.0 standards documents.

Modelling Standards and Conventions

• One physical SCD column standard. effective_from_datetime / effective_to_datetime / updated_datetime is confirmed as the single physical standard across all zones; date-grain effectivity is now explicitly limited to slowly-moving reference data. A dbt snapshot column mapping has been added (dbt 1.9+ snapshot_meta_column_names), with deleted-row tracking via hard_deletes: new_record and its dbt_is_deleted flag.
• Surrogate key macro updated. The standard is dbt_utils.generate_surrogate_key([...]), replacing the retired dbt_utils.surrogate_key() (removed in dbt_utils v1.0), with a note on its changed null handling. Databricks identity-column limitations are extended (partitioning, ALTER TABLE).
• Aligned with the naming standards. Keysets are keys__{entity} (double underscore) everywhere; reference data objects carry the ref_ prefix with singular nouns; Databricks catalog patterns align to the __ separator discipline.
• Platform limits stated. Unity Catalog naming limits (255 characters; no ., / or space; stored lowercase) are called out so the patterns can be extended safely.
• Inventory extended. An object-type prefix table is added covering mv_ (materialised views) and st_ (streaming tables), with an explicit rule that storage format (Delta/Iceberg) is never encoded in names.

Adopting 2.0 over a 1.x estate

Organisations already operating on the 1.x conventions may retain them: the standards are opinionated but adaptable, and the value lies in consistency of application rather than in any individual token. Where a 1.x estate is being extended rather than rebuilt, we recommend adopting the 2.0 rules for new domains and converging opportunistically.

Release 1.x (2024–2025)

Initial publication of the Enterprise Data Intelligence Blueprint: the level structure (Level 0 enterprise context, Level 1 enterprise architecture, Level 2 domain architecture), the zone and layer model, the hybrid federated mesh topology, the Snappy / dbt / DataHub metadata architecture, and the original naming standards and modelling framework.