Change History

Return to home

This page records the substantive changes between major releases of the blueprint. The blueprint documents describe the current position only. Changes between releases are recorded here.

Release 2.1 (September 2026)

A restructure of the EDW (Silver) zone. All other zones, levels and standards are unchanged from 2.0.

• EDW (Silver) sub-layer model adopted. The Silver zone is reorganised from transformation-step staging (edw_stg micro-steps feeding base marts in edw) into three sub-layers. Each represents a distinct stage of transformation: edw__cleansed (1:1 with source, owned by the central hub), edw__standardised (glossary-mapped and source-separated, domain-owned) and edw__integrated (unprefixed business entities). Transformation micro-steps become the workbench vocabulary within each sub-layer. They are ephemeral by default, with materialised exceptions (keymaps, business-validation and desensitisation boundaries). The model is documented in Level 2 and the Standards.
• Keymaps given a full contract. Grain, standard columns (match method, rule reference, confidence, effective dating), resolution rules and Data Vault Same-As Link semantics. Keymaps move to edw__integrated.
• Business history materialised once. Raw keeps arrival history. SCD2 is default-on for core entities in the integrated sub-layer. It tracks history only for attributes whose changes matter to reporting. dim_/fact_ become Infomart consumption terms. Base marts in edw are retired.
• Infomart repositioned as optional and views-by-default over integrated entities, materialised only where a serving pattern requires it (Fabric mirroring, open sharing, performance). The im_stg schema is renamed im__stg under the separator discipline, as are the optional vault schemas (edw__rv, edw__bv).
• Central hub naming. The central platform domain is named central_hub across examples and diagrams.
• EDW approach positioned against dbt, Data Vault, HOOK and Kimball. The Modelling Framework compares the methods by job and states, for each, what is shared, what is different and the reason. The approach aligns with HOOK in not building a raw vault and in identity handling, and extends it with the keymap, the standardised sub-layer and the integrated entity.
• Standardised and integrated entities. Every standardised entity is scoped to its source, {entity}__{scope}, and every concept has one integrated entity as the final step of standardisation. The integrated entity is a view over a single source-scoped entity until survivorship, cross-source deduplication, Type 2 history or read cost requires it to be materialised, with the reason recorded as meta.materialisation_reason. The standardised naming pattern {source_system}__{entity} is replaced. The HOOK-aligned option and its edw__hook schema, published earlier in 2.1, are withdrawn.
• One Silver schema for standardisation. The edw__integrated schema and folder are removed. The integrated entity is in edw__standardised under the unscoped name, beside the source-scoped entities, the keymaps and the reference entities. Workbench models that are materialised are placed in edw__cleansed__stg and edw__standardised__stg, so the layer schemas show entities only. Folders map to schemas one to one.
• Keys and identity resolution. New section in the Modelling Framework: the order of preference for identifiers, the checks made before a keyset is registered, and issuing system against system of record. Keysets are named by the issuing system, and the scope on a standardised entity may differ from the keyset.
• Worked dbt example. The example project tree in Platform Standards now follows a sales domain with Customer sourced from SAP and the TechnologyOne system that replaced it, invoices from Gentrack carrying the customer id, and orders as a union across the two systems. A worked example covers the registry seeds, keymap, survivorship and retirement of a system. It is linked from the Level 2 and Platform Standards sections that introduce the Silver sub-layers.
• Conformed dimensions placed in the Infomart. Dimensional modelling is treated as a consumption requirement. Conformance logic is resolved in the integrated entity, and the domain that owns the entity builds the conformed dimension once as an Infomart product. The Level 1 two-tier mart model is replaced accordingly.
• Modelling non-negotiables added. The Modelling Framework lists the requirements that apply whichever modelling method is adopted.
• Governance tool note. DataHub remains the illustrative governance tool, and is noted as largely interchangeable with Microsoft Purview data governance (Data Map and Unified Catalog) wherever it is named.
• Keyset renamed to keymap. The key mapping table is now called a keymap, and its object prefix changes from keys__ to keymap__. The term keyset means the qualifier prefixed to a key in the form {keyset}|{business_key}. A keyset identifies one set of identifiers from one issuer, and an issuer may have more than one.
• Data Vault 2.0 deprecated. Data Vault 2.0 is no longer recommended for new work. The keymap (Same-As Link) and the business-concept orientation are retained, and dimensional (Kimball) modelling is retained in the Infomart (Gold) zone only.

Release 2.0 (July 2026)

Release 2.0 is a fact-refresh and consistency pass, not a restructure. The architecture, zone model and level structure are unchanged from 1.x. Positions and product references have been updated to the July 2026 Databricks and Microsoft Fabric products. Inline links to official sources are provided wherever a fact has changed.

Highlights

• Fabric and Power BI serving guidance re-ranked. Mirrored Azure Databricks Unity Catalog (GA) combined with Direct Lake on OneLake (GA) is now the default zero-copy path for Fabric-first consumers. The serving options in Level 2 are re-ordered and their security caveats restated.
• Ingestion and orchestration modernised around Lakeflow. Lakeflow Connect managed connectors are the first-choice ingestion pattern for supported sources. Lakeflow Declarative Pipelines (formerly Delta Live Tables) is added alongside dbt as a first-class transformation option. Lakeflow Jobs (formerly Databricks Workflows) is the primary orchestrator, and ADF is repositioned as a hybrid-connectivity exception.
• Sharing updated for OpenSharing (formerly Delta Sharing). Guidance now covers the expanded set of shareable assets (views, materialised views, streaming tables, volumes, models and managed Iceberg tables), OIDC token federation in preference to long-lived bearer tokens, and the cost implications of view-based shares.
• Naming standards unified. A single separator discipline now applies across the estate: __ (double underscore) separates all structural name components. There is also one canonical environment taxonomy, one transformation-step vocabulary and one SCD column set. See Standards & Conventions, including a consolidated old-to-new product name mapping.
• Naming coverage extended to the post-2025 platform surface, including service principals, external locations, storage credentials, SQL warehouses, serverless usage policies, Databricks Apps, Genie Agents, metric views and Fabric workspaces.
• Cost governance updated to the two-track model: cluster policies with enforced tags for classic compute, and serverless usage policies (Public Preview) with budgets for serverless compute.
• Diagrams progressively redrawn as maintainable, text-defined (Mermaid) figures; flow and topology diagrams reflect the updated 2.0 positions.

Level 0: Enterprise Context

Largely unchanged. The content is refreshed, not restructured. The prose has been edited lightly, and Funding and costing structures now reflects the two-track cost governance model for classic and serverless compute.

Level 1: Enterprise Architecture

The enterprise architecture reasoning of 1.0 remains sound and is retained. This includes domain-centric design, the hybrid federated mesh topology, the logical data warehouse and the metadata architecture principles. The substantive changes:

• Added ABAC, Governed Tags and Data Classification: the now-GA mechanism for scaling row-level security, masking and PII controls beyond per-table approaches. PII tagging is upgraded. In 1.0, PII tagging was a manual, user-driven activity. In 2.0, governed tags provide the controlled vocabulary, agentic classification proposes or applies the tags, and human stewards verify.
• Added Table Formats and Interoperability: Delta remains the default. Managed Apache Iceberg and the Iceberg REST Catalog are now GA and change the interoperability options. The Iceberg REST Catalog removes the 1.0-era limitation that external engines could not access Unity Catalog governed tables.
• Added Metric Views and Business Semantics: Unity Catalog metric views are added to the Snappy / dbt / DataHub semantic-lineage architecture as the governed runtime semantic layer.
• Restructured Enterprise Billing around a two-track cost governance model: cluster policies for classic compute; serverless usage policies and budgets for serverless. Since 1.0, the platform has shifted towards serverless compute, where cluster policies do not apply. Job-level cost attribution now uses the system.lakeflow tables (GA January 2026), removing the dependency on job clusters. The system-table estate has expanded materially since 1.0.
• Added a security watch-list note covering Lakewatch (announced June 2026; not yet evaluated).
• Updated product terminology throughout (e.g. Lakeflow Jobs, formerly Databricks Workflows).

Level 2: Domain Architecture

This is the most revised document in the 2.0 release. The architecture reasoning of 1.x remains sound. The underlying products have changed considerably.

• Ingestion has been rebuilt around Lakeflow Connect managed connectors as the first-choice pattern, with Zerobus Ingest added for streaming without a bus. Following the general availability of Lakeflow (June 2025), ADF changes from the default batch mover to a hybrid-connectivity exception, and the native SharePoint connector replaces the Fivetran SharePoint use case.
• Transformation, previously under development, is now written: dbt remains the lead SQL framework, with Lakeflow Declarative Pipelines added as a first-class option.
• Orchestration changes the default: Lakeflow Jobs is the primary orchestrator, including for dbt.
• Data sharing and delivery is updated for OpenSharing. Shareable assets expand beyond tables and change feeds. OIDC token federation is now the recommended recipient authentication. The cost model is refreshed: consumer-borne compute applies to direct table shares, with provider-side serverless materialisation cost for view/MV/streaming-table shares to open recipients.
• A new section, Databricks-native consumption surfaces, introduces AI/BI Dashboards, Genie and Databricks Apps, and the decision gate they create before any Power BI investment.
• Microsoft Fabric and Power BI access is re-ranked around the now-GA Mirrored Azure Databricks Unity Catalog and the two Direct Lake flavours, with a decision tree and per-pattern cost table. Private-endpoint workspaces are now supported for mirroring (GA via the VNet data gateway), and DirectQuery is re-affirmed as the per-user Unity Catalog enforcement path.
• CICD is now written around Declarative Automation Bundles (formerly Databricks Asset Bundles), deployment modes and workload identity federation.
• Observability retains Elementary (verified actively maintained) and adds Unity Catalog Data Quality Monitoring and pipeline expectations as complementary native tiers.
• Privacy preservation upgrades from per-table mechanisms to ABAC policies with governed tags, retaining dynamic views for share-scoped filtering.

Mapping from the 1.x Fabric/Power BI option list:

1.x option	2026 position
1. Share via Delta Sharing	Still valid. Now scoped to the cross-org niche (Option 4, as OpenSharing)
2. Directlake via ADLSGen2	Valid but superseded as default by mirroring (now Option 5, niche)
3. Fabric mirrored unity catalog	GA July 2025. Promoted to Option 1, the default zero-copy path
4. PowerBI Import via SQL Endpoint	Still sound. Now automatable via Publish-to-Power-BI (Option 2)
5. PowerBI DirectQuery via SQL Endpoint	Re-affirmed as the per-user UC enforcement path (Option 3)
6. Replicate into Fabric	Superseded. Prefer UC managed tables in OneLake + OneLake catalog federation (Option 6)

Platform Standards

The 2.0 naming standards unify the conventions into single canonical rulings and track platform renames through to July 2026:

• Separator discipline tightened. Double underscore (__) now separates all structural components of a name; single underscore (_) appears only within a token (e.g. ambo_sim). Catalog granularity patterns are therefore corporate__raw__dev, machine-parseable even where domain names themselves contain single underscores. The discipline applies consistently to keys__{entity} keysets, the stg__ staging prefix in both EDW and Infomart zones, and schedule suffixes (__hourly/__daily/__weekly).
• Single environment taxonomy. One canonical environment table (dev/test/pat/prod promoted; sandbox/poc non-promoted) now governs every section. Catalogs, containers, jobs, groups, bundles and Key Vaults all use the same vocabulary, with pat as the sanctioned production-acceptance token.
• One ruling per concern. A single transformation-step vocabulary; one Entra group standard (grp-, including platform, scope, role and environment); one physical SCD column set (effective_from_datetime / effective_to_datetime / updated_datetime); the orchestrate job token; the environment token now mandatory in Key Vault names; lowercase, space-free cluster and workflow names; job/pipeline event logging defaulting to the meta schema of the domain catalog (with a dedicated audit catalog retained as a stricter-isolation option); the automation-safe owning_domain tag key; singular reference objects; and dbt sources located top-level (models/sources/).
• Product renames adopted. Delta Sharing → OpenSharing; Delta Live Tables → Lakeflow Declarative Pipelines; Databricks Workflows → Lakeflow Jobs; Databricks Asset Bundles → Declarative Automation Bundles; Genie Spaces → Genie Agents. See Appendix A for the full mapping with sources.
• New naming coverage. Service principals, Unity Catalog infrastructure objects (external locations, storage credentials, connections, secret scopes), SQL warehouses, cluster policies and serverless usage policies, Databricks Apps, Genie Agents, metric views, streaming tables, sharing recipients, and Microsoft Fabric / Power BI objects.
• Platform constraints made explicit. A consolidated limits table (Appendix B) with per-pattern length budgets and an abbreviation-registry rule. The Key Vault pattern now includes an environment token. Storage account patterns now include abbreviation and length rules.
• Production deployment hardened. Declarative Automation Bundles now deploy production targets to a shared workspace path and run as a service principal, authenticated via workload identity federation. 1.x deployed both dev and prod under a personal user’s workspace folder and ran production as a named user.

Modelling Framework

• Semantic layers updated for the governed-runtime era. Unity Catalog metric views (GA and open-sourced as at July 2026) are added as the governed runtime implementation option alongside the dbt Semantic Layer and BI-tool semantic models, with new guidance on where to define measures.
• Consistency pass only. The modelling framework itself (concepts, perspectives, levels, lifecycle, domain and canonical models, warehouse model types, hierarchies) is unchanged from 1.x. This release aligns terminology and examples with the 2.0 standards documents.

Modelling Standards and Conventions

• One physical SCD column standard. effective_from_datetime / effective_to_datetime / updated_datetime is confirmed as the single physical standard across all zones. Date-grain effectivity is now explicitly limited to slowly-moving reference data. A dbt snapshot column mapping has been added (dbt 1.9+ snapshot_meta_column_names), with deleted-row tracking via hard_deletes: new_record and its dbt_is_deleted flag.
• Surrogate key macro updated. The standard is dbt_utils.generate_surrogate_key([...]), replacing the retired dbt_utils.surrogate_key() (removed in dbt_utils v1.0), with a note on its changed null handling. Databricks identity-column limitations are extended (partitioning, ALTER TABLE).
• Aligned with the naming standards. Keysets are keys__{entity} (double underscore) everywhere. Reference data objects have the ref_ prefix with singular nouns. Databricks catalog patterns align to the __ separator discipline.
• Platform limits stated. Unity Catalog naming limits (255 characters; no ., / or space; stored lowercase) are stated so that the patterns can be extended safely.
• Inventory extended. An object-type prefix table is added covering mv_ (materialised views) and st_ (streaming tables), with an explicit rule that storage format (Delta/Iceberg) is never encoded in names.

Adopting 2.0 over a 1.x estate

Organisations already operating on the 1.x conventions may retain them. The standards are opinionated but adaptable. The value is in consistent application, not in any individual token. Where a 1.x estate is being extended rather than rebuilt, we recommend adopting the 2.0 rules for new domains and converging opportunistically.

Release 1.x (2024–2025)

Initial publication of the Enterprise Data Intelligence Blueprint: the level structure (Level 0 enterprise context, Level 1 enterprise architecture, Level 2 domain architecture), the zone and layer model, the hybrid federated mesh topology, the Snappy / dbt / DataHub metadata architecture, and the original naming standards and modelling framework.