From 91da480ea61544415223605865d908ceda3b57aa Mon Sep 17 00:00:00 2001 From: ravi-databricks Date: Tue, 23 Sep 2025 09:45:16 -0700 Subject: [PATCH 1/2] Added breaking changes to release doc --- CHANGELOG.md | 27 +++++++++++++++++++++++++++ docs/content/releases/_index.md | 29 +++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 96104a1..da47413 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,33 @@ # Changelog ## [v0.0.10] +### ⚠️ Breaking Changes +- **DPM Mode Removal**: Legacy DLT pipelines using the previous DPM mode must be migrated to the default publishing mode before upgrading. This change is metadata-only and doesn't impact existing datasets, but is irreversible. +- **invoke_dlt_pipeline Argument Changes**: Method arguments now require layer-specific prefixes (bronze_ or silver_) to support apply_changes_from_snapshot in both layers. This affects existing pipeline configurations using the previous argument naming. + +### Migration Guide +1. **DPM Mode Migration**: + - Before upgrading to v0.0.10, update pipeline JSON settings as per Databricks documentation [Migrate to the default publishing mode](https://docs.databricks.com/aws/en/dlt/migrate-to-dpm#migrate-to-the-default-publishing-mode) + - This is a one-way migration - ensure all stakeholders are informed + - Verify pipeline functionality in test environment first + +2. **invoke_dlt_pipeline Updates**: + - Method signature changed to support layer-specific functions: + ```python + invoke_dlt_pipeline( + spark, + layer, + bronze_custom_transform_func=None, # Previously: custom_transform_func + silver_custom_transform_func=None, # New in v0.0.10 + bronze_next_snapshot_and_version=None, # Previously: next_snapshot_and_version + silver_next_snapshot_and_version=None # New in v0.0.10 + ) + ``` + - Layer-specific functions allow different transformations for bronze and silver layers + - Existing code using single custom_transform_func should move to bronze_custom_transform_func + - Existing code using next_snapshot_and_version should move to bronze_next_snapshot_and_version + - Review and update all pipeline configurations using this method + ### Added - Added apply_changes_from_snapshot support in silver layer [PR](https://github.com/databrickslabs/dlt-meta/pull/187) - Added UI using databricks lakehouse app for onboarding/deploy commands [PR](https://github.com/databrickslabs/dlt-meta/pull/168) diff --git a/docs/content/releases/_index.md b/docs/content/releases/_index.md index 616808a..3f5924f 100644 --- a/docs/content/releases/_index.md +++ b/docs/content/releases/_index.md @@ -5,6 +5,35 @@ weight: 80 draft: false --- # v0.0.10 + +## ⚠️ Breaking Changes +- **DPM Mode Removal**: Legacy DLT pipelines using the previous DPM mode must be migrated to the default publishing mode before upgrading. The older flag (previously set via radio button) is no longer available. This change is metadata-only and doesn't impact existing datasets, but is irreversible. +- **Multi-Level Namespace Changes**: Custom schema qualification in table names is no longer supported. Tables must be created without database qualifiers. +- **invoke_dlt_pipeline Argument Changes**: Method arguments now require layer-specific prefixes (bronze_ or silver_) to support apply_changes_from_snapshot in both layers. This affects existing pipeline configurations using the previous argument naming. + +## Migration Guide +1. **DPM Mode Migration**: + - Before upgrading to v0.0.10, update pipeline JSON settings as per Databricks documentation [Migrate to the default publishing mode](https://docs.databricks.com/aws/en/dlt/migrate-to-dpm#migrate-to-the-default-publishing-mode) + - This is a one-way migration - ensure all stakeholders are informed + - Verify pipeline functionality in test environment first + +2. **invoke_dlt_pipeline Updates**: + - Method signature changed to support layer-specific functions: + ```python + invoke_dlt_pipeline( + spark, + layer, + bronze_custom_transform_func=None, # Previously: custom_transform_func + silver_custom_transform_func=None, # New in v0.0.10 + bronze_next_snapshot_and_version=None, # Previously: next_snapshot_and_version + silver_next_snapshot_and_version=None # New in v0.0.10 + ) + ``` + - Layer-specific functions allow different transformations for bronze and silver layers + - Existing code using single custom_transform_func should move to bronze_custom_transform_func + - Existing code using next_snapshot_and_version should move to bronze_next_snapshot_and_version + - Review and update all pipeline configurations using this method + ## Enhancements - Added apply_changes_from_snapshot support in silver layer [PR](https://github.com/databrickslabs/dlt-meta/pull/187) - Added UI using databricks lakehouse app for onboarding/deploy commands [PR](https://github.com/databrickslabs/dlt-meta/pull/168) From d2fba21c3f4330a380f99af649e37337396207e0 Mon Sep 17 00:00:00 2001 From: ravi-databricks Date: Thu, 6 Nov 2025 14:49:11 -0800 Subject: [PATCH 2/2] Changed DPM Mode removal docs change --- CHANGELOG.md | 2 +- docs/content/releases/_index.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index da47413..6453e6d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## [v0.0.10] ### ⚠️ Breaking Changes -- **DPM Mode Removal**: Legacy DLT pipelines using the previous DPM mode must be migrated to the default publishing mode before upgrading. This change is metadata-only and doesn't impact existing datasets, but is irreversible. +- **DPM Mode Flag Removal from v0.0.9**: DLT-META v0.0.9 pipelines using DPM mode flag must be migrated to the default publishing mode before upgrading. This change is metadata-only and doesn't impact existing datasets, but is irreversible. - **invoke_dlt_pipeline Argument Changes**: Method arguments now require layer-specific prefixes (bronze_ or silver_) to support apply_changes_from_snapshot in both layers. This affects existing pipeline configurations using the previous argument naming. ### Migration Guide diff --git a/docs/content/releases/_index.md b/docs/content/releases/_index.md index 3f5924f..02a118c 100644 --- a/docs/content/releases/_index.md +++ b/docs/content/releases/_index.md @@ -7,7 +7,7 @@ draft: false # v0.0.10 ## ⚠️ Breaking Changes -- **DPM Mode Removal**: Legacy DLT pipelines using the previous DPM mode must be migrated to the default publishing mode before upgrading. The older flag (previously set via radio button) is no longer available. This change is metadata-only and doesn't impact existing datasets, but is irreversible. +- **DPM Mode Removal**: DLT-META v0.0.9 pipelines using DPM mode flag must be migrated to the default publishing mode before upgrading. This change is metadata-only and doesn't impact existing datasets, but is irreversible. - **Multi-Level Namespace Changes**: Custom schema qualification in table names is no longer supported. Tables must be created without database qualifiers. - **invoke_dlt_pipeline Argument Changes**: Method arguments now require layer-specific prefixes (bronze_ or silver_) to support apply_changes_from_snapshot in both layers. This affects existing pipeline configurations using the previous argument naming.