Skip to content

[Feature] Handle new fields in databricks_pipeline resource #5249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alexott merged 2 commits into from
Dec 10, 2025
Merged

[Feature] Handle new fields in databricks_pipeline resource #5249

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
66092df
[Feature] Handle new fields in `databricks_pipeline` resource
alexott Dec 3, 2025
4dee80d
Merge branch 'main' into feature/pipeline-new-fields
alexott Dec 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions NEXT_CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

### New Features and Improvements

* Handle new fields in `databricks_pipeline` resource ([#5249](https://github.com/databricks/terraform-provider-databricks/pull/5249))
* Recreate `databricks_credential` on the `name` change ([#5248](https://github.com/databricks/terraform-provider-databricks/pull/5248)).
* Allow to specify default catalog and schema for `databricks_dashboard` ([#5259](https://github.com/databricks/terraform-provider-databricks/pull/5259)).

Expand Down
4 changes: 3 additions & 1 deletion docs/resources/pipeline.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,8 @@ The following arguments are supported:
* `include` - Paths to include.
* `exclude` - Paths to exclude.
* `gateway_definition` - The definition of a gateway pipeline to support CDC. Consists of following attributes:
* `connection_id` - Immutable. The Unity Catalog connection this gateway pipeline uses to communicate with the source.
* `connection_id` - Deprecated, Immutable. The Unity Catalog connection this gateway pipeline uses to communicate with the source. *Use `connection_name` instead!*
* `connection_name` - Immutable. The Unity Catalog connection that this gateway pipeline uses to communicate with the source.
* `gateway_storage_catalog` - Required, Immutable. The name of the catalog for the gateway pipeline's storage location.
* `gateway_storage_name` - Required. The Unity Catalog-compatible naming for the gateway storage location. This is the destination to use for the data that is extracted by the gateway. Lakeflow Declarative Pipelines system will automatically create the storage location under the catalog and schema.
* `gateway_storage_schema` - Required, Immutable. The name of the schema for the gateway pipelines's storage location.
Expand Down Expand Up @@ -167,6 +168,7 @@ The configuration for a managed ingestion pipeline. These settings cannot be use
* `connection_name` - Immutable. The Unity Catalog connection this ingestion pipeline uses to communicate with the source. Specify either ingestion_gateway_id or connection_name.
* `ingestion_gateway_id` - Immutable. Identifier for the ingestion gateway used by this ingestion pipeline to communicate with the source. Specify either ingestion_gateway_id or connection_name.
* `objects` - Required. Settings specifying tables to replicate and the destination for the replicated tables.
* `source_configurations` - Array of objects describing top-level source configurations. See the [REST API docs](https://docs.databricks.com/api/workspace/pipelines/create#ingestion_definition-source_configurations) for reference.
* `table_configuration` - Configuration settings to control the ingestion of tables. These settings are applied to all tables in the pipeline.

## Attribute Reference
Expand Down
13 changes: 8 additions & 5 deletions pipelines/resource_pipeline.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,11 +200,14 @@ func (Pipeline) CustomizeSchema(s *common.CustomizableSchema) *common.Customizab
s.SchemaPath("storage").SetForceNew()
// catalog can be updated in-place, but switching between storage and catalog requires recreation
// (handled in CustomizeDiff)
s.SchemaPath("gateway_definition", "connection_id").SetForceNew()
s.SchemaPath("gateway_definition", "gateway_storage_catalog").SetForceNew()
s.SchemaPath("gateway_definition", "gateway_storage_schema").SetForceNew()
s.SchemaPath("ingestion_definition", "connection_name").SetForceNew()
s.SchemaPath("ingestion_definition", "ingestion_gateway_id").SetForceNew()
for _, field := range []string{"connection_id", "connection_name",
"gateway_storage_catalog", "gateway_storage_schema"} {
s.SchemaPath("gateway_definition", field).SetForceNew()
}
for _, field := range []string{"connection_name", "ingestion_gateway_id",
"ingest_from_uc_foreign_catalog"} {
s.SchemaPath("ingestion_definition", field).SetForceNew()
}

// Required fields
s.SchemaPath("library", "glob", "include").SetRequired()
Expand Down
Loading