Releases: databricks/dbt-databricks
Releases · databricks/dbt-databricks
v1.11.0
Features
- Support databricks_tags for MV/STs
- Add support for scalar SQL functions (SQL UDFs) (1197)
- Add liquid clustering config for materialized views and streaming tables (thanks @reflection!) (1101)
- Add official support for
insert_overwriteincremental strategy for SQL warehouses. This strategy now usesREPLACE ONsyntax for all compute types (previouslyINSERT OVERWRITE). This behavior is gated behind behavior flaguse_replace_on_for_insert_overwritewhich defaultstrue(1025) - Add support for Databricks query tags; NOTE: This feature is still in private preview, so it is not generally usable from the adapter yet.
- Add support for managed iceberg when
table_formatis set toiceberg. This behavior is gated behind behavior flaguse_managed_icebergwhich defaults tofalse - Support delete+insert incremental strategy (thanks @canbekley!) (1217); if you have a current DBR (17.1+), this uses REPLACE ON for efficient replacement. Otherwise uses a 'delete from' statement follow by an insert statement (i.e. it is not atomic).
Fixes
- BREAKING: Fix column order mismatch bug in incremental models by using INSERT BY NAME syntax (#1211)
- When using
on_schema_change: sync_all_columns, dbt previously used positional column matching in INSERT statements, causing values to be inserted into wrong columns when column order changed - Now uses Databricks
INSERT BY NAMEsyntax to match columns by name instead of position, preventing data corruption - Breaking Change: Requires Databricks Runtime 12.2 LTS or higher
- Users on older runtimes should pin to dbt-databricks 1.10.x
- Affects all incremental strategies:
append,insert_overwrite,replace_where, andmerge(via table creation)
- When using
- Fix case-sensitivity issues with column name handling in persist_docs and config diff operations (#1215)
- Fixed KeyError when column names in models had different casing than YAML schema definitions
- Improved efficiency of column tags and comments change detection to use case-insensitive comparison
- Use backtick quoting for everything to avoid errors with special characters (1186)
- Ensure column compare always uses lower case names (since Databricks stores internally as lower case) (1190)
- Fix incompatible schema error during streaming table creation (1235)
- Reintroduce support for external table type so as not to break users (1240)
Under the Hood
- Materialized views now uses
CREATE OR REPLACEwhere appropriate, instead of DROP + CREATE - Refactor to use Databricks SDK for API calls (1185)
- Update dependency versions, and start using uv (1199)
- Upgrade ruff and mypy (1207)
- Allow create or replace semantics on full refresh in Mat V2 (1210)
- Add centralized DBR capability system for managing version-dependent features with per-compute caching (#1218)
- BREAKING: Removing the 'use_info_schema_for_columns' behavior flag, as we have a better mechanism for getting complex type information - DESCRIBE EXTENDED ... AS JSON. This is a breaking change because it requires a modern DBR (or SQL Warehouse) in order to function (1226). If this is breaking for you, you will either need to upgrade your cluster to 16.4+ or pin dbt-databricks to 1.10.x.
- Use atomic
CREATE OR REPLACEinstead of DROP + CREATE for managed Iceberg tables - BREAKING: Drop support for python 3.9, adds 3.13 (1240)
- BREAKING: Flipping the default for USE_USER_FOLDER_FOR_PYTHON to true (1248)
v1.10.14
v1.10.13
What's Changed
Fixes
- Fix issue causing MV/STs to always trigger as having their config changed (1181)
- Fix pydantic v2 deprecation warning "Valid config keys have changed in V2" (thanks @Korijn!) (1194)
- Fix snapshots not applying databricks_tags config (thanks @jeremyyeo!) (1192)
- Fix to respect varchar and char when using describe extended as json (1220)
Under the hood
New Contributors
- @Korijn made their first contribution in #1194
- @jeremyyeo made their first contribution in #1192
- @zqureshi made their first contribution in #1223
Full Changelog: v1.10.12...v1.10.13
v1.10.12
v1.10.11
What's Changed
- fix: Set correct cluster type on databricks connection by @ericj-db in #1158
- fix: Handle ANSI mode for pandas DataFrame conversion by @benc-db in #1157
- chore: Modify integration workflow to simplify running external PRs by @benc-db in #1162
- fix: add check in _get to validate model had gettable config by @colin-rogers-dbt in #1164
- fix: Remove external path on intermediate tables for python incremental models by @gsolasab in #1161
- fix: get_columns_in_relation branching for STs by @ericj-db in #1167
- fix: Improve perf on schema enumeration/validation by @benc-db in #1168
- fix: don't pass nodes that don't have
databaseattribute to build_catalog relation by @colin-rogers-dbt in #1170 - Fix: model-level override logic for is_cluster by @ericj-db in #1172
Full Changelog: v1.10.10...v1.10.11
v1.10.10
What's Changed
- DOCS: Update Databricks Workflow to match new naming Databricks Lakeflow Jobs by @excavator-matt in #1145
- write tables to different catalogs by @colin-rogers-dbt in #1129
- feat(1111): add pr linting to enforce conventional commits by @Jeremynadal33 in #1126
- chore: Tighten version range for higher risk dependencies by @ericj-db in #1147
- fix: COMMENT ON syntax not supported prior to 16.1 by @benc-db in #1151
New Contributors
- @excavator-matt made their first contribution in #1145
Full Changelog: v1.10.9...v1.10.10
v1.10.9
v1.10.8
What's Changed
- Add fallback logic for known error types for AS JSON by @ericj-db in #1131
- Support insert_overwrite incremental strategy for SQL warehouses by @ericj-db in #1127
- Simplify connection management to align with base adapter by @ericj-db in #1105
- fix: check constraints keep getting replaced by (thanks @Jeremynadal33!) in #1110
- Fix no-op logic when no databricks_tag changes to apply by @ericj-db in #1132
New Contributors
- @Jeremynadal33 made their first contribution in #1110
Full Changelog: v1.10.7...v1.10.8