Skip to content

Releases: databricks/dbt-databricks

v1.11.0

03 Nov 18:31
8cda62e

Choose a tag to compare

Features

  • Support databricks_tags for MV/STs
  • Add support for scalar SQL functions (SQL UDFs) (1197)
  • Add liquid clustering config for materialized views and streaming tables (thanks @reflection!) (1101)
  • Add official support for insert_overwrite incremental strategy for SQL warehouses. This strategy now uses REPLACE ON syntax for all compute types (previously INSERT OVERWRITE). This behavior is gated behind behavior flag use_replace_on_for_insert_overwrite which defaults true (1025)
  • Add support for Databricks query tags; NOTE: This feature is still in private preview, so it is not generally usable from the adapter yet.
  • Add support for managed iceberg when table_format is set to iceberg. This behavior is gated behind behavior flag use_managed_iceberg which defaults to false
  • Support delete+insert incremental strategy (thanks @canbekley!) (1217); if you have a current DBR (17.1+), this uses REPLACE ON for efficient replacement. Otherwise uses a 'delete from' statement follow by an insert statement (i.e. it is not atomic).

Fixes

  • BREAKING: Fix column order mismatch bug in incremental models by using INSERT BY NAME syntax (#1211)
    • When using on_schema_change: sync_all_columns, dbt previously used positional column matching in INSERT statements, causing values to be inserted into wrong columns when column order changed
    • Now uses Databricks INSERT BY NAME syntax to match columns by name instead of position, preventing data corruption
    • Breaking Change: Requires Databricks Runtime 12.2 LTS or higher
    • Users on older runtimes should pin to dbt-databricks 1.10.x
    • Affects all incremental strategies: append, insert_overwrite, replace_where, and merge (via table creation)
  • Fix case-sensitivity issues with column name handling in persist_docs and config diff operations (#1215)
    • Fixed KeyError when column names in models had different casing than YAML schema definitions
    • Improved efficiency of column tags and comments change detection to use case-insensitive comparison
  • Use backtick quoting for everything to avoid errors with special characters (1186)
  • Ensure column compare always uses lower case names (since Databricks stores internally as lower case) (1190)
  • Fix incompatible schema error during streaming table creation (1235)
  • Reintroduce support for external table type so as not to break users (1240)

Under the Hood

  • Materialized views now uses CREATE OR REPLACE where appropriate, instead of DROP + CREATE
  • Refactor to use Databricks SDK for API calls (1185)
  • Update dependency versions, and start using uv (1199)
  • Upgrade ruff and mypy (1207)
  • Allow create or replace semantics on full refresh in Mat V2 (1210)
  • Add centralized DBR capability system for managing version-dependent features with per-compute caching (#1218)
  • BREAKING: Removing the 'use_info_schema_for_columns' behavior flag, as we have a better mechanism for getting complex type information - DESCRIBE EXTENDED ... AS JSON. This is a breaking change because it requires a modern DBR (or SQL Warehouse) in order to function (1226). If this is breaking for you, you will either need to upgrade your cluster to 16.4+ or pin dbt-databricks to 1.10.x.
  • Use atomic CREATE OR REPLACE instead of DROP + CREATE for managed Iceberg tables
  • BREAKING: Drop support for python 3.9, adds 3.13 (1240)
  • BREAKING: Flipping the default for USE_USER_FOLDER_FOR_PYTHON to true (1248)

v1.10.14

22 Oct 16:34
cbc6d34

Choose a tag to compare

What's Changed

Full Changelog: v1.10.13...v1.10.14

v1.10.13

21 Oct 21:22
d5352d7

Choose a tag to compare

What's Changed

Fixes

  • Fix issue causing MV/STs to always trigger as having their config changed (1181)
  • Fix pydantic v2 deprecation warning "Valid config keys have changed in V2" (thanks @Korijn!) (1194)
  • Fix snapshots not applying databricks_tags config (thanks @jeremyyeo!) (1192)
  • Fix to respect varchar and char when using describe extended as json (1220)

Under the hood

New Contributors

Full Changelog: v1.10.12...v1.10.13

v1.10.12

08 Sep 22:31
64c6347

Choose a tag to compare

What's Changed

New Contributors

Full Changelog: v1.10.11...v1.10.12

v1.10.11

04 Sep 16:58
e5cf006

Choose a tag to compare

What's Changed

  • fix: Set correct cluster type on databricks connection by @ericj-db in #1158
  • fix: Handle ANSI mode for pandas DataFrame conversion by @benc-db in #1157
  • chore: Modify integration workflow to simplify running external PRs by @benc-db in #1162
  • fix: add check in _get to validate model had gettable config by @colin-rogers-dbt in #1164
  • fix: Remove external path on intermediate tables for python incremental models by @gsolasab in #1161
  • fix: get_columns_in_relation branching for STs by @ericj-db in #1167
  • fix: Improve perf on schema enumeration/validation by @benc-db in #1168
  • fix: don't pass nodes that don't have database attribute to build_catalog relation by @colin-rogers-dbt in #1170
  • Fix: model-level override logic for is_cluster by @ericj-db in #1172

Full Changelog: v1.10.10...v1.10.11

v1.10.10

20 Aug 17:12
b6261f2

Choose a tag to compare

What's Changed

New Contributors

Full Changelog: v1.10.9...v1.10.10

v1.10.9

07 Aug 23:17
92f1442

Choose a tag to compare

What's Changed

Full Changelog: v1.10.8...v1.10.9

v1.10.8

05 Aug 04:59
166e1e3

Choose a tag to compare

What's Changed

  • Add fallback logic for known error types for AS JSON by @ericj-db in #1131
  • Support insert_overwrite incremental strategy for SQL warehouses by @ericj-db in #1127
  • Simplify connection management to align with base adapter by @ericj-db in #1105
  • fix: check constraints keep getting replaced by (thanks @Jeremynadal33!) in #1110
  • Fix no-op logic when no databricks_tag changes to apply by @ericj-db in #1132

New Contributors

Full Changelog: v1.10.7...v1.10.8

v1.10.7

31 Jul 17:46
8f2884c

Choose a tag to compare

What's Changed

Full Changelog: 1.10.6...v1.10.7

1.10.6

30 Jul 22:56
0f50254

Choose a tag to compare

What's Changed

  • Only run DESCRIBE TABLE EXTENDED .. AS JSON for DBR >= 16.2 by @ericj-db in #1117

Full Changelog: v1.10.5...1.10.6