Skip to content

chore: optimize dtransform payloads and have compaction always enabled #6331

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Sep 11, 2025
Merged

chore: optimize dtransform payloads and have compaction always enabled #6331

merged 2 commits into from
Sep 11, 2025

Conversation

atzoum
Copy link
Contributor

@atzoum atzoum commented Sep 10, 2025 •
edited
Loading

Description

  • From now on payload compaction will always be used as long as it is supported by rudder-transformer
  • During destination transformation we are no longer including information about libraries and credentials, since this is information pertaining only to user transformations

Linear Ticket

resolves PIPE-2379

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@atzoum atzoum force-pushed the chore.dtransformCompaction branch 3 times, most recently from abacaf1 to a33fb25 Compare September 10, 2025 13:18
@atzoum atzoum marked this pull request as ready for review September 10, 2025 14:02
@atzoum atzoum force-pushed the chore.dtransformCompaction branch from a33fb25 to 58a9e5b Compare September 10, 2025 16:48
@codecov Codecov
Copy link

codecov bot commented Sep 10, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.34%. Comparing base (84687fb) to head (ca768e8).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #6331      +/-   ##
==========================================
+ Coverage   77.23%   77.34%   +0.11%     
==========================================
  Files         524      524              
  Lines       70690    70640      -50     
==========================================
+ Hits        54595    54639      +44     
+ Misses      13238    13164      -74     
+ Partials     2857     2837      -20

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

koladilip
koladilip approved these changes Sep 11, 2025
View reviewed changes
@koladilip
Copy link
Contributor

can we do hotfix release for this?

@atzoum
Copy link
Contributor Author

atzoum commented Sep 11, 2025

can we do hotfix release for this?

@koladilip I believe this can wait to be included in the next release

@atzoum atzoum enabled auto-merge (squash) September 11, 2025 07:39
@atzoum atzoum force-pushed the chore.dtransformCompaction branch from 68bad2a to ca768e8 Compare September 11, 2025 07:40
@atzoum atzoum merged commit 63af856 into master Sep 11, 2025
89 of 91 checks passed
@atzoum atzoum deleted the chore.dtransformCompaction branch September 11, 2025 09:22
This was referenced Sep 15, 2025
Merged
Merged
satishrudderstack pushed a commit that referenced this pull request Sep 15, 2025
@devops-github-rudderstack
chore: prerelease 1.59.0-rc.1 (#6348)
b431bf4 
🤖 I have created a release *beep* *boop*
---


##
[1.59.0-rc.1](v1.58.0...v1.59.0-rc.1)
(2025-09-15)


### Features

* **router:** add request latency and count metrics for router
deliveries
([#6288](#6288))
([c8dd8ef](c8dd8ef))


### Bug Fixes

* avoid running migrations for notifier from slaves
([#6303](#6303))
([761eb46](761eb46))
* **batchrouter:** support configurable column ordering in CSVs in sftp
([#6342](#6342))
([652215b](652215b))
* populate table level schemas createdAt updatedAt
([#6344](#6344))
([a76a509](a76a509))
* remove unwanted table level schemas
([#6296](#6296))
([761eb46](761eb46))
* **router:** kinesis rate exceeded errors are being aborted with http
400 ([#6329](#6329))
([f739d7f](f739d7f))
* update ID regex to include underscores in error cleanup
([#6315](#6315))
([bc1a751](bc1a751))


### Miscellaneous

* add logs for events coming with an incorrect source and sourceRunId
([#6309](#6309))
([17afa60](17afa60))
* add readme for OAuth module
([#6335](#6335))
([3c9f251](3c9f251))
* bump go-kit
([#6321](#6321))
([87499e0](87499e0))
* bump mvdan.cc/gofumpt to v0.9.1
([#6328](#6328))
([cf9f653](cf9f653))
* capture stats for failed transformer client requests
([#6300](#6300))
([e10d5c1](e10d5c1))
* clean up legacy MTU logic
([#6314](#6314))
([a2e753d](a2e753d))
* clean up logs to trace double-encoded HLL value issue
([#6310](#6310))
([c707aea](c707aea))
* drop unused services/oauth v1 and minor v2 tidy-ups
([#6275](#6275))
([c1b84c1](c1b84c1))
* improve googlesheets oauth flow and error handling
([#6340](#6340))
([738d420](738d420))
* optimize dtransform payloads and have compaction always enabled
([#6331](#6331))
([63af856](63af856))
* remove proc_error jobsdb
([#6287](#6287))
([5b92665](5b92665))
* renaming mirroring modes
([#6325](#6325))
([84687fb](84687fb))
* **router:** ignore throttling costs in pickup throttler
([#6316](#6316))
([af85b92](af85b92))
* sync release v1.58.0 to main branch
([#6294](#6294))
([565630e](565630e))
* unify adaptive and static throttling configuration
([#6327](#6327))
([077b0fb](077b0fb))
* update rudderlabs/build-scan-push-action to 1.7.0
([#6320](#6320))
([edb02d3](edb02d3))
* updating keydb client
([#6301](#6301))
([0f9e3ec](0f9e3ec))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
satishrudderstack pushed a commit that referenced this pull request Sep 16, 2025
@devops-github-rudderstack
chore: release 1.59.0 (#6347)
ffba3b8 
🤖 I have created a release *beep* *boop*
---


##
[1.59.0](v1.58.0...v1.59.0)
(2025-09-15)


### Features

* **router:** add request latency and count metrics for router
deliveries
([#6288](#6288))
([c8dd8ef](c8dd8ef))


### Bug Fixes

* avoid running migrations for notifier from slaves
([#6303](#6303))
([761eb46](761eb46))
* **batchrouter:** support configurable column ordering in CSVs in sftp
([#6342](#6342))
([652215b](652215b))
* populate table level schemas createdAt updatedAt
([#6344](#6344))
([a76a509](a76a509))
* remove unwanted table level schemas
([#6296](#6296))
([761eb46](761eb46))
* **router:** kinesis rate exceeded errors are being aborted with http
400 ([#6329](#6329))
([f739d7f](f739d7f))
* update ID regex to include underscores in error cleanup
([#6315](#6315))
([bc1a751](bc1a751))


### Miscellaneous

* add logs for events coming with an incorrect source and sourceRunId
([#6309](#6309))
([17afa60](17afa60))
* add readme for OAuth module
([#6335](#6335))
([3c9f251](3c9f251))
* bump go-kit
([#6321](#6321))
([87499e0](87499e0))
* bump mvdan.cc/gofumpt to v0.9.1
([#6328](#6328))
([cf9f653](cf9f653))
* capture stats for failed transformer client requests
([#6300](#6300))
([e10d5c1](e10d5c1))
* clean up legacy MTU logic
([#6314](#6314))
([a2e753d](a2e753d))
* clean up logs to trace double-encoded HLL value issue
([#6310](#6310))
([c707aea](c707aea))
* drop unused services/oauth v1 and minor v2 tidy-ups
([#6275](#6275))
([c1b84c1](c1b84c1))
* improve googlesheets oauth flow and error handling
([#6340](#6340))
([738d420](738d420))
* optimize dtransform payloads and have compaction always enabled
([#6331](#6331))
([63af856](63af856))
* remove proc_error jobsdb
([#6287](#6287))
([5b92665](5b92665))
* renaming mirroring modes
([#6325](#6325))
([84687fb](84687fb))
* **router:** ignore throttling costs in pickup throttler
([#6316](#6316))
([af85b92](af85b92))
* sync release v1.58.0 to main branch
([#6294](#6294))
([565630e](565630e))
* unify adaptive and static throttling configuration
([#6327](#6327))
([077b0fb](077b0fb))
* update rudderlabs/build-scan-push-action to 1.7.0
([#6320](#6320))
([edb02d3](edb02d3))
* updating keydb client
([#6301](#6301))
([0f9e3ec](0f9e3ec))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ktgowtham ktgowtham ktgowtham approved these changes

@koladilip koladilip koladilip approved these changes

@cisse21 cisse21 Awaiting requested review from cisse21

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@atzoum @koladilip @ktgowtham