Skip to content

feat(server-core, query-orchestrator): Use the fastest xxhash for cache keys and model versions #9397

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

KSDaemon
Copy link
Member

@KSDaemon KSDaemon commented Mar 27, 2025

Switch to using faster xxhash for cache keys and versions of data models instead of slow crypto.createHash('md5').
Use the fastest node xxhash implementation, written in ... of course rust :)

XXH3: generates 64 or 128-bit hashes, using vectorized arithmetic.

Here are some more info for those who is interested:

Changes:

  • use xxh3 in server-core package
  • use xxh3 in query-orchestrator package
  • use xxh3 in backend-shared package
  • use xxh3 in cubestore-driver package
  • add @node-rs/xxhash package

@KSDaemon KSDaemon force-pushed the feat/xxhash branch 5 times, most recently from 64626e7 to bf7a59a Compare March 27, 2025 16:01
@KSDaemon KSDaemon marked this pull request as ready for review March 27, 2025 21:11
@KSDaemon KSDaemon requested review from a team as code owners March 27, 2025 21:11
const key = crypto.createHash('md5')
.update(args.map((v) => JSON.stringify(v)).join(','))
.digest('hex');
const key = xxh3.xxh64(args.map((v) => JSON.stringify(v)).join(',')).toString(16);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make a function somewhere in @cubejs-backend-shared, with a name like defaultHasherHex16 (or defaultHasher and a tiny interface over it) and use it everywhere in the code instead of defining and requiring @node-rs/xxhash in each package.

This approach is how the Rust language does it: DefaultHasher. It will allow us to change it on the fly for testing via envs or replace the implementation whenever necessary.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely agree! Great point!

Copy link

codecov bot commented Apr 7, 2025

Codecov Report

Attention: Patch coverage is 76.92308% with 3 lines in your changes missing coverage. Please review.

Project coverage is 47.88%. Comparing base (452633c) to head (fc996a4).

Files with missing lines Patch % Lines
...ackages/cubejs-server-core/src/core/CompilerApi.js 40.00% 3 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (452633c) and HEAD (fc996a4). Click for more details.

HEAD has 2 uploads less than BASE
Flag BASE (452633c) HEAD (fc996a4)
cubesql 2 0
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #9397       +/-   ##
===========================================
- Coverage   76.40%   47.88%   -28.53%     
===========================================
  Files         400      171      -229     
  Lines      104055    21306    -82749     
  Branches     3685     3685               
===========================================
- Hits        79508    10203    -69305     
+ Misses      24112    10668    -13444     
  Partials      435      435               
Flag Coverage Δ
cube-backend 47.88% <76.92%> (+<0.01%) ⬆️
cubesql ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@KSDaemon KSDaemon marked this pull request as draft April 25, 2025 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants