Skip to content

refactor: use LogStore in Snapshot / LogSegment APIs #3452

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 20, 2025

Conversation

roeap
Copy link
Collaborator

@roeap roeap commented May 20, 2025

Description

Moving to kernel also means a significant change in our path handling. THus far we rely on object store's Path, often relative to the delta table root. In parctical terms this right now also hinders us from processing tables where data files are in different locations. More importantly though, kernel handles all path references as fully qualified URLs. As a pre-factor to further kernel adoption, we change the APIs for the internal Snapshots and LogSegement to use LogStore. LogStore carries a reference to the full table location and exposes a prefixed (to table root) as well as a non-prefixed ObjectStore along with kernels Engine. Tjis will allow us to have chanegs in functionality much more isolated in future PRs.

This PR does not change an processing logic, and just defers the decision which store to use further into the stack.

@github-actions github-actions bot added binding/python Issues for the Python package binding/rust Issues for the Rust crate labels May 20, 2025
Copy link

codecov bot commented May 20, 2025

Codecov Report

Attention: Patch coverage is 55.49738% with 85 lines in your changes missing coverage. Please review.

Project coverage is 71.70%. Comparing base (9bc3b1b) to head (d6f6ec7).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
crates/core/src/logstore/mod.rs 37.25% 30 Missing and 2 partials ⚠️
crates/core/src/kernel/snapshot/log_segment.rs 58.18% 0 Missing and 23 partials ⚠️
crates/core/src/kernel/snapshot/mod.rs 63.04% 1 Missing and 16 partials ⚠️
crates/core/src/kernel/snapshot/replay.rs 14.28% 0 Missing and 6 partials ⚠️
crates/core/src/kernel/transaction/mod.rs 50.00% 2 Missing and 1 partial ⚠️
crates/core/src/table/state.rs 71.42% 0 Missing and 2 partials ⚠️
python/src/lib.rs 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3452      +/-   ##
==========================================
- Coverage   71.77%   71.70%   -0.07%     
==========================================
  Files         152      152              
  Lines       46588    46577      -11     
  Branches    46588    46577      -11     
==========================================
- Hits        33437    33400      -37     
- Misses      11001    11026      +25     
- Partials     2150     2151       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@roeap roeap enabled auto-merge May 20, 2025 08:40
@rtyler rtyler added this to the Rust v1.0.0 milestone May 20, 2025
@roeap roeap added this pull request to the merge queue May 20, 2025
Merged via the queue into delta-io:main with commit 0b1c615 May 20, 2025
23 of 24 checks passed
@roeap roeap deleted the refactor/use-logstore branch May 20, 2025 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/python Issues for the Python package binding/rust Issues for the Rust crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants