Skip to content

Conversation

@VladLazar
Copy link
Contributor

Problem

There's a bunch of TODOs in the import code.

Summary of changes

  1. Bound max import byte range to 128MiB. This might still be too high, given the default job concurrency, but it needs to be balanced with going back and forth to S3.
  2. Prevent unsigned overflow when determining key range splits for concurrent jobs
  3. Use sharded ranges to estimate task size when splitting jobs
  4. Bubble up errors that we might hit due to invalid data in the bucket back to the storage controller.
  5. Tweak the import bucket S3 client configuration.

@VladLazar VladLazar changed the title Vlad/import flow fixups pageserver: various import flow fixups May 28, 2025
@github-actions
Copy link

github-actions bot commented May 28, 2025

8525 tests run: 7942 passed, 0 failed, 583 skipped (full report)


Flaky tests (4)

Postgres 17

Postgres 15

Postgres 14

Code coverage* (full report)

  • functions: 32.3% (9068 of 28076 functions)
  • lines: 48.6% (79963 of 164567 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
c227269 at 2025-05-30T12:42:22.581Z :recycle:

@VladLazar VladLazar marked this pull request as ready for review May 29, 2025 09:12
@VladLazar VladLazar requested a review from a team as a code owner May 29, 2025 09:12
@VladLazar VladLazar requested review from erikgrinaker and problame and removed request for erikgrinaker May 29, 2025 09:12
@VladLazar VladLazar enabled auto-merge May 30, 2025 12:14
@VladLazar VladLazar added this pull request to the merge queue May 30, 2025
Merged via the queue into main with commit 6d95a3f May 30, 2025
98 checks passed
@VladLazar VladLazar deleted the vlad/import-flow-fixups branch May 30, 2025 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants