Skip to content

[CI] S3RepositoryAnalysisRestIT testRepositoryAnalysis failing #127399

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
elasticsearchmachine opened this issue Apr 25, 2025 · 2 comments · Fixed by #127405
Closed

[CI] S3RepositoryAnalysisRestIT testRepositoryAnalysis failing #127399

elasticsearchmachine opened this issue Apr 25, 2025 · 2 comments · Fixed by #127405
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs needs:risk Requires assignment of a risk label (low, medium, blocker) Team:Distributed Coordination Meta label for Distributed Coordination team >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Apr 25, 2025 •
edited
Loading

Build Scans:

Reproduction Line:

./gradlew ":x-pack:plugin:snapshot-repo-test-kit:qa:s3:javaRestTest" --tests "org.elasticsearch.repositories.blobstore.testkit.analyze.S3RepositoryAnalysisRestIT.testRepositoryAnalysis" -Dtests.seed=462B553BF49FAAEF -Dtests.locale=mr -Dtests.timezone=Antarctica/Troll -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:46363], URI [/_snapshot/repository/_analyze?blob_count=10&seed=-4738119588702646532&max_blob_size=10mb&timeout=120s&concurrency=4], status line [HTTP/1.1 500 Internal Server Error]
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[repository] failed to copy blob after write: [test-blob-5-DiFyXJk1S0G6OmbrdwriyA]"}],"type":"repository_verification_exception","reason":"[repository] Elasticsearch observed the storage system underneath this repository behaved incorrectly which indicates it is not suitable for use with Elasticsearch snapshots. Typically this happens when using storage other than AWS S3 which incorrectly claims to be S3-compatible. If so, please report this incompatibility to your storage supplier. Do not report Elasticsearch issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using a genuine AWS S3 
[truncated]

Issue Reasons:

  • [main] 3 failures in test testRepositoryAnalysis (1.6% fail rate in 193 executions)
  • [main] 2 failures in pipeline elasticsearch-periodic-platform-support (40.0% fail rate in 5 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.
@elasticsearchmachine elasticsearchmachine added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Apr 25, 2025
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 2 failures in test testRepositoryAnalysis (1.1% fail rate in 182 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Apr 25, 2025
@elasticsearchmachine
Mute org.elasticsearch.repositories.blobstore.testkit.analyze.S3Repos…
79e9dfd 
…itoryAnalysisRestIT testRepositoryAnalysis #127399
@elasticsearchmachine elasticsearchmachine added needs:risk Requires assignment of a risk label (low, medium, blocker) Team:Distributed Coordination Meta label for Distributed Coordination team labels Apr 25, 2025
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Apr 25, 2025
@DaveCTurner
S3BlobContainer: Revert broadened exception handler again
4c86693 
Catching `Exception` instead of `SdkException` in `copyBlob` and
`executeMultipart` led to failures in `S3RepositoryAnalysisRestIT` due
to the injected exceptions getting wrapped in `IOExceptions` that
prevented them from being caught and handled in `BlobAnalyzeAction`.

Repeat of elastic#126731, regressed due to elastic#126843
Closes elastic#127399
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Apr 25, 2025
@DaveCTurner
S3BlobContainer: Revert broadened exception handler again
78b4fb6 
Catching `Exception` instead of `SdkException` in `copyBlob` and
`executeMultipart` led to failures in `S3RepositoryAnalysisRestIT` due
to the injected exceptions getting wrapped in `IOExceptions` that
prevented them from being caught and handled in `BlobAnalyzeAction`.

Repeat of elastic#126731, regressed due to elastic#126843
Closes elastic#127399
@DaveCTurner DaveCTurner mentioned this issue Apr 25, 2025
Merged
@bcully bcully closed this as completed in 5c753a8 Apr 25, 2025
DaveCTurner added a commit that referenced this issue Apr 26, 2025
@DaveCTurner
S3BlobContainer: Revert broadened exception handler again (#127405)
6fe0aa8 
Catching `Exception` instead of `SdkException` in `copyBlob` and
`executeMultipart` led to failures in `S3RepositoryAnalysisRestIT` due
to the injected exceptions getting wrapped in `IOExceptions` that
prevented them from being caught and handled in `BlobAnalyzeAction`.

Repeat of #126731, regressed due to #126843
Closes #127399
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs needs:risk Requires assignment of a risk label (low, medium, blocker) Team:Distributed Coordination Meta label for Distributed Coordination team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

S3BlobContainer: Revert broadened exception handler again DaveCTurner/elasticsearch
1 participant
@elasticsearchmachine