Skip to content

[CI] MinioRepositoryAnalysisRestIT testRepositoryAnalysis failing #122670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
elasticsearchmachine opened this issue Feb 15, 2025 · 6 comments · Fixed by #127166
Closed

[CI] MinioRepositoryAnalysisRestIT testRepositoryAnalysis failing #122670

elasticsearchmachine opened this issue Feb 15, 2025 · 6 comments · Fixed by #127166
Assignees
@ywangd
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Coordination Meta label for Distributed Coordination team >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Feb 15, 2025 •
edited
Loading

Build Scans:

Reproduction Line:

./gradlew ":x-pack:plugin:snapshot-repo-test-kit:qa:minio:javaRestTest" --tests "org.elasticsearch.repositories.blobstore.testkit.analyze.MinioRepositoryAnalysisRestIT.testRepositoryAnalysis" -Dtests.seed=6B294144CF0B45D0 -Dtests.locale=ast-ES -Dtests.timezone=America/Dominica -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.net.SocketTimeoutException: 60.000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]

Issue Reasons:

  • [main] 2 failures in test testRepositoryAnalysis (0.3% fail rate in 763 executions)
  • [main] 2 failures in pipeline elasticsearch-periodic-platform-support (15.4% fail rate in 13 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.
@elasticsearchmachine elasticsearchmachine added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Feb 15, 2025
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch 9.0

Mute Reasons:

  • [9.0] 2 failures in test testRepositoryAnalysis (1.4% fail rate in 138 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Feb 15, 2025
@elasticsearchmachine
Mute org.elasticsearch.repositories.blobstore.testkit.analyze.MinioRe…
1ff3021 
…positoryAnalysisRestIT testRepositoryAnalysis #122670
@elasticsearchmachine elasticsearchmachine added the needs:risk Requires assignment of a risk label (low, medium, blocker) label Feb 15, 2025
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

@elasticsearchmachine elasticsearchmachine added the Team:Distributed Coordination Meta label for Distributed Coordination team label Feb 15, 2025
@ywangd ywangd self-assigned this Feb 17, 2025
@ywangd ywangd added low-risk An open issue or test failure that is a low risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels Feb 17, 2025
@ywangd
Copy link
Member

ywangd commented Feb 17, 2025

The repo analysis request timed out. Should be a test or test environment issue. Marking it low risk.

@ywangd
Copy link
Member

ywangd commented Feb 17, 2025

It seems to me that Minio is slow at aborting multipart upload or its behaviour for concurrent multipart uploads is not as robust as what we expect.

[2025-02-16T06:24:31,081][INFO ][o.e.r.b.t.a.RepositoryAnalyzeAction] [test-cluster-0] running analysis of repository [repository] using path [temp-analysis-pDnBcyt2T32puRcorBLDpQ]
...
[2025-02-16T06:24:46,618][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [YmU4YWQ4NTctZWNhZC00NzE3LWI3OTgtZmUxY2M1NDAxZjlhLjk3MjUyNmFkLTkwM2ItNGRhZC04OGFhLTRkYmVkNTMyNzI3MngxNzM5NzE5NDcxNDA1ODg1NDUw] initiated at [2025-02-16T06:24:31.405-0900]
... Many more lines of similar logging messages
[YmU4YWQ4NTctZWNhZC00NzE3LWI3OTgtZmUxY2M1NDAxZjlhLjNkZWFjN2Q1LWVkOWEtNGJlZS1iYzJmLWRiZGMxNjlmMzYxMngxNzM5NzE5NDcxNDA1ODg1NTE2] initiated at [2025-02-16T06:24:31.405-0900]
[2025-02-16T06:25:28,492][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload 
...
... TIMED OUT

It attempted to abort the same stale multipart upload for ~42s and there are other aborts as well. The aborts likely did not complete when the request timed out at 60seconds. In the cleanup logs, we see

[2025-02-16T06:25:32,014][INFO ][o.e.r.b.t.a.MinioRepositoryAnalysisRestIT] [testRepositoryAnalysis] There are still tasks running after this test that might break subsequent tests [cluster:admin/repository/analyze, cluster:admin/repository/analyze/register, health-node[c]].

So that the analyze was still running. Not sure what we can do on the ES side if this is a Minio issue. Maybe adding and/or enabling more logs to get better picture of the progress.

@DaveCTurner DaveCTurner mentioned this issue Apr 15, 2025
Closed
@DaveCTurner
Copy link
Contributor

I also believe this is a MinIO issue and have opened minio/minio#21189 to report it. I managed to reproduce the problem while capturing network traffic using tcpdump and it seems pretty clear that there's a problem here. The aborts are all reporting success (i.e. 204 No Content) and yet the upload continues to appear in the list.

DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Apr 22, 2025
@DaveCTurner
Skip listing MPUs if TTL set to -1
e15b99f 
Recent versions of MinIO will sometimes leak multi-part uploads under
concurrent load, leaving them in the `ListMultipartUploads` output even
though they cannot be aborted. Today this causes repository analysis to
fail since compare-and-exchange operations will not even start if there
are any pre-existing uploads. This commit makes it possible to skip this
pre-flight check (and accept the performance consequences) by adjusting
the relevant settings.

Workaround for minio/minio#21189
Closes elastic#122670
@DaveCTurner DaveCTurner mentioned this issue Apr 22, 2025
Merged
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Apr 23, 2025
@DaveCTurner
Skip listing MPUs if TTL set to -1 (elastic#127166)
155fa0f 
Recent versions of MinIO will sometimes leak multi-part uploads under
concurrent load, leaving them in the `ListMultipartUploads` output even
though they cannot be aborted. Today this causes repository analysis to
fail since compare-and-exchange operations will not even start if there
are any pre-existing uploads. This commit makes it possible to skip this
pre-flight check (and accept the performance consequences) by adjusting
the relevant settings.

Workaround for minio/minio#21189
Closes elastic#122670
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Apr 23, 2025
@DaveCTurner
Skip listing MPUs if TTL set to -1 (elastic#127166)
29e1e5a 
Recent versions of MinIO will sometimes leak multi-part uploads under
concurrent load, leaving them in the `ListMultipartUploads` output even
though they cannot be aborted. Today this causes repository analysis to
fail since compare-and-exchange operations will not even start if there
are any pre-existing uploads. This commit makes it possible to skip this
pre-flight check (and accept the performance consequences) by adjusting
the relevant settings.

Workaround for minio/minio#21189
Closes elastic#122670
elasticsearchmachine pushed a commit that referenced this issue Apr 23, 2025
@DaveCTurner
Skip listing MPUs if TTL set to -1 (#127166) (#127206)
96f32bc 
Recent versions of MinIO will sometimes leak multi-part uploads under
concurrent load, leaving them in the `ListMultipartUploads` output even
though they cannot be aborted. Today this causes repository analysis to
fail since compare-and-exchange operations will not even start if there
are any pre-existing uploads. This commit makes it possible to skip this
pre-flight check (and accept the performance consequences) by adjusting
the relevant settings.

Workaround for minio/minio#21189
Closes #122670
elasticsearchmachine pushed a commit that referenced this issue Apr 23, 2025
@DaveCTurner
Skip listing MPUs if TTL set to -1 (#127166) (#127205)
c2e1a31 
Recent versions of MinIO will sometimes leak multi-part uploads under
concurrent load, leaving them in the `ListMultipartUploads` output even
though they cannot be aborted. Today this causes repository analysis to
fail since compare-and-exchange operations will not even start if there
are any pre-existing uploads. This commit makes it possible to skip this
pre-flight check (and accept the performance consequences) by adjusting
the relevant settings.

Workaround for minio/minio#21189
Closes #122670
@ywangd
Copy link
Member

ywangd commented Apr 28, 2025

Amazing. Thanks David!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ywangd ywangd

Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Coordination Meta label for Distributed Coordination team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Skip listing MPUs if TTL set to -1 DaveCTurner/elasticsearch
3 participants
@ywangd @DaveCTurner @elasticsearchmachine