Skip to content

[CI] S3BlobStoreRepositoryTests testRequestStats failing #88841

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
fcofdez opened this issue Jul 27, 2022 · 7 comments
Closed

[CI] S3BlobStoreRepositoryTests testRequestStats failing #88841

fcofdez opened this issue Jul 27, 2022 · 7 comments
Assignees
@fcofdez
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs low-risk An open issue or test failure that is a low risk to future releases stalled Team:Distributed Coordination Meta label for Distributed Coordination team >test-failure Triaged test failures from CI

Comments

@fcofdez
Copy link
Contributor

fcofdez commented Jul 27, 2022

Build scan:
https://gradle-enterprise.elastic.co/s/mkusffe2imt3i/tests/:plugins:repository-s3:internalClusterTest/org.elasticsearch.repositories.s3.S3BlobStoreRepositoryTests/testRequestStats

Reproduction line:
./gradlew ':plugins:repository-s3:internalClusterTest' --tests "org.elasticsearch.repositories.s3.S3BlobStoreRepositoryTests.testRequestStats" -Dtests.seed=E93397D14D4A1FEE -Dtests.locale=de-DE -Dtests.timezone=Asia/Chongqing -Druntime.java=17 -Dtests.fips.enabled=true

Applicable branches:
7.17

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.repositories.s3.S3BlobStoreRepositoryTests&tests.test=testRequestStats

Failure excerpt:

java.lang.AssertionError: SDK sent [{ListObjects=15, PutMultipartObject=36, PutObject=127, GetObject=129}] calls and handler measured [{PutObject=126, ListObjects=15, GetObject=129, PutMultipartObject=36}] calls expected:<{PutObject=126, ListObjects=15, GetObject=129, PutMultipartObject=36}> but was:<{ListObjects=15, PutMultipartObject=36, PutObject=127, GetObject=129}>

  at __randomizedtesting.SeedInfo.seed([E93397D14D4A1FEE:F6B723C8789934D1]:0)
  at org.junit.Assert.fail(Assert.java:88)
  at org.junit.Assert.failNotEquals(Assert.java:834)
  at org.junit.Assert.assertEquals(Assert.java:118)
  at org.elasticsearch.repositories.blobstore.ESMockAPIBasedRepositoryIntegTestCase.testRequestStats(ESMockAPIBasedRepositoryIntegTestCase.java:237)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:833)
@fcofdez fcofdez added Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Jul 27, 2022
@fcofdez fcofdez self-assigned this Jul 27, 2022
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

fcofdez added a commit to fcofdez/elasticsearch that referenced this issue Sep 8, 2022
@fcofdez
Add SDK request logging to debug failures of S3BlobStoreRepositoryTes…
b306426 
…ts#testRequestStats

Relates elastic#88841
@fcofdez fcofdez mentioned this issue Sep 8, 2022
Merged
fcofdez added a commit that referenced this issue Sep 8, 2022
@fcofdez
Add SDK request logging to debug failures of S3BlobStoreRepositoryTes…
7a07853 
…ts#testRequestStats (#89912)

Relates #88841
@fcofdez
Copy link
Contributor Author

fcofdez commented Oct 20, 2022

After adding more logging in #89912 I've been trying to figure out what happens here and it looks like somehow the connection is closed before the stats collector handler is called, in that case the SDK retries an extra time. See the logs:

  1> [2022-10-16T20:31:04,330][DEBUG][c.a.request              ] [node_t0] Sending Request: PUT http://127.0.0.1:43125 /bucket/test/1/meta-95tImZOxS8OIUXJ2xOP5bQ.dat Headers: (amz-sdk-invocation-id: 97e83af6-38bd-
a874-f90b-8094c050d850, Content-Length: 644, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.270 Linux/3.10.0-1160.76.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/19+36-2238 java/19 vendor/Oracle_Co
rporation cfg/retry-mode/legacy, x-amz-acl: private, x-amz-storage-class: STANDARD, )
  1> [2022-10-16T20:31:04,347][DEBUG][c.a.request              ] [node_t0] Retrying Request: PUT http://127.0.0.1:43125 /bucket/test/1/meta-95tImZOxS8OIUXJ2xOP5bQ.dat Headers: (amz-sdk-invocation-id: 97e83af6-38bd
-a874-f90b-8094c050d850, Content-Length: 644, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.270 Linux/3.10.0-1160.76.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/19+36-2238 java/19 vendor/Oracle_C
orporation cfg/retry-mode/legacy, x-amz-acl: private, x-amz-storage-class: STANDARD, ) <--- This happens without receiving a response
  1> [2022-10-16T20:31:04,374][DEBUG][c.a.request              ] [node_t0] Sending Request: PUT http://127.0.0.1:43125 /bucket/test/1/indices/oqJtOmwkQI6m_QujVtEz2A/meta-lfQ15IMBiJVIEG-QpOyx.dat Headers: (amz-sdk-
invocation-id: e60d4d9a-d071-bba2-940b-3d1134e719c0, Content-Length: 705, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.270 Linux/3.10.0-1160.76.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/19+36-
2238 java/19 vendor/Oracle_Corporation cfg/retry-mode/legacy, x-amz-acl: private, x-amz-storage-class: STANDARD, )
  1> [2022-10-16T20:31:04,376][DEBUG][c.a.request              ] [node_t0] Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Internal Server Error (Service: Amazon S3; Status Code: 500; E
rror Code: 500 Internal Server Error; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
  1> [2022-10-16T20:31:04,376][DEBUG][c.a.request              ] [node_t0] Retrying Request: PUT http://127.0.0.1:43125 /bucket/test/1/meta-95tImZOxS8OIUXJ2xOP5bQ.dat Headers: (amz-sdk-invocation-id: 97e83af6-38bd
-a874-f90b-8094c050d850, Content-Length: 644, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.270 Linux/3.10.0-1160.76.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/19+36-2238 java/19 vendor/Oracle_C
orporation cfg/retry-mode/legacy, x-amz-acl: private, x-amz-storage-class: STANDARD, )
  1> [2022-10-16T20:31:04,383][DEBUG][c.a.request              ] [node_t0] Sending Request: PUT http://127.0.0.1:43125 /bucket/test/1/snap-95tImZOxS8OIUXJ2xOP5bQ.dat Headers: (amz-sdk-invocation-id: 86848675-4392-
87b6-9ff1-7283aa6a880d, Content-Length: 376, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.270 Linux/3.10.0-1160.76.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/19+36-2238 java/19 vendor/Oracle_Co
rporation cfg/retry-mode/legacy, x-amz-acl: private, x-amz-storage-class: STANDARD, )
  1> [2022-10-16T20:31:04,399][DEBUG][c.a.request              ] [node_t0] Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Internal Server Error (Service: Amazon S3; Status Code: 500; E
rror Code: 500 Internal Server Error; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
  1> [2022-10-16T20:31:04,400][DEBUG][c.a.request              ] [node_t0] Retrying Request: PUT http://127.0.0.1:43125 /bucket/test/1/snap-95tImZOxS8OIUXJ2xOP5bQ.dat Headers: (amz-sdk-invocation-id: 86848675-4392
-87b6-9ff1-7283aa6a880d, Content-Length: 376, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.270 Linux/3.10.0-1160.76.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/19+36-2238 java/19 vendor/Oracle_C
orporation cfg/retry-mode/legacy, x-amz-acl: private, x-amz-storage-class: STANDARD, )
  1> [2022-10-16T20:31:04,403][DEBUG][c.a.request              ] [node_t0] Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Internal Server Error (Service: Amazon S3; Status Code: 500; E
rror Code: 500 Internal Server Error; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
  1> [2022-10-16T20:31:04,404][DEBUG][c.a.request              ] [node_t0] Retrying Request: PUT http://127.0.0.1:43125 /bucket/test/1/indices/oqJtOmwkQI6m_QujVtEz2A/meta-lfQ15IMBiJVIEG-QpOyx.dat Headers: (amz-sdk
-invocation-id: e60d4d9a-d071-bba2-940b-3d1134e719c0, Content-Length: 705, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.270 Linux/3.10.0-1160.76.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/19+36
-2238 java/19 vendor/Oracle_Corporation cfg/retry-mode/legacy, x-amz-acl: private, x-amz-storage-class: STANDARD, )
  1> [2022-10-16T20:31:04,451][DEBUG][c.a.request              ] [node_t0] Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Internal Server Error (Service: Amazon S3; Status Code: 500; E
rror Code: 500 Internal Server Error; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
  1> [2022-10-16T20:31:04,451][DEBUG][c.a.request              ] [node_t0] Retrying Request: PUT http://127.0.0.1:43125 /bucket/test/1/snap-95tImZOxS8OIUXJ2xOP5bQ.dat Headers: (amz-sdk-invocation-id: 86848675-4392
-87b6-9ff1-7283aa6a880d, Content-Length: 376, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.270 Linux/3.10.0-1160.76.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/19+36-2238 java/19 vendor/Oracle_C
orporation cfg/retry-mode/legacy, x-amz-acl: private, x-amz-storage-class: STANDARD, )
  1> [2022-10-16T20:31:04,491][DEBUG][c.a.request              ] [node_t0] Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Internal Server Error (Service: Amazon S3; Status Code: 500; E
rror Code: 500 Internal Server Error; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
  1> [2022-10-16T20:31:04,491][DEBUG][c.a.request              ] [node_t0] Retrying Request: PUT http://127.0.0.1:43125 /bucket/test/1/indices/oqJtOmwkQI6m_QujVtEz2A/meta-lfQ15IMBiJVIEG-QpOyx.dat Headers: (amz-sdk
-invocation-id: e60d4d9a-d071-bba2-940b-3d1134e719c0, Content-Length: 705, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.270 Linux/3.10.0-1160.76.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/19+36
-2238 java/19 vendor/Oracle_Corporation cfg/retry-mode/legacy, x-amz-acl: private, x-amz-storage-class: STANDARD, )
  1> [2022-10-16T20:31:04,512][DEBUG][c.a.request              ] [node_t0] Received successful response: 200, AWS Request ID: null
  1> [2022-10-16T20:31:04,513][DEBUG][c.a.request              ] [node_t0] x-amzn-RequestId: not available
  1> [2022-10-16T20:31:04,513][DEBUG][c.a.request              ] [node_t0] AWS Request ID: not available
  1> [2022-10-16T20:31:04,575][DEBUG][c.a.request              ] [node_t0] Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Internal Server Error (Service: Amazon S3; Status Code: 500; E
rror Code: 500 Internal Server Error; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
  1> [2022-10-16T20:31:04,576][DEBUG][c.a.request              ] [node_t0] Retrying Request: PUT http://127.0.0.1:43125 /bucket/test/1/meta-95tImZOxS8OIUXJ2xOP5bQ.dat Headers: (amz-sdk-invocation-id: 97e83af6-38bd
-a874-f90b-8094c050d850, Content-Length: 644, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.270 Linux/3.10.0-1160.76.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/19+36-2238 java/19 vendor/Oracle_C
orporation cfg/retry-mode/legacy, x-amz-acl: private, x-amz-storage-class: STANDARD, )

I'm not entirely sure what's causing this, any exception thrown within the test handlers should log an error and that's not happening here, therefore something is messing around with the connection.

@volodk85 volodk85 added the low-risk An open issue or test failure that is a low risk to future releases label Oct 23, 2023
@fcofdez fcofdez mentioned this issue Nov 8, 2023
Closed
@repantis repantis added Team:Distributed Coordination Meta label for Distributed Coordination team and removed Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. labels Nov 5, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

ywangd added a commit to ywangd/elasticsearch that referenced this issue Nov 11, 2024
@ywangd
[Test] Enable logging for AmazonHttpClient
e598915 
If sending request fails locally without reaching the server, the
retryable exception is logged differently. This PR enables the logging
for this scenario.

Relates: elastic#88841
Relates: elastic#101608
@ywangd ywangd mentioned this issue Nov 11, 2024
Merged
@ywangd
Copy link
Member

ywangd commented Nov 11, 2024

I spent some time looking into this CI failure from 6 month ago. The symptom is as what @fcofdez described above. There was a retrying for PUT straight after the original send without a error response in between, i.e.:

[2024-04-02T22:35:15,091][DEBUG][c.a.request              ] [node_t5] Sending Request: PUT http://127.0.0.1:34915 /bucket/test/1/indices/BLIVDOXbRUOvlwD7vqxMOw/0/index-x6obheRgTKSEjwfh8f1naQ Parameters: ({"x-purpose":["SnapshotMetadata"]}Headers: (amz-sdk-invocation-id: 05fac6b2-15e0-973e-94d5-e33f8c98b613, Content-Length: 1436, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.270 Linux/5.15.0-1053-gcp OpenJDK_64-Bit_Server_VM/21.0.2+13-58 java/21.0.2 vendor/Oracle_Corporation cfg/retry-mode/legacy, x-amz-acl: private, x-amz-storage-class: STANDARD, )
[2024-04-02T22:35:15,111][DEBUG][c.a.request              ] [node_t5] Retrying Request: PUT http://127.0.0.1:34915 /bucket/test/1/indices/BLIVDOXbRUOvlwD7vqxMOw/0/index-x6obheRgTKSEjwfh8f1naQ Parameters: ({"x-purpose":["SnapshotMetadata"]}Headers: (amz-sdk-invocation-id: 05fac6b2-15e0-973e-94d5-e33f8c98b613, Content-Length: 1436, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.270 Linux/5.15.0-1053-gcp OpenJDK_64-Bit_Server_VM/21.0.2+13-58 java/21.0.2 vendor/Oracle_Corporation cfg/retry-mode/legacy, x-amz-acl: private, x-amz-storage-class: STANDARD, )
[2024-04-02T22:35:15,133][DEBUG][c.a.request              ] [node_t5] Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Internal Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Server Error; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
[2024-04-02T22:35:15,135][DEBUG][c.a.request              ] [node_t5] Retrying Request: PUT http://127.0.0.1:34915 /bucket/test/1/indices/BLIVDOXbRUOvlwD7vqxMOw/0/index-x6obheRgTKSEjwfh8f1naQ Parameters: ({"x-purpose":["SnapshotMetadata"]}Headers: (amz-sdk-invocation-id: 05fac6b2-15e0-973e-94d5-e33f8c98b613, Content-Length: 1436, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.270 Linux/5.15.0-1053-gcp OpenJDK_64-Bit_Server_VM/21.0.2+13-58 java/21.0.2 vendor/Oracle_Corporation cfg/retry-mode/legacy, x-amz-acl: private, x-amz-storage-class: STANDARD, )
[2024-04-02T22:35:15,338][DEBUG][c.a.request              ] [node_t5] Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Internal Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Server Error; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
[2024-04-02T22:35:15,340][DEBUG][c.a.request              ] [node_t5] Retrying Request: PUT http://127.0.0.1:34915 /bucket/test/1/indices/BLIVDOXbRUOvlwD7vqxMOw/0/index-x6obheRgTKSEjwfh8f1naQ Parameters: ({"x-purpose":["SnapshotMetadata"]}Headers: (amz-sdk-invocation-id: 05fac6b2-15e0-973e-94d5-e33f8c98b613, Content-Length: 1436, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.12.270 Linux/5.15.0-1053-gcp OpenJDK_64-Bit_Server_VM/21.0.2+13-58 java/21.0.2 vendor/Oracle_Corporation cfg/retry-mode/legacy, x-amz-acl: private, x-amz-storage-class: STANDARD, )
[2024-04-02T22:35:15,368][DEBUG][c.a.request              ] [node_t5] Received successful response: 200, AWS Request ID: null

Note the 1st retry follows the original send immediately while all other retries have a Received error response message before it. Looking into the aws sdk code, I think this can happen when the request fails locally without reaching the server.

https://github.com/aws/aws-sdk-java/blob/df2ce8afe8f1c1ecebc110b9c451de4c904250fc/aws-java-sdk-core/src/main/java/com/amazonaws/http/AmazonHttpClient.java#L1189-L1191

In this case, the error is logged with the com.amazonaws.http.AmazonHttpClient logger instead of com.amazonaws.request logger. I raised #116560 to enable the logging. This test fails very rarely, every half a year or so (build-stats). So we will need wait for a while before it happens again. In the meantime, I am labelling this as stalled.

@ywangd ywangd added the stalled label Nov 11, 2024
elasticsearchmachine pushed a commit that referenced this issue Nov 13, 2024
@ywangd
[Test] Enable logging for AmazonHttpClient (#116560)
5204902 
If sending request fails locally without reaching the server, the
retryable exception is logged differently. This PR enables the logging
for this scenario.

Relates: #88841 Relates: #101608
ywangd added a commit to ywangd/elasticsearch that referenced this issue Nov 13, 2024
@ywangd
[Test] Enable logging for AmazonHttpClient (elastic#116560)
02dfa61 
If sending request fails locally without reaching the server, the
retryable exception is logged differently. This PR enables the logging
for this scenario.

Relates: elastic#88841 Relates: elastic#101608
(cherry picked from commit 5204902)

# Conflicts:
#	modules/repository-s3/src/internalClusterTest/java/org/elasticsearch/repositories/s3/S3BlobStoreRepositoryTests.java
elasticsearchmachine pushed a commit that referenced this issue Nov 13, 2024
@ywangd
[Test] Enable logging for AmazonHttpClient (#116560) (#116702)
88a1b69 
If sending request fails locally without reaching the server, the
retryable exception is logged differently. This PR enables the logging
for this scenario.

Relates: #88841 Relates: #101608
(cherry picked from commit 5204902)

# Conflicts:
#	modules/repository-s3/src/internalClusterTest/java/org/elasticsearch/repositories/s3/S3BlobStoreRepositoryTests.java
jozala pushed a commit that referenced this issue Nov 13, 2024
@ywangd @jozala
[Test] Enable logging for AmazonHttpClient (#116560)
1d9f48f 
If sending request fails locally without reaching the server, the
retryable exception is logged differently. This PR enables the logging
for this scenario.

Relates: #88841 Relates: #101608
smalyshev pushed a commit to smalyshev/elasticsearch that referenced this issue Nov 13, 2024
@ywangd @smalyshev
[Test] Enable logging for AmazonHttpClient (elastic#116560)
d6e16b6 
If sending request fails locally without reaching the server, the
retryable exception is logged differently. This PR enables the logging
for this scenario.

Relates: elastic#88841 Relates: elastic#101608
afoucret pushed a commit to afoucret/elasticsearch that referenced this issue Nov 14, 2024
@ywangd @afoucret
[Test] Enable logging for AmazonHttpClient (elastic#116560)
7018f1d 
If sending request fails locally without reaching the server, the
retryable exception is logged differently. This PR enables the logging
for this scenario.

Relates: elastic#88841 Relates: elastic#101608
alexey-ivanov-es pushed a commit to alexey-ivanov-es/elasticsearch that referenced this issue Nov 28, 2024
@ywangd @alexey-ivanov-es
[Test] Enable logging for AmazonHttpClient (elastic#116560)
f8d9c6f 
If sending request fails locally without reaching the server, the
retryable exception is logged differently. This PR enables the logging
for this scenario.

Relates: elastic#88841 Relates: elastic#101608
@ywangd
Copy link
Member

ywangd commented Jan 13, 2025

Since last update (2 months ago), there has no any meaningful CI failures. The 3 failures are unrelated:

@ywangd
Copy link
Member

ywangd commented Feb 17, 2025

Still no new failure. Just keep waiting.

@DaveCTurner
Copy link
Contributor

With the AWS SDK now upgraded to v2 in #126843 I believe this test failure is now either gone or, at the very least, changed beyond all recognition, so I'm closing it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fcofdez fcofdez

Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs low-risk An open issue or test failure that is a low risk to future releases stalled Team:Distributed Coordination Meta label for Distributed Coordination team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@fcofdez @ywangd @DaveCTurner @elasticsearchmachine @repantis @volodk85