[CI] JsonLogsFormatAndParseIT testElementsPresentOnAllLinesOfLog failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #4581 / amazonlinux-2_platform-support-aws
• elasticsearch-periodic-platform-support #4576 / amazonlinux-2023_platform-support-aws

Reproduction Line:

./gradlew ":qa:unconfigured-node-name:javaRestTest" --tests "org.elasticsearch.unconfigured_node_name.JsonLogsFormatAndParseIT.testElementsPresentOnAllLinesOfLog" -Dtests.seed=C4137AF58BE5F61F -Dtests.locale=vai-Latn -Dtests.timezone=Australia/Currie -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [GET], host [http://127.0.0.1:43369], URI [_cluster/health?wait_for_events=languid], status line [HTTP/1.1 408 Request Timeout]
{"cluster_name":"javaRestTest","status":"green","timed_out":true,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"unassigned_primary_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":61,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":22811,"active_shards_percent_as_number":100.0}

Issue Reasons:

• [8.x] 2 failures in test testElementsPresentOnAllLinesOfLog (2.2% fail rate in 90 executions)
• [8.x] 2 failures in pipeline elasticsearch-periodic-platform-support (50.0% fail rate in 4 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[mark-vieira]

Interestingly this looks to be failing with a similar timeout as #111632 and on the same platform (Amazon Linux 2023). Related?

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[rjernst]

The failure here has nothing to do with the test. The log test passed, and this failure happened during cleanup, where the health API timed out. So I am reassigning to data management for investigation.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[masseyke]

It looks like there is something in a task queue that is just not getting cleared out. One of those reports task_max_waiting_in_queue_millis of 32.8s, and the other 40s. I haven't been able to reproduce this locally (yet), and don't see any logging that might help.

[nielsbauman]

Closing, see #111431 (comment)