[CI] MlWithSecurityIT test {yaml=ml/data_frame_analytics_crud/Test get stats on newly created config} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #7804 / almalinux-8-aarch64_checkpart4_platform-support-arm
• elasticsearch-pull-request #68161 / part-4

Reproduction Line:

./gradlew ":x-pack:plugin:ml:qa:ml-with-security:yamlRestTest" --tests "org.elasticsearch.smoketest.MlWithSecurityIT.test {yaml=ml/data_frame_analytics_crud/Test get stats on newly created config}" -Dtests.seed=9C71502EE61EC5DB -Dtests.locale=en-BS -Dtests.timezone=SystemV/AST4ADT -Druntime.java=24

Applicable branches:
9.0

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [ml/data_frame_analytics_crud:930]: expected [2xx] status code but api [ml.get_data_frame_analytics_stats] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":null,"stack_trace":"org.elasticsearch.action.NoShardAvailableActionException\n"}],"type":"exception","reason":"all shards failed","caused_by":{"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".ml-stats-000001","node":null,"reason":{"type":"no_shard_available_action_exception","reason":null,"stack_trace":"org.elasticsearch.action.NoShardAvailableActionException\n"}}],"stack_trace":"Failed to execute phase [query], all shards failed; shardFailures {[_na_][.ml-stats-000001][0]: org.elasticsearch.action.NoShardAvailableActionException\n}\n\tat [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPha
[truncated]

Issue Reasons:

• [9.0] 2 failures in test test {yaml=ml/data_frame_analytics_crud/Test get stats on newly created config} (0.4% fail rate in 470 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test test {yaml=ml/data_frame_analytics_crud/Test get stats on newly created config} (0.7% fail rate in 409 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (1.0% fail rate in 199 executions)

Build Scans:

• elasticsearch-periodic-platform-support #7245 / sles-15_platform-support-unix
• elasticsearch-pull-request #64533 / part-4-fips
• elasticsearch-pull-request #63357 / part-4

[elasticsearchmachine]

This has been muted on branch 9.0

Mute Reasons:

• [9.0] 2 failures in test test {yaml=ml/data_frame_analytics_crud/Test get stats on newly created config} (0.4% fail rate in 470 executions)

Build Scans:

• elasticsearch-periodic-platform-support #7804 / almalinux-8-aarch64_checkpart4_platform-support-arm
• elasticsearch-pull-request #68161 / part-4