[CI] MlWithSecurityIT test {yaml=ml/trained_model_cat_apis/Test cat trained models} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #7503 / openjdk23_entitlements_true_checkpart4_java-matrix
• elasticsearch-pull-request #66683 / part-4

Reproduction Line:

./gradlew ":x-pack:plugin:ml:qa:ml-with-security:yamlRestTest" --tests "org.elasticsearch.smoketest.MlWithSecurityIT.test {yaml=ml/trained_model_cat_apis/Test cat trained models}" -Dtests.seed=9EF47C0940D8BA01 -Dtests.jvm.argline="-Des.entitlements.enabled=true" -Dtests.locale=es-GQ -Dtests.timezone=Europe/Madrid -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [ml/trained_model_cat_apis:94]: expected [2xx] status code but api [cat.ml_trained_models] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:41323][indices:data/read/search[phase/query]]","stack_trace":"org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41323][indices:data/read/search[phase/query]]\n"}],"type":"exception","reason":"Searching for stats for models [lang_ident_model_1,a-regression-model-1,a-regression-model-0] failed","caused_by":{"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".ml-stats-000001","node":"JE72XTihR_acnUE9dmujHw","reason":{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:41323][indices:data/read/search[phase/query]]","stack_trace":"org.elasticsearch.action.NoShardAvailableA
[truncated]

Issue Reasons:

• [main] 2 failures in test test {yaml=ml/trained_model_cat_apis/Test cat trained models} (1.2% fail rate in 162 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch 9.0

Mute Reasons:

• [9.0] 2 failures in test test {yaml=ml/trained_model_cat_apis/Test cat trained models} (1.9% fail rate in 103 executions)

Build Scans:

• elasticsearch-periodic-platform-support #7279 / sles-15_platform-support-unix
• elasticsearch-pull-request #64833 / part-4

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test test {yaml=ml/trained_model_cat_apis/Test cat trained models} (1.2% fail rate in 162 executions)

Build Scans:

• elasticsearch-periodic #7503 / openjdk23_entitlements_true_checkpart4_java-matrix
• elasticsearch-pull-request #66683 / part-4