[CI] SmokeTestMultiNodeClientYamlTestSuiteIT class failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #6834 / rhel-7_platform-support-unix
• elasticsearch-periodic #6772 / openjdk21_checkpart1_java-matrix

Reproduction Line:

undefined

Applicable branches:
8.18

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

undefined

Issue Reasons:

• [8.18] 2 failures in class org.elasticsearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT (0.5% fail rate in 368 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[martijnvg]

The first mentioned CI failure is just one failing test (search.vectors/41_knn_search_bbq_hnsw:108]): https://gradle-enterprise.elastic.co/s/girtcq3zrfols/tests/task/:qa:smoke-test-multinode:yamlRestTest/details/org.elasticsearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT/test%20%7Byaml%3Dsearch.vectors%2F41_knn_search_bbq_hnsw%2FTest%20knn%20search%7D?top-execution=1

The second failure looks like ES failed to start. There are no ES log files in the artifact zip file for the multi node smoke test module, but I did find this:

<![CDATA[[2025-03-07T06:01:17,718][INFO ][o.e.n.j.JdkVectorLibrary ] [[Test worker]] vec_caps=1
[2025-03-07T06:01:17,734][INFO ][o.e.n.NativeAccess       ] [[Test worker]] Using native vector library; to disable start with -Dorg.elasticsearch.nativeaccess.enableVectorLibrary=false
[2025-03-07T06:01:17,764][INFO ][o.e.n.NativeAccess       ] [[Test worker]] Using [jdk] native provider and native methods for [Linux]
[2025-03-07T00:01:20,686][INFO ][o.e.t.c.l.DefaultLocalClusterHandle] [suite] Starting Elasticsearch test cluster 'test-cluster'
[2025-03-07T00:01:20,700][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-1]] Starting Elasticsearch node 'test-cluster-0'
[2025-03-07T00:01:20,700][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-0]] Starting Elasticsearch node 'test-cluster-1'
[2025-03-07T00:01:20,702][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-0]] Creating installation for node 'test-cluster-1' in /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741325639982297895/elastic/elasticsearch-periodic-platform-support/qa/smoke-test-multinode/build/testrun/yamlRestTest/temp/test-cluster15503777498312756333/test-cluster-1
[2025-03-07T00:01:20,701][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-1]] Creating installation for node 'test-cluster-0' in /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741325639982297895/elastic/elasticsearch-periodic-platform-support/qa/smoke-test-multinode/build/testrun/yamlRestTest/temp/test-cluster15503777498312756333/test-cluster-0
[2025-03-07T00:01:20,706][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-1]] Distribution for node 'test-cluster-0': DefaultDistributionDescriptor{version=8.18.0, snapshot=true, distributionDir=/opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741325639982297895/elastic/elasticsearch-periodic-platform-support/distribution/archives/integ-test-zip/build/install, type=INTEG_TEST}
[2025-03-07T00:01:20,706][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-0]] Distribution for node 'test-cluster-1': DefaultDistributionDescriptor{version=8.18.0, snapshot=true, distributionDir=/opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741325639982297895/elastic/elasticsearch-periodic-platform-support/distribution/archives/integ-test-zip/build/install, type=INTEG_TEST}
[2025-03-07T00:01:20,750][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-0]] Installing modules {analysis-common=org.elasticsearch.test.cluster.local.DefaultPluginInstallSpec@7a7af686, ingest-common=org.elasticsearch.test.cluster.local.DefaultPluginInstallSpec@2c5b3784, health-shards-availability=org.elasticsearch.test.cluster.local.DefaultPluginInstallSpec@37b0b8bd, data-streams=org.elasticsearch.test.cluster.local.DefaultPluginInstallSpec@2bf567ed, reindex=org.elasticsearch.test.cluster.local.DefaultPluginInstallSpec@6966ad3, mapper-extras=org.elasticsearch.test.cluster.local.DefaultPluginInstallSpec@73848a92} into node 'test-cluster-1
[2025-03-07T00:01:20,751][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-1]] Installing modules {analysis-common=org.elasticsearch.test.cluster.local.DefaultPluginInstallSpec@7a7af686, ingest-common=org.elasticsearch.test.cluster.local.DefaultPluginInstallSpec@2c5b3784, health-shards-availability=org.elasticsearch.test.cluster.local.DefaultPluginInstallSpec@37b0b8bd, data-streams=org.elasticsearch.test.cluster.local.DefaultPluginInstallSpec@2bf567ed, reindex=org.elasticsearch.test.cluster.local.DefaultPluginInstallSpec@6966ad3, mapper-extras=org.elasticsearch.test.cluster.local.DefaultPluginInstallSpec@73848a92} into node 'test-cluster-0
[2025-03-07T00:01:23,310][INFO ][process-output           ] [[elasticsearch-keystore-log-forwarder]] Created elasticsearch keystore in /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741325639982297895/elastic/elasticsearch-periodic-platform-support/qa/smoke-test-multinode/build/testrun/yamlRestTest/temp/test-cluster15503777498312756333/test-cluster-0/config/elasticsearch.keystore
[2025-03-07T00:01:23,367][INFO ][o.e.t.c.u.ProcessReaper  ] [[test-cluster-node-executor-1]] Launching reaper: /opt/buildkite-agent/.gradle/jdks/oracle_corporation-24-amd64-linux.2/bin/java -Xms4m -Xmx16m -jar /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741325639982297895/elastic/elasticsearch-periodic-platform-support/build-tools/reaper/build/libs/reaper.jar /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741325639982297895/elastic/elasticsearch-periodic-platform-support/qa/smoke-test-multinode/build/testrun/yamlRestTest/temp/reaper-14065574039405444835
[2025-03-07T00:01:23,437][INFO ][process-output           ] [[elasticsearch-keystore-log-forwarder]] Created elasticsearch keystore in /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741325639982297895/elastic/elasticsearch-periodic-platform-support/qa/smoke-test-multinode/build/testrun/yamlRestTest/temp/test-cluster15503777498312756333/test-cluster-1/config/elasticsearch.keystore
[2025-03-07T00:01:25,526][INFO ][o.e.t.c.l.DefaultLocalClusterHandle] [suite] Stopping Elasticsearch test cluster 'test-cluster', forcibly: true
[2025-03-07T00:01:25,540][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-0]] Shutting down node 'test-cluster-1'
[2025-03-07T00:01:25,541][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-1]] Shutting down node 'test-cluster-0'
]]></system-out>
  <system-err><![CDATA[Mar 07, 2025 6:01:20 AM org.apache.lucene.internal.vectorization.VectorizationProvider lookup
WARNING: You are running with Java 23 or later. To make full use of the Vector API, please update Apache Lucene.
java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741325639982297895/elastic/elasticsearch-periodic-platform-support/qa/smoke-test-multinode/build/testrun/yamlRestTest/temp/test-cluster15503777498312756333/test-cluster-0/distro/modules/x-pack-core
	at org.elasticsearch.cli.CliToolProvider.loadJars(CliToolProvider.java:87)
	at org.elasticsearch.cli.CliToolProvider.load(CliToolProvider.java:59)
	at org.elasticsearch.server.cli.ServerCli.loadTool(ServerCli.java:262)
	at org.elasticsearch.server.cli.ServerCli.autoConfigureSecurity(ServerCli.java:171)
	at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:91)
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:55)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:101)
	at org.elasticsearch.cli.Command.main(Command.java:54)
	at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:65)
Caused by: java.nio.file.NoSuchFileException: /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741325639982297895/elastic/elasticsearch-periodic-platform-support/qa/smoke-test-multinode/build/testrun/yamlRestTest/temp/test-cluster15503777498312756333/test-cluster-0/distro/modules/x-pack-core
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.asIOException(UnixException.java:115)
	at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:452)
	at java.base/java.nio.file.Files.newDirectoryStream(Files.java:445)
	at java.base/java.nio.file.Files.list(Files.java:3419)
	at org.elasticsearch.cli.CliToolProvider.loadJars(CliToolProvider.java:78)
	... 8 more
java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741325639982297895/elastic/elasticsearch-periodic-platform-support/qa/smoke-test-multinode/build/testrun/yamlRestTest/temp/test-cluster15503777498312756333/test-cluster-1/distro/modules/x-pack-core
	at org.elasticsearch.cli.CliToolProvider.loadJars(CliToolProvider.java:87)
	at org.elasticsearch.cli.CliToolProvider.load(CliToolProvider.java:59)
	at org.elasticsearch.server.cli.ServerCli.loadTool(ServerCli.java:262)
	at org.elasticsearch.server.cli.ServerCli.autoConfigureSecurity(ServerCli.java:171)
	at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:91)
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:55)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:101)
	at org.elasticsearch.cli.Command.main(Command.java:54)
	at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:65)
Caused by: java.nio.file.NoSuchFileException: /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741325639982297895/elastic/elasticsearch-periodic-platform-support/qa/smoke-test-multinode/build/testrun/yamlRestTest/temp/test-cluster15503777498312756333/test-cluster-1/distro/modules/x-pack-core
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.asIOException(UnixException.java:115)
	at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:452)
	at java.base/java.nio.file.Files.newDirectoryStream(Files.java:445)
	at java.base/java.nio.file.Files.list(Files.java:3419)
	at org.elasticsearch.cli.CliToolProvider.loadJars(CliToolProvider.java:78)
	... 8 more

Files that should be in the distribution are missing? Would @elastic/es-delivery be able to take a look at this?

[jozala]

Hi @martijnvg
The failures with Can't run executable: [...] does not exist, java.lang.RuntimeException: Execution of elasticsearch-plugin failed with exit code 74 and java.nio.file.NoSuchFileException: [...]/distro[..] have been caused by the test cluster setup bug fixed in: #124628.

If this was the only reason for the test failures, then I believe you can safely close this ticket now.

[martijnvg]

Hey @jozala, besides the other test failure I mentioned in my previous comment, I only recall seeing these no such file exceptions. So I think it is safe to close this issue.