Skip to content

[CI] AsyncSearchSecurityIT class failing #122940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
elasticsearchmachine opened this issue Feb 19, 2025 · 7 comments
Closed

[CI] AsyncSearchSecurityIT class failing #122940

elasticsearchmachine opened this issue Feb 19, 2025 · 7 comments
Assignees
@mosche
Labels
:Core/Infra/Scripting Scripting abstractions, Painless, and Mustache low-risk An open issue or test failure that is a low risk to future releases stateful Marking issues only relevant for stateful releases Team:Core/Infra Meta label for core/infra team >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Feb 19, 2025 •
edited
Loading

Build Scans:

Reproduction Line:

undefined

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

undefined

Issue Reasons:

  • [8.x] 2 consecutive failures in class org.elasticsearch.xpack.search.AsyncSearchSecurityIT
  • [8.x] 11 consecutive failures in step windows-2022_checkpart4_platform-support-windows
  • [8.x] 11 consecutive failures in step windows-2019_checkpart4_platform-support-windows
  • [8.x] 22 failures in class org.elasticsearch.xpack.search.AsyncSearchSecurityIT (3.3% fail rate in 661 executions)
  • [8.x] 11 failures in step windows-2022_checkpart4_platform-support-windows (100.0% fail rate in 11 executions)
  • [8.x] 11 failures in step windows-2019_checkpart4_platform-support-windows (100.0% fail rate in 11 executions)
  • [8.x] 15 failures in pipeline elasticsearch-periodic-platform-support (83.3% fail rate in 18 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.
@elasticsearchmachine elasticsearchmachine added :Search/Search Search-related issues that do not fall into other categories >test-failure Triaged test failures from CI labels Feb 19, 2025
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 2 failures in class org.elasticsearch.xpack.search.AsyncSearchSecurityIT (0.9% fail rate in 225 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Feb 19, 2025
@elasticsearchmachine
Mute org.elasticsearch.xpack.search.AsyncSearchSecurityIT org.elastic…
04422cd 
…search.xpack.search.AsyncSearchSecurityIT #122940
@elasticsearchmachine elasticsearchmachine added Team:Search Meta label for search team needs:risk Requires assignment of a risk label (low, medium, blocker) labels Feb 19, 2025
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-search (Team:Search)

@benwtrent benwtrent added blocker :Core/Infra/Entitlements Entitlements infrastructure and removed :Search/Search Search-related issues that do not fall into other categories needs:risk Requires assignment of a risk label (low, medium, blocker) labels Feb 19, 2025
@benwtrent
Copy link
Member

Node crashed with this:

[2025-02-18T15:29:32,354][INFO ][o.e.c.m.DataStreamFailureStoreSettings] [yamlRestTest-0] Updated data stream name patterns for enabling failure store to [[]]
[2025-02-18T15:29:43,334][ERROR][o.e.b.Elasticsearch      ] [yamlRestTest-0] fatal exception while booting Elasticsearch org.elasticsearch.entitlement.runtime.api.NotEntitledException: Not entitled: component [x-pack-ml], module [org.elasticsearch.ml], class [class org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener], entitlement [file], operation [read], path [/dev/shm/bk/bk-agent-prod-gcp-1739892163433299512/elastic/elasticsearch-pull-request/x-pack/qa/smoke-test-security-with-mustache/build/testclusters/yamlRestTest-0/tmp/controller_log_15797]
	at [email protected]/org.elasticsearch.entitlement.runtime.policy.PolicyManager.notEntitled(PolicyManager.java:413)
	at [email protected]/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:270)
	at [email protected]/org.elasticsearch.entitlement.runtime.policy.PolicyManager.checkFileRead(PolicyManager.java:259)
	at [email protected]/org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker.check$java_io_FileInputStream$(ElasticsearchEntitlementChecker.java:1119)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java)
	at [email protected]/org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:288)
	at [email protected]/org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:277)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
	at [email protected]/org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:130)
	at [email protected]/org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:97)
	at [email protected]/org.elasticsearch.xpack.ml.process.ProcessPipes.connectLogStream(ProcessPipes.java:158)
	at [email protected]/org.elasticsearch.xpack.ml.process.NativeController.<init>(NativeController.java:85)
	at [email protected]/org.elasticsearch.xpack.ml.process.NativeController.makeNativeController(NativeController.java:66)
	at [email protected]/org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:1021)
	at [email protected]/org.elasticsearch.node.NodeConstruction.lambda$construct$17(NodeConstruction.java:923)
	at [email protected]/org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:186)
	at java.base/java.util.stream.ReferencePipeline$7$1FlatMap.accept(ReferencePipeline.java:289)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)

Marking as a blocker as its an entire test suite being muted.

@elasticsearchmachine elasticsearchmachine added Team:Core/Infra Meta label for core/infra team and removed Team:Search Meta label for search team labels Feb 19, 2025
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-core-infra (Team:Core/Infra)

@mosche mosche self-assigned this Feb 20, 2025
@mosche mosche added :Core/Infra/Scripting Scripting abstractions, Painless, and Mustache and removed :Core/Infra/Entitlements Entitlements infrastructure labels Feb 20, 2025
@mosche
Copy link
Contributor

mosche commented Feb 20, 2025 •
edited
Loading

2nd / above mentioned test failure was due to missing entitlements for reading / writing the temp folder. This has been fixed shortly after that failure.

1st failure turns out to be a fatal error that seems to be related to painless. @jdconrad would you be able to have a look?

hs_err_pid53359.log

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (constantPool.cpp:703), pid=53359, tid=53361
#  Error: ShouldNotReachHere()
#
# JRE version: OpenJDK Runtime Environment (23.0+37) (build 23+37-2369)
# Java VM: OpenJDK 64-Bit Server VM (23+37-2369, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Problematic frame:
# V  [libjvm.so+0x5a9a6c]  ConstantPool::klass_ref_at(int, Bytecodes::Code, JavaThread*)+0x17c
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /opt/buildkite-agent/builds/bk-agent-prod-aws-1739970403926477931/elastic/elasticsearch-periodic-platform-support/distribution/archives/linux-aarch64-tar/build/install/elasticsearch-9.1.0-SNAPSHOT/core.53359)
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=CLDR -Des.distribution.type=tar --enable-native-access=org.elasticsearch.nativeaccess,org.apache.lucene.core -XX:ReplayDataFile=logs/replay_pid%p.log -Djava.security.manager=allow -XX:+UseG1GC -Djava.io.tmpdir=/opt/buildkite-agent/builds/bk-agent-prod-aws-1739970403926477931/elastic/elasticsearch-periodic-platform-support/x-pack/plugin/async-search/qa/security/build/testrun/javaRestTest/temp/test-cluster15352557808681566926/test-cluster-1/tmp --add-modules=jdk.incubator.vector -XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.setAsTypeCache -XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.asTypeUncached -Dorg.apache.lucene.store.defaultReadAdvice=normal -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=/opt/buildkite-agent/builds/bk-agent-prod-aws-1739970403926477931/elastic/elasticsearch-periodic-platform-support/x-pack/plugin/async-search/qa/security/build/testrun/javaRestTest/temp/test-cluster15352557808681566926/test-cluster-1/logs -XX:ErrorFile=/opt/buildkite-agent/builds/bk-agent-prod-aws-1739970403926477931/elastic/elasticsearch-periodic-platform-support/x-pack/plugin/async-search/qa/security/build/testrun/javaRestTest/temp/test-cluster15352557808681566926/test-cluster-1/logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/opt/buildkite-agent/builds/bk-agent-prod-aws-1739970403926477931/elastic/elasticsearch-periodic-platform-support/x-pack/plugin/async-search/qa/security/build/testrun/javaRestTest/temp/test-cluster15352557808681566926/test-cluster-1/logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m -Xms512m -Xmx512m -ea -esa -Dtests.testfeatures.enabled=true -Dingest.geoip.downloader.enabled.default=false -XX:MaxDirectMemorySize=268435456 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 --module-path=/opt/buildkite-agent/builds/bk-agent-prod-aws-1739970403926477931/elastic/elasticsearch-periodic-platform-support/distribution/archives/linux-aarch64-tar/build/install/elasticsearch-9.1.0-SNAPSHOT/lib --add-modules=jdk.net --add-modules=ALL-MODULE-PATH -Djdk.module.main=org.elasticsearch.server org.elasticsearch.server/org.elasticsearch.bootstrap.Elasticsearch

Host: AArch64, 32 cores, 123G, Ubuntu 20.04.6 LTS
Time: Wed Feb 19 03:23:20 2025 -10 elapsed time: 18.072565 seconds (0d 0h 0m 18s)

---------------  T H R E A D  ---------------

Current thread (0x0000ffff9c02f150):  JavaThread "main"             [_thread_in_vm, id=53361, stack(0x0000ffffa09a2000,0x0000ffffa0aa2000) (1024K)]

Stack: [0x0000ffffa09a2000,0x0000ffffa0aa2000],  sp=0x0000ffffa0a9e040,  free space=1008k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x5a9a6c]  ConstantPool::klass_ref_at(int, Bytecodes::Code, JavaThread*)+0x17c  (constantPool.cpp:703)
V  [libjvm.so+0xa269f8]  LinkResolver::resolve_method_statically(Bytecodes::Code, constantPoolHandle const&, int, JavaThread*)+0x54
V  [libjvm.so+0x40c8b4]  Bytecode_invoke::static_target(JavaThread*)+0xa4
V  [libjvm.so+0xc2a5e8]  SharedRuntime::find_callee_info_helper(vframeStream&, Bytecodes::Code&, CallInfo&, JavaThread*)+0x3f8
V  [libjvm.so+0xc2a748]  SharedRuntime::find_callee_info(Bytecodes::Code&, CallInfo&, JavaThread*)+0x68
V  [libjvm.so+0xc2b4e8]  SharedRuntime::resolve_helper(bool, bool, JavaThread*)+0xc8
V  [libjvm.so+0xc2bafc]  SharedRuntime::resolve_opt_virtual_call_C(JavaThread*)+0x3c
v  ~RuntimeStub::resolve_opt_virtual_call 0x0000ffff8bc94010
J 8779 c2 org.elasticsearch.painless.lookup.PainlessLookupBuilder.addPainlessField(Ljava/lang/Class;Ljava/lang/Class;Ljava/lang/String;Ljava/lang/Class;Ljava/util/Map;Ljava/util/Map;)V [email protected] (767 bytes) @ 0x0000ffff8c9047d8 [0x0000ffff8c901f40+0x0000000000002898]
J 8790 c2 org.elasticsearch.painless.lookup.PainlessLookupBuilder.addPainlessField(Ljava/lang/ClassLoader;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;Ljava/util/Map;)V [email protected] (188 bytes) @ 0x0000ffff8c8fb4a8 [0x0000ffff8c8fb1c0+0x00000000000002e8]
J 8828 c1 org.elasticsearch.painless.lookup.PainlessLookupBuilder.buildFromWhitelists(Ljava/util/List;Ljava/util/Map;Ljava/util/Map;)Lorg/elasticsearch/painless/lookup/PainlessLookup; [email protected] (666 bytes) @ 0x0000ffff850062c8 [0x0000ffff85004840+0x0000000000001a88]
j  org.elasticsearch.painless.PainlessScriptEngine.<init>(Lorg/elasticsearch/common/settings/Settings;Ljava/util/Map;)V+148 [email protected]
j  org.elasticsearch.painless.PainlessPlugin.getScriptEngine(Lorg/elasticsearch/common/settings/Settings;Ljava/util/Collection;)Lorg/elasticsearch/script/ScriptEngine;+142 [email protected]
j  org.elasticsearch.script.ScriptModule.<init>(Lorg/elasticsearch/common/settings/Settings;Ljava/util/List;)V+178 [email protected]
j  org.elasticsearch.node.NodeConstruction.createScriptService(Lorg/elasticsearch/common/settings/SettingsModule;Lorg/elasticsearch/threadpool/ThreadPool;Lorg/elasticsearch/node/NodeServiceProvider;)Lorg/elasticsearch/script/ScriptService;+27 [email protected]
j  org.elasticsearch.node.NodeConstruction.prepareConstruction(Lorg/elasticsearch/env/Environment;Lorg/elasticsearch/plugins/PluginsLoader;Lorg/elasticsearch/node/NodeServiceProvider;Z)Lorg/elasticsearch/node/NodeConstruction;+173 [email protected]
j  org.elasticsearch.node.Node.<init>(Lorg/elasticsearch/env/Environment;Lorg/elasticsearch/plugins/PluginsLoader;)V+11 [email protected]
j  org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Lorg/elasticsearch/env/Environment;Lorg/elasticsearch/plugins/PluginsLoader;Lorg/elasticsearch/bootstrap/Bootstrap;)V+8 [email protected]
j  org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Lorg/elasticsearch/bootstrap/Bootstrap;)V+16 [email protected]
j  org.elasticsearch.bootstrap.Elasticsearch.main([Ljava/lang/String;)V+27 [email protected]
v  ~StubRoutines::call_stub 0x0000ffff8bc58114
V  [libjvm.so+0x7d7368]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x218
V  [libjvm.so+0x87d2e4]  jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, JavaThread*) [clone .constprop.1]+0x234
V  [libjvm.so+0x87f10c]  jni_CallStaticVoidMethod+0xec
C  [libjli.so+0x3a9c]  invokeStaticMainWithArgs+0x78
C  [libjli.so+0x5400]  JavaMain+0xdb0
C  [libjli.so+0x7b2c]  ThreadJavaMain+0xc
C  [libpthread.so.0+0x7624]  start_thread+0x184
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v  ~RuntimeStub::resolve_opt_virtual_call 0x0000ffff8bc94010
J 8779 c2 org.elasticsearch.painless.lookup.PainlessLookupBuilder.addPainlessField(Ljava/lang/Class;Ljava/lang/Class;Ljava/lang/String;Ljava/lang/Class;Ljava/util/Map;Ljava/util/Map;)V [email protected] (767 bytes) @ 0x0000ffff8c9047d8 [0x0000ffff8c901f40+0x0000000000002898]
J 8790 c2 org.elasticsearch.painless.lookup.PainlessLookupBuilder.addPainlessField(Ljava/lang/ClassLoader;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;Ljava/util/Map;)V [email protected] (188 bytes) @ 0x0000ffff8c8fb4a8 [0x0000ffff8c8fb1c0+0x00000000000002e8]
J 8828 c1 org.elasticsearch.painless.lookup.PainlessLookupBuilder.buildFromWhitelists(Ljava/util/List;Ljava/util/Map;Ljava/util/Map;)Lorg/elasticsearch/painless/lookup/PainlessLookup; [email protected] (666 bytes) @ 0x0000ffff850062c8 [0x0000ffff85004840+0x0000000000001a88]
j  org.elasticsearch.painless.PainlessScriptEngine.<init>(Lorg/elasticsearch/common/settings/Settings;Ljava/util/Map;)V+148 [email protected]
j  org.elasticsearch.painless.PainlessPlugin.getScriptEngine(Lorg/elasticsearch/common/settings/Settings;Ljava/util/Collection;)Lorg/elasticsearch/script/ScriptEngine;+142 [email protected]
j  org.elasticsearch.script.ScriptModule.<init>(Lorg/elasticsearch/common/settings/Settings;Ljava/util/List;)V+178 [email protected]
j  org.elasticsearch.node.NodeConstruction.createScriptService(Lorg/elasticsearch/common/settings/SettingsModule;Lorg/elasticsearch/threadpool/ThreadPool;Lorg/elasticsearch/node/NodeServiceProvider;)Lorg/elasticsearch/script/ScriptService;+27 [email protected]
j  org.elasticsearch.node.NodeConstruction.prepareConstruction(Lorg/elasticsearch/env/Environment;Lorg/elasticsearch/plugins/PluginsLoader;Lorg/elasticsearch/node/NodeServiceProvider;Z)Lorg/elasticsearch/node/NodeConstruction;+173 [email protected]
j  org.elasticsearch.node.Node.<init>(Lorg/elasticsearch/env/Environment;Lorg/elasticsearch/plugins/PluginsLoader;)V+11 [email protected]
j  org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Lorg/elasticsearch/env/Environment;Lorg/elasticsearch/plugins/PluginsLoader;Lorg/elasticsearch/bootstrap/Bootstrap;)V+8 [email protected]
j  org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Lorg/elasticsearch/bootstrap/Bootstrap;)V+16 [email protected]
j  org.elasticsearch.bootstrap.Elasticsearch.main([Ljava/lang/String;)V+27 [email protected]
v  ~StubRoutines::call_stub 0x0000ffff8bc58114

@mosche
Copy link
Contributor

mosche commented Feb 24, 2025

Unmuting to gather more samples, we know to little about above yet.
Could be related to https://bugs.openjdk.org/browse/JDK-8346911

@breskeby breskeby added the stateful Marking issues only relevant for stateful releases label Feb 25, 2025
@arpadkiraly arpadkiraly added low-risk An open issue or test failure that is a low risk to future releases and removed blocker labels Apr 10, 2025
@mosche
Copy link
Contributor

mosche commented Apr 10, 2025

The 1st issue (entitlement issue) was fixed already. Closing this as there hasn't been another JVM crash after unmuting.

@mosche mosche closed this as completed Apr 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mosche mosche

Labels
:Core/Infra/Scripting Scripting abstractions, Painless, and Mustache low-risk An open issue or test failure that is a low risk to future releases stateful Marking issues only relevant for stateful releases Team:Core/Infra Meta label for core/infra team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@breskeby @mosche @benwtrent @elasticsearchmachine @arpadkiraly