`ConcurrentModificationException` in `TransportClusterStatsAction`

[lkts]

Elasticsearch Version

9.1

Installed Plugins

No response

Java Version

bundled

OS Version

---

Problem Description

This error occurs:

        java.util.ConcurrentModificationException
	at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1096)
	at java.base/java.util.ArrayList$Itr.next(ArrayList.java:1050)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.admin.cluster.stats.VersionStats.of(VersionStats.java:53)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.lambda$newResponseAsync$0(TransportClusterStatsAction.java:209)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:105)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:401)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:321)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:217)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:175)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.SubscribableListener.lambda$andThenApply$2(SubscribableListener.java:535)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:465)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.SubscribableListener.newForked(SubscribableListener.java:133)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.SubscribableListener.andThenApply(SubscribableListener.java:535)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.newResponseAsync(TransportClusterStatsAction.java:186)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.newResponseAsync(TransportClusterStatsAction.java:82)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.nodes.TransportNodesAction$1.lambda$onCompletion$4(TransportNodesAction.java:167)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.nodes.TransportNodesAction.lambda$doExecute$0(TransportNodesAction.java:179)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:40)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.base/java.lang.Thread.run(Thread.java:1447)

Steps to Reproduce

unclear

Logs (if relevant)

No response

[lkts]

I suspect that this can be the modification

elasticsearch/server/src/main/java/org/elasticsearch/action/support/nodes/TransportNodesAction.java

Line 115 in 66733cb

responses.clear();

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nielsbauman]

Thanks for reporting this, @lkts! Do you happen to remember in what context you witnessed this exception? The responses.clear() line you linked might very well be the culprit, but it only runs when the task/request is cancelled. So, before deciding whether we want to give this any further attention, I'd like to understand the situation in which this error occurred a bit better.

[lkts]

I don't have any additional context unfortunately. This is from serverless logs.

[nielsbauman]

Based on the suspicion that this line is the culprit:

elasticsearch/server/src/main/java/org/elasticsearch/action/support/nodes/TransportNodesAction.java

Line 115 in 66733cb

responses.clear();

I'm going to redirect this to the distributed coordination team, as it looks to me like this is something that the TransportNodesAction class should be able to handle. Feel free to route back to Data Management if the TransportClusterStatsAction class is doing something wrong.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)