Term vector API on stateless search nodes #129902

kingherc · 2025-06-24T09:03:59Z

Up to now, the (m)term vector API real-time requests were being executed on the indexing nodes of serverless (see #94257). However, we would like to execute them on the search nodes, similar to real-time (m)GETs. This PR does that, by introducing an intermediate action for search nodes to become up-to-date with an indexing node in respect to the term vector API request, before executing it locally on the search node.

The new intermediate action searches for any of the requested document IDs in the shard's LiveVersionMap and if it finds any of them there, it means the search nodes need to be refreshed in order to capture the new document IDs before searching for them.

Relates ES-12112

Up to now, the (m)term vector API real-time requests were being executed on the indexing nodes of serverless. However, we would like to execute them on the search nodes, similar to real-time (m)GETs. This PR does that, by introducing an intermediate action for search nodes to become up-to-date with an indexing node in respect to the term vector API request, before executing it locally on the search node. The new intermediate action searches for any of the requested document IDs in the shard's LiveVersionMap and if it finds any of them there, it means the search nodes need to be refreshed in order to capture the new document IDs before searching for them. Relates ES-12112

elasticsearchmachine · 2025-07-01T17:06:22Z

Pinging @elastic/es-search (Team:Search)

elasticsearchmachine · 2025-07-01T17:06:23Z

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

tlrx · 2025-07-02T13:17:00Z

.../src/main/java/org/elasticsearch/action/termvectors/TransportEnsureDocsSearchableAction.java

+ * with a no-op (the search node can proceed to process the term vector request). The action may trigger an external refresh
+ * to ensure the search shards are up to date before returning the no-op.
+ */
+public class TransportEnsureDocsSearchableAction extends TransportSingleShardAction<


Could we have only the ActionType and the request in server, and the rest lies into the serverless code base? Similarly to StatelessPrimaryRelocationAction?

Yes, I would prefer that too.

tlrx · 2025-07-02T13:18:20Z

server/src/main/java/org/elasticsearch/index/engine/Engine.java

@@ -902,6 +902,10 @@ protected final GetResult getFromSearcher(Get get, Engine.Searcher searcher, boo
        }
    }

+    public boolean isDocumentInLiveVersionMap(BytesRef uid) {


Even if the method name is meaningful, could we add a bit of documentation?

tlrx · 2025-07-02T13:25:08Z

server/src/main/java/org/elasticsearch/action/termvectors/TransportTermVectorsAction.java

-        );
+            );
+        if (iterator == null) {
+            return null;


So it will execute on the indexing shard in that case?

I think that it'll execute the request in the receiving node. I think that we should return an empty iterator instead.

tlrx · 2025-07-02T13:29:35Z

...src/main/java/org/elasticsearch/action/termvectors/TransportShardMultiTermsVectorAction.java

+        ShardId shardId,
+        ActionListener<MultiTermVectorsShardResponse> listener
+    ) throws IOException {
+        if (DiscoveryNode.isStateless(clusterService.getSettings())) {


I suppose we can capture this only once instead of reevaluating for every request?

fcofdez

Looks good! I left a few comments

fcofdez · 2025-07-02T19:48:48Z

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/mtermvectors/30_routing.yml

@@ -7,11 +7,6 @@ routing:
          settings:
           index:
              number_of_shards: 5
-              number_of_replicas: 0
-
-  - do:


is this removed intentionally?

fcofdez · 2025-07-02T19:52:51Z

.../src/main/java/org/elasticsearch/action/termvectors/TransportEnsureDocsSearchableAction.java

+                logger.debug("refreshing index shard [{}] due to mtv_eds", shardId);
+                BasicReplicationRequest refreshRequest = new BasicReplicationRequest(shardId);
+                refreshRequest.waitForActiveShards(ActiveShardCount.NONE);
+                client.executeLocally(TransportShardRefreshAction.TYPE, refreshRequest, l.delegateFailureAndWrap((ll, r) -> {


can we add a comment here pointing that we have to go through the transport action instead of just calling refresh in the indexShard instance to ensure that the unpromotable shards converge towards the refreshed generation?

fcofdez · 2025-07-02T19:56:23Z

.../src/main/java/org/elasticsearch/action/termvectors/TransportEnsureDocsSearchableAction.java

+                // ongoing refresh and before the search shards being updated with the new commit, because the documents are
+                // guaranteed to be the in the live version map archive until search shards are updated with the new commit.
+                // Thus, we can safely respond immediately as a no-op.
+                logger.debug("mts_eds does not require refresh of index shard [{}]", shardId);


I think you meant mtv_eds?

fcofdez · 2025-07-02T19:57:43Z

.../src/main/java/org/elasticsearch/action/termvectors/TransportEnsureDocsSearchableAction.java

+        assert DiscoveryNode.hasRole(clusterService.getSettings(), DiscoveryNodeRole.INDEX_ROLE)
+            : ACTION_NAME + " should only be executed on a stateless indexing node";
+        logger.debug("received request with {} docs", request.docIds.length);
+        getExecutor(shardId).execute(() -> ActionListener.run(listener, l -> {


are we ok failing the request if the primary moved in the meantime?

fcofdez · 2025-07-02T19:58:05Z

.../src/main/java/org/elasticsearch/action/termvectors/TransportEnsureDocsSearchableAction.java

+ * with a no-op (the search node can proceed to process the term vector request). The action may trigger an external refresh
+ * to ensure the search shards are up to date before returning the no-op.
+ */
+public class TransportEnsureDocsSearchableAction extends TransportSingleShardAction<


Yes, I would prefer that too.

fcofdez · 2025-07-02T20:00:55Z

server/src/main/java/org/elasticsearch/action/termvectors/TransportTermVectorsAction.java

-        );
+            );
+        if (iterator == null) {
+            return null;


I think that it'll execute the request in the receiving node. I think that we should return an empty iterator instead.

fcofdez · 2025-07-02T20:01:50Z

server/src/main/java/org/elasticsearch/index/engine/Engine.java

@@ -902,6 +902,10 @@ protected final GetResult getFromSearcher(Get get, Engine.Searcher searcher, boo
        }
    }

+    public boolean isDocumentInLiveVersionMap(BytesRef uid) {


maybe it's better if we name this isDocumentSearchable? more generic and less tied to implementation details. And +1 to add more docs.

kingherc self-assigned this Jun 24, 2025

elasticsearchmachine added v9.1.0 serverless-linked Added by automation, don't add manually labels Jun 24, 2025

kingherc force-pushed the non-issue/ES-12112-termvectors branch 3 times, most recently from 59876ce to dd19b4c Compare June 24, 2025 11:49

elasticsearchmachine added v9.2.0 and removed v9.1.0 labels Jun 26, 2025

kingherc force-pushed the non-issue/ES-12112-termvectors branch from dd19b4c to ab4e1ed Compare June 30, 2025 11:01

kingherc force-pushed the non-issue/ES-12112-termvectors branch from 82434e3 to e3b6a4b Compare July 1, 2025 15:23

kingherc marked this pull request as ready for review July 1, 2025 17:05

kingherc requested review from fcofdez and tlrx July 1, 2025 17:06

tlrx reviewed Jul 2, 2025

View reviewed changes

fcofdez reviewed Jul 2, 2025

View reviewed changes

Term vector API on stateless search nodes #129902

Are you sure you want to change the base?

Term vector API on stateless search nodes #129902

Uh oh!

Conversation

kingherc commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 1, 2025

Uh oh!

elasticsearchmachine commented Jul 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fcofdez left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kingherc commented Jun 24, 2025 •

edited

Loading