ES|QL - Allow full text functions to be used in STATS ... WHERE #125479

carlosdelest · 2025-03-24T11:03:47Z

Allows full text functions to be used in STATS ... WHERE:

FROM books 
| STATS c = count(*) where author:"tolkien"

Just filters work for now - we could expand this into BY for stats in a follow up.

Closes #125481

carlosdelest · 2025-03-24T11:23:53Z

...plugin/esql/compute/src/main/java/org/elasticsearch/compute/lucene/LuceneQueryEvaluator.java

@@ -54,6 +54,7 @@ public record ShardConfig(Query query, IndexSearcher searcher) {}
    private final List<ShardState> perShardState;

    protected LuceneQueryEvaluator(BlockFactory blockFactory, ShardConfig[] shards) {
+        assert shards.length > 0 : "LuceneQueryEvaluator expects shard configs";


This will help detect non-pushable expressions that need to be created with ShardConfig information

elasticsearchmachine · 2025-03-24T13:54:45Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-03-24T13:54:45Z

Pinging @elastic/kibana-esql (ES|QL-ui)

bpintea

I think it looks good, but was wondering about an if'ace update.

bpintea · 2025-03-25T16:46:20Z

...rc/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/FullTextFunction.java

+        }
+        for (Expression child : expression.children()) {
+            if (child instanceof FullTextFunction ftf) {
+                failures.add(


Nit: we could maybe add a method that builds the failure message and reuse it above as well.

bpintea · 2025-03-25T17:24:06Z

...ugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/PhysicalOperationProviders.java

@@ -21,6 +23,7 @@ interface PhysicalOperationProviders {
    PhysicalOperation groupingPhysicalOperation(
        AggregateExec aggregateExec,
        PhysicalOperation source,
-        LocalExecutionPlannerContext context
+        LocalExecutionPlannerContext context,
+        List<EsPhysicalOperationProviders.ShardContext> shardContexts


It's feels a bit odd to provide an implementation piece of EsPhysicalOperationProviders (in the list) to the interface, given that actually EsPhysicalOperationProviders, which implements this if'ace, already has the list of shardContexts. I wonder if we could embbed it into the LocalExecutionPlannerContext itself, since it's local execution specific anyways.
Maybe it should be passed it to LocalExecutionPlaner#plan from ComputeService#runCompute, where the list of shard contexts is already available?

That makes sense - I did that change in 2a8e987, and some follow up work in c842f14

elasticsearchmachine · 2025-03-26T07:40:59Z

Hi @carlosdelest, I've created a changelog YAML for you.

…-search-functions-stats # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

…ch-functions-stats' into enhancement/esql-text-search-functions-stats

fang-xing-esql

Thank you @carlosdelest! I added a couple of comments related to the test coverage and messages.

fang-xing-esql · 2025-03-26T23:44:54Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/match-function.csv-spec

+
+from books 
+| where length(title) > 40 
+| stats c = count(*) where match(title, "Lord")


Here are some suggestions on the test coverages that I can think of, these also apply to the other full text functions and operators:

Can we have some tests with a bit more complicated predicates in the where clause under stats? For example some combinations of and, or and not, with functions that can or cannot be pushed down to Lucent? Perhaps borrow some queries from the existing tests where the where clause is not under stats.

I wonder how score works with where under stats, can we have some tests to capture how it behaves?

Can we have some tests to cover multiple aggregate functions with where clause under stats? The predicates for each aggregation can have some overlaps, we have some optimization rules to deal with overlapped predicates under stats.

Can we have some test with aggregation and grouping(BY) with full text functions under where clause?

Add some full text functions with options for the completeness.

Can we have some tests with a bit more complicated predicates in the where clause under stats? For example some combinations of and, or and not, with functions that can or cannot be pushed down to Lucent? Perhaps borrow some queries from the existing tests where the where clause is not under stats.
Can we have some tests to cover multiple aggregate functions with where clause under stats? The predicates for each aggregation can have some overlaps, we have some optimization rules to deal with overlapped predicates under stats.
Can we have some test with aggregation and grouping(BY) with full text functions under where clause?
Add some full text functions with options for the completeness.

I added testing in 3963fd3, hopefully that works!

I wonder how score works with where under stats, can we have some tests to capture how it behaves?

It affects scoring as well. I'm thinking that

FROM my_index METADATA _score | WHERE match(field, "query") | STATS c = AVG(_score)

is the same as

FROM my_index METADATA _score | STATS c = AVG(_score) WHERE match(field, "query")

so using a FTF in a STATS WHERE clause affects scoring as well. I think it's better for consistency, but happy to discuss with the team.

fang-xing-esql · 2025-03-27T00:49:08Z

...rc/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/FullTextFunction.java

        } else {
            plan.forEachExpression(FullTextFunction.class, ftf -> {
-                failures.add(fail(ftf, "[{}] {} is only supported in WHERE commands", ftf.functionName(), ftf.functionType()));
+                failures.add(
+                    fail(ftf, "[{}] {} is only supported in WHERE and STATS ... WHERE commands", ftf.functionName(), ftf.functionType())


STATS ... WHERE commands looks a bit weird to me, especially the ..., but I also had hard time thinking of a good way to describe this situation, some candidates that I can think of are:

{} is only supported in WHERE commands ===> implies standalone or nested WHERE command {} is only supported as filters in WHERE commands {} is only supported as predicates in WHERE commands {} is only supported in WHERE commands and WHERE commands under STATS commands {} is only supported as filters in WHERE and STATS commands {} is only supported as predicates in WHERE and STATS commands

is only supported as filters in WHERE and STATS commands works for me - changed it in 02f8748

# Conflicts: # x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/lucene/LuceneQueryEvaluator.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/180_match_operator.yml

…-search-functions-stats # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalPhysicalPlanOptimizerTests.java

carlosdelest · 2025-05-05T14:02:13Z

@fang-xing-esql @bpintea I added some testing to this PR that had fallen off my priority list for a while. Can you please review it?

Add full text functions usage in STATS.. WHERE

5f4eff5

elasticsearchmachine added the v9.1.0 label Mar 24, 2025

carlosdelest added >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL ES|QL-ui Impacts ES|QL UI Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch and removed v9.1.0 labels Mar 24, 2025

Fix test

748e576

carlosdelest commented Mar 24, 2025

View reviewed changes

carlosdelest added 2 commits March 24, 2025 12:36

Skip rest compat test

dc1dbe6

Fix test error message

3117d50

carlosdelest marked this pull request as ready for review March 24, 2025 13:54

elasticsearchmachine removed the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Mar 24, 2025

carlosdelest requested review from tteofili, ioanatia, ChrisHegarty, fang-xing-esql, svilen-mihaylov-elastic and afoucret March 24, 2025 13:54

bpintea reviewed Mar 25, 2025

View reviewed changes

carlosdelest added 2 commits March 26, 2025 08:26

Remove match operator check

3a753d2

Included ShardContext list into LocalExecutionPlannerContext

2a8e987

carlosdelest added auto-backport Automatically create backport pull requests when merged v8.19.0 v9.1.0 labels Mar 26, 2025

Update docs/changelog/125479.yaml

c99d373

github-actions bot deployed to docs-preview March 26, 2025 07:41 View deployment

carlosdelest added 2 commits March 26, 2025 08:41

Merge remote-tracking branch 'origin/main' into enhancement/esql-text…

2cc6898

…-search-functions-stats # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

Merge remote-tracking branch 'carlosdelest/enhancement/esql-text-sear…

425f146

…ch-functions-stats' into enhancement/esql-text-search-functions-stats

github-actions bot deployed to docs-preview March 26, 2025 07:42 View deployment

Add shard context information to aggregate filters

c842f14

github-actions bot deployed to docs-preview March 26, 2025 08:36 View deployment

fang-xing-esql reviewed Mar 27, 2025

View reviewed changes

carlosdelest added 5 commits April 28, 2025 17:20

Allow FTFs in WHERE clauses but not in grouping clauses

dfce035

Add CSV tests for STATS

3963fd3

Add stats scores test

85a2374

github-actions bot deployed to docs-preview May 5, 2025 11:09 View deployment

Fix capabilities

351d1ff

github-actions bot deployed to docs-preview May 5, 2025 12:57 View deployment

carlosdelest added 2 commits May 5, 2025 15:54

Fix test

52dd7b9

Changed error message

02f8748

github-actions bot deployed to docs-preview May 5, 2025 13:56 View deployment

[CI] Auto commit changes from spotless

f9bf170

github-actions bot deployed to docs-preview May 5, 2025 14:04 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ES|QL - Allow full text functions to be used in STATS ... WHERE #125479

ES|QL - Allow full text functions to be used in STATS ... WHERE #125479

carlosdelest commented Mar 24, 2025 •

edited

Loading

carlosdelest Mar 24, 2025

elasticsearchmachine commented Mar 24, 2025

elasticsearchmachine commented Mar 24, 2025

bpintea left a comment

bpintea Mar 25, 2025

bpintea Mar 25, 2025

carlosdelest Mar 26, 2025 •

edited

Loading

elasticsearchmachine commented Mar 26, 2025

fang-xing-esql left a comment

fang-xing-esql Mar 26, 2025

carlosdelest May 5, 2025

fang-xing-esql Mar 27, 2025

carlosdelest May 5, 2025

carlosdelest commented May 5, 2025

ES|QL - Allow full text functions to be used in STATS ... WHERE #125479

Are you sure you want to change the base?

ES|QL - Allow full text functions to be used in STATS ... WHERE #125479

Conversation

carlosdelest commented Mar 24, 2025 • edited Loading

carlosdelest Mar 24, 2025

Choose a reason for hiding this comment

elasticsearchmachine commented Mar 24, 2025

elasticsearchmachine commented Mar 24, 2025

bpintea left a comment

Choose a reason for hiding this comment

bpintea Mar 25, 2025

Choose a reason for hiding this comment

bpintea Mar 25, 2025

Choose a reason for hiding this comment

carlosdelest Mar 26, 2025 • edited Loading

Choose a reason for hiding this comment

elasticsearchmachine commented Mar 26, 2025

fang-xing-esql left a comment

Choose a reason for hiding this comment

fang-xing-esql Mar 26, 2025

Choose a reason for hiding this comment

carlosdelest May 5, 2025

Choose a reason for hiding this comment

fang-xing-esql Mar 27, 2025

Choose a reason for hiding this comment

carlosdelest May 5, 2025

Choose a reason for hiding this comment

carlosdelest commented May 5, 2025

carlosdelest commented Mar 24, 2025 •

edited

Loading

carlosdelest Mar 26, 2025 •

edited

Loading