ESQL: Compute engine support for tagged queries #128521
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Begins adding support for running "tagged queries" to the compute engine. Here, it's just the `LuceneSourceOperator` because that's useful and contained. Example time! Say you are running: ``` FROM foo | STATS MAX(v) BY ROUND_TO(g, 0, 100, 1000, 100000) ``` It's *often* faster to run this as four queries: * The docs that round to `0` * The docs that round to `100` * The docs that round to `1000` * The docs that round to `100000` This creates an ESQL operator that can run these queries, one after the other and attach those tags. Aggs uses this trick and it's *way* faster when it can push down count queries, but it's still faster when it pushes doc loading things. This implementation in `LuceneSourceOperator` is quite similar to the doc loading version in _search. I don't have performance measurements yet because I haven't plugged this into the language. In _search we call this `filter-by-filter` and enable it when each group averages to more than 5000 documents and when there isn't an `_doc_count` field. It's faster in those cases not to push. I expect we'll be pretty similar.
nik9000
added
>non-issue
auto-backport
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
elasticsearchmachine
added
the
Team:Analytics
|
This should also work well for things like: With an extension to this PR that enables this behavior for |
|
The test failure was quite real - it came from attempting to reuse the scorer from one slice with a different query. I'll push a fix. |
nik9000
commented
May 28, 2025
| if (currentScorer == null || currentScorer.leafReaderContext() != leaf) { | ||
| if (currentScorer == null // First time | ||
| || currentScorer.leafReaderContext() != leaf // Moved to a new leaf | ||
| || currentScorer.weight != currentSlice.weight() // Moved to a new query |
There was a problem hiding this comment.
This last bit of the if statement took most of a day to figure out I needed. But tests caught it.
| /** | ||
| * Tags to add to the data returned by this query. | ||
| */ | ||
| List<Object> tags() { |
There was a problem hiding this comment.
I'm not entirely sure Object is the right thing. It works, but we might want Supplier<Block> or something more specific. But for now this is good enough.
There was a problem hiding this comment.
Yes, the object list can provide a better debugging message, but the block supplier might be better; otherwise, we would need to provide the exact boxed type for numeric values.
There was a problem hiding this comment.
Yeah, the getting the boxed type perfect could be tricky. Suppliers are quite explicit. Let's keep it as is for now and rework when we find a rough edge.
dnhatn
approved these changes
May 29, 2025
| @@ -121,6 +120,9 @@ protected Page getCheckedOutput() throws IOException { | |||
| if (scorer == null) { | |||
| remainingDocs = 0; | |||
| } else { | |||
| if (scorer.tags().isEmpty() == false) { | |||
There was a problem hiding this comment.
I think we can leverage this and min/max later too.
Right! I tried to do the next bit and it got big so I put that down. |
💔 Backport failed
You can use sqren/backport to manually backport by running |
nik9000
added a commit
to nik9000/elasticsearch
that referenced
this pull request
May 29, 2025
Begins adding support for running "tagged queries" to the compute engine. Here, it's just the `LuceneSourceOperator` because that's useful and contained. Example time! Say you are running: ``` FROM foo | STATS MAX(v) BY ROUND_TO(g, 0, 100, 1000, 100000) ``` It's *often* faster to run this as four queries: * The docs that round to `0` * The docs that round to `100` * The docs that round to `1000` * The docs that round to `100000` This creates an ESQL operator that can run these queries, one after the other and attach those tags. Aggs uses this trick and it's *way* faster when it can push down count queries, but it's still faster when it pushes doc loading things. This implementation in `LuceneSourceOperator` is quite similar to the doc loading version in _search. I don't have performance measurements yet because I haven't plugged this into the language. In _search we call this `filter-by-filter` and enable it when each group averages to more than 5000 documents and when there isn't an `_doc_count` field. It's faster in those cases not to push. I expect we'll be pretty similar.
|
backport: #128638 |
|
Backported with #128638 |
joshua-adams-1
pushed a commit
to joshua-adams-1/elasticsearch
that referenced
this pull request
Jun 3, 2025
Begins adding support for running "tagged queries" to the compute engine. Here, it's just the `LuceneSourceOperator` because that's useful and contained. Example time! Say you are running: ``` FROM foo | STATS MAX(v) BY ROUND_TO(g, 0, 100, 1000, 100000) ``` It's *often* faster to run this as four queries: * The docs that round to `0` * The docs that round to `100` * The docs that round to `1000` * The docs that round to `100000` This creates an ESQL operator that can run these queries, one after the other and attach those tags. Aggs uses this trick and it's *way* faster when it can push down count queries, but it's still faster when it pushes doc loading things. This implementation in `LuceneSourceOperator` is quite similar to the doc loading version in _search. I don't have performance measurements yet because I haven't plugged this into the language. In _search we call this `filter-by-filter` and enable it when each group averages to more than 5000 documents and when there isn't an `_doc_count` field. It's faster in those cases not to push. I expect we'll be pretty similar.
Samiul-TheSoccerFan
pushed a commit
to Samiul-TheSoccerFan/elasticsearch
that referenced
this pull request
Jun 5, 2025
Begins adding support for running "tagged queries" to the compute engine. Here, it's just the `LuceneSourceOperator` because that's useful and contained. Example time! Say you are running: ``` FROM foo | STATS MAX(v) BY ROUND_TO(g, 0, 100, 1000, 100000) ``` It's *often* faster to run this as four queries: * The docs that round to `0` * The docs that round to `100` * The docs that round to `1000` * The docs that round to `100000` This creates an ESQL operator that can run these queries, one after the other and attach those tags. Aggs uses this trick and it's *way* faster when it can push down count queries, but it's still faster when it pushes doc loading things. This implementation in `LuceneSourceOperator` is quite similar to the doc loading version in _search. I don't have performance measurements yet because I haven't plugged this into the language. In _search we call this `filter-by-filter` and enable it when each group averages to more than 5000 documents and when there isn't an `_doc_count` field. It's faster in those cases not to push. I expect we'll be pretty similar.
nik9000
added a commit
that referenced
this pull request
Jun 11, 2025
Begins adding support for running "tagged queries" to the compute engine. Here, it's just the `LuceneSourceOperator` because that's useful and contained. Example time! Say you are running: ``` FROM foo | STATS MAX(v) BY ROUND_TO(g, 0, 100, 1000, 100000) ``` It's *often* faster to run this as four queries: * The docs that round to `0` * The docs that round to `100` * The docs that round to `1000` * The docs that round to `100000` This creates an ESQL operator that can run these queries, one after the other and attach those tags. Aggs uses this trick and it's *way* faster when it can push down count queries, but it's still faster when it pushes doc loading things. This implementation in `LuceneSourceOperator` is quite similar to the doc loading version in _search. I don't have performance measurements yet because I haven't plugged this into the language. In _search we call this `filter-by-filter` and enable it when each group averages to more than 5000 documents and when there isn't an `_doc_count` field. It's faster in those cases not to push. I expect we'll be pretty similar.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Begins adding support for running "tagged queries" to the compute engine. Here, it's just the
LuceneSourceOperatorbecause that's useful and contained.Example time! Say you are running:
It's often faster to run this as four queries:
01001000100000This creates an ESQL operator that can run these queries, one after the other and attach those tags.
Aggs uses this trick and it's way faster when it can push down count queries, but it's still faster when it pushes doc loading things. This implementation in
LuceneSourceOperatoris quite similar to the doc loading version in _search.I don't have performance measurements yet because I haven't plugged this into the language. In _search we call this
filter-by-filterand enable it when each group averages to more than 5000 documents and when there isn't an_doc_countfield. It's faster in those cases not to push. I expect we'll be pretty similar.