Skip to content

Run coordinating can_match in field-caps #127734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 6, 2025

Conversation

dnhatn
Copy link
Member

@dnhatn dnhatn commented May 5, 2025

Currently, we don't run the coordinating can_match to skip unmatched shards in field-caps. Most of the time, this is fine, but the current field-caps fails when the target shards are unavailable, even if they don't match the index filter. This change integrates the coordinating can_match into field-caps to prevent failures in such cases.

@dnhatn dnhatn changed the title Run coordinator can_match in field-caps Run coordinating can_match in field-caps May 5, 2025
@elasticsearchmachine
Copy link
Collaborator

Hi @dnhatn, I've created a changelog YAML for you.

@dnhatn dnhatn added the :Search Foundations/Search Catch all for Search Foundations label May 5, 2025
@dnhatn dnhatn requested review from nik9000, smalyshev and quux00 May 5, 2025 23:57
@dnhatn dnhatn marked this pull request as ready for review May 5, 2025 23:58
nodeToShards.computeIfAbsent(shard.currentNodeId(), node -> new ArrayList<>()).add(shard);
boolean canMatch = true;
final ShardId shardId = shardIt.shardId();
if (indexFilter != null && indexFilter instanceof MatchAllQueryBuilder == false) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use CanMatchPreFilterSearchPhase to avoid duplicates, but this approach is simpler.

@elasticsearchmachine elasticsearchmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch labels May 6, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search-foundations (Team:Search Foundations)

@dnhatn
Copy link
Member Author

dnhatn commented May 6, 2025

I discussed this with Nik via another channel. We should explore two alternative options to improve resilience due to shard unavailability:

  1. Decompress the mappings on the coordinator to eliminate reliance on shard availability. However, this approach could become more complex when an index filter is provided.

  2. Allow partial_results in field-caps for ES|QL calls.

That said, this PR should help avoid accessing the frozen tier entirely when it doesn't match the query's time range filter.

@dnhatn dnhatn added the auto-backport Automatically create backport pull requests when merged label May 6, 2025
@dnhatn dnhatn merged commit 73f5125 into elastic:main May 6, 2025
17 checks passed
@dnhatn dnhatn deleted the can-match-field-caps branch May 6, 2025 16:52
@elasticsearchmachine
Copy link
Collaborator

💔 Backport failed

Status Branch Result
8.19 Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 127734

parkertimmins pushed a commit to parkertimmins/elasticsearch that referenced this pull request May 7, 2025
Currently, we don't run the coordinating can_match to skip unmatched 
shards in field-caps. Most of the time, this is fine, but the current 
field-caps fails when the target shards are unavailable, even if they
don't match the index filter. This change integrates the coordinating
can_match into field-caps to prevent failures in such cases.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/ES|QL AKA ESQL auto-backport Automatically create backport pull requests when merged backport pending >enhancement :Search Foundations/Search Catch all for Search Foundations Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch v8.19.0 v9.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants