Skip to content

EQL: consider applying tier filters also to field_caps #126467

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
luigidellaquila opened this issue Apr 8, 2025 · 1 comment
Open

EQL: consider applying tier filters also to field_caps #126467

luigidellaquila opened this issue Apr 8, 2025 · 1 comment
Labels
:Analytics/EQL EQL querying >non-issue Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)

Comments

@luigidellaquila
Copy link
Contributor

luigidellaquila commented Apr 8, 2025

In some scenarios, the user could expect that if a query only involves data from a specific tier, it doesn't consider other tiers at all, not even for planning purposes.

This is not true today, eg. if a foo field is declared as keyword in the hot tier indices and as text in the frozen tier indices , even if the query has

"filter": {
    "term": { "_tier": "data_hot" }
  }

accessing that field in the query will still lead to conflicts, eg.

Cannot use field [foo] due to ambiguities being mapped as [2] incompatible types

This is due to the fact that field_caps does not take filters into consideration, and the planning is executed base on the mappings of all the indices.

This is by design, but sometimes it's confusing, so we could consider reviewing it or providing an option for a different default.

For completeness, changing this behavior could lead to errors on existing queries:

  • suppose you have N indices idx1 ... idxN
  • idx1 has a field called foo, other indices don't
  • a query like
    GET /idx*/_eql/search
    {
      "query": "any where foo == 10 or bar == 20",
      "filter": {
        "term": { "_tier": "data_hot" }
      }
    }
    
    succeeds as long as idx1 is on the hot tier
  • as soon as idx1 is moved to the frozen tier, with current defaults the query keeps working fine; if we change the defaults, completely ignoring the frozen tier, the query will start failing.
@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Apr 8, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/EQL EQL querying >non-issue Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)
Projects
None yet
Development

No branches or pull requests

2 participants