Skip to content

ESQL: Search results for wildcards and selectors return errors instead of empty results when indices are missing #125995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
jbaiera opened this issue Mar 31, 2025 · 5 comments
Open

ESQL: Search results for wildcards and selectors return errors instead of empty results when indices are missing #125995

jbaiera opened this issue Mar 31, 2025 · 5 comments
Labels
:Analytics/ES|QL AKA ESQL >bug Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)

Comments

@jbaiera
Copy link
Member

jbaiera commented Mar 31, 2025

If you search for an index using a wildcard pattern, and the pattern does not match any indices, you will receive a successful response with no results from the _search API, but an error from the _query API:

> GET /missing*/_search

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 0,
    "successful": 0,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": 0.0,
    "hits": []
  }
}

> POST /_query
> {"query": "FROM missing*"}

{
  "error": {
    "root_cause": [
      {
        "type": "verification_exception",
        "reason": "Found 1 problem\nline 1:1: Unknown index [missing*]"
      }
    ],
    "type": "verification_exception",
    "reason": "Found 1 problem\nline 1:1: Unknown index [missing*]"
  },
  "status": 400
}

This can also be seen when searching a data stream's failure store if the failure store does not exist yet (e.g. there have been no failures logged yet).

> GET /my_ds-1::failures/_search

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 0,
    "successful": 0,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": 0.0,
    "hits": []
  }
}

> POST /_query
> {"query": "FROM my_ds-1::failures"}

{
  "error": {
    "root_cause": [
      {
        "type": "verification_exception",
        "reason": "Found 1 problem\nline 1:1: Unknown index [my_ds-1::failures]"
      }
    ],
    "type": "verification_exception",
    "reason": "Found 1 problem\nline 1:1: Unknown index [my_ds-1::failures]"
  },
  "status": 400
}

Executing the query operation with error tracing enabled on the latest mainline version reveals the following stack traces:

org.elasticsearch.xpack.esql.VerificationException: Found 1 problem
line 1:1: Unknown index [missing*]
	at org.elasticsearch.xpack.esql.analysis.Analyzer.verify(Analyzer.java:196)
	at org.elasticsearch.xpack.esql.analysis.Analyzer.analyze(Analyzer.java:190)
	at org.elasticsearch.xpack.esql.session.EsqlSession.lambda$analyzedPlan$12(EsqlSession.java:342)
	at org.elasticsearch.xpack.esql.session.EsqlSession.analyzeAndMaybeRetry(EsqlSession.java:555)
	at org.elasticsearch.xpack.esql.session.EsqlSession.lambda$analyzedPlan$21(EsqlSession.java:387)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:394)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:314)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:343)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)
	at org.elasticsearch.xpack.esql.session.EsqlSession.lambda$analyzedPlan$20(EsqlSession.java:384)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:394)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:314)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:343)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)
	at org.elasticsearch.xpack.esql.session.EsqlSession.lambda$preAnalyzeIndices$27(EsqlSession.java:489)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:233)
	at org.elasticsearch.xpack.esql.session.IndexResolver.lambda$resolveAsMergedMapping$0(IndexResolver.java:91)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at [email protected]/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:413)
	at [email protected]/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:203)
	at [email protected]/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:197)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:350)
	at [email protected]/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:413)
	at [email protected]/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:111)
	at [email protected]/org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecuteForked(TransportFieldCapabilitiesAction.java:165)
	at [email protected]/org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.lambda$executeRequest$1(TransportFieldCapabilitiesAction.java:136)
	at [email protected]/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at [email protected]/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:34)
	at [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Mar 31, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@jbaiera jbaiera mentioned this issue Mar 31, 2025
Open
3 tasks
@alex-spies
Copy link
Contributor

alex-spies commented Apr 1, 2025 •
edited
Loading

Thanks @jbaiera . I think we've discussed this in the team before and pondered whether this inconsistency with _search is okay or should be considered a bug. IMHO it isn't a bug.

If no index matches the FROM pattern, we cannot determine the types in a query like FROM missing* | EVAL x = 2*y; but because ESQL's results are tabular, there should be an x column with some type. I think it's not clear what to do in such cases, and it becomes less clear when the x column is used in other downstream commands. We probably shouldn't fail with "no such field [y]" either, because that'd be misleading - y could exist when using a wider index pattern and its absence is not the root cause of the failed query.

I agree, however, that the error message is misleading for FROM my_ds-1::failures if there were no failures yet - it suggests a typo in the index name rather than the index just being effectively empty.

@jbaiera
Copy link
Member Author

jbaiera commented Apr 4, 2025

Hmm, that is indeed a tricky problem to solve. If there were no EVAL x = 2 * y on the end of the query, I would assume this query would be feasible to run. Any eval statement could use variables that don't have definitions provided by the query up until that point. I don't think we should treat an empty wildcard pattern as the only problem with that kind of query.

Granted, an empty wildcard means no schema would be returned in the response either. Is the lack of any column list in the final result a problem?

@alex-spies
Copy link
Contributor

Granted, an empty wildcard means no schema would be returned in the response either. Is the lack of any column list in the final result a problem?

An empty column list can work, although we have identified issues with our current approach that give rise to bugs in edge cases that we haven't solved yet.

In general, an empty column list will be treated as if the whole result was empty - meaning, no rows either. This can be seen when intentionally dropping all columns: FROM idx | KEEP field | DROP field. Unfortunately, there are situations where the number of rows matter (even when there are no columns), and we have bugs like this one: #120272

That said, I think we should be able to return an empty result for FROM my_ds-1::failures in case this index just doesn't exist yet, as that'd be a better UX than users not being able to distinguish between a typo in the index name and just no results yet.

Or can the schema of the failure store maybe be known even when it doesn't exist yet?

@jbaiera
Copy link
Member Author

jbaiera commented Apr 9, 2025

Or can the schema of the failure store maybe be known even when it doesn't exist yet?

Technically the failure store schema is controlled by Elasticsearch, so it's possible that we could render a schema even for the failure indices that do not exist yet.

This rendered schema would need to be only used if the failure store has no backing indices in it. Users can add arbitrary indices to the failure store for operational reasons, so we make no guarantees about the shape or collection of mappings in the failure store when reading it. We only ensure that we write failure store data to indices that Elasticsearch creates itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Analytics/ES|QL AKA ESQL >bug Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jbaiera @elasticsearchmachine @alex-spies