ESQL: Search results for wildcards and selectors return errors instead of empty results when indices are missing

[jbaiera]

If you search for an index using a wildcard pattern, and the pattern does not match any indices, you will receive a successful response with no results from the _search API, but an error from the _query API:

> GET /missing*/_search

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 0,
    "successful": 0,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": 0.0,
    "hits": []
  }
}

> POST /_query
> {"query": "FROM missing*"}

{
  "error": {
    "root_cause": [
      {
        "type": "verification_exception",
        "reason": "Found 1 problem\nline 1:1: Unknown index [missing*]"
      }
    ],
    "type": "verification_exception",
    "reason": "Found 1 problem\nline 1:1: Unknown index [missing*]"
  },
  "status": 400
}

This can also be seen when searching a data stream's failure store if the failure store does not exist yet (e.g. there have been no failures logged yet).

> GET /my_ds-1::failures/_search

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 0,
    "successful": 0,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": 0.0,
    "hits": []
  }
}

> POST /_query
> {"query": "FROM my_ds-1::failures"}

{
  "error": {
    "root_cause": [
      {
        "type": "verification_exception",
        "reason": "Found 1 problem\nline 1:1: Unknown index [my_ds-1::failures]"
      }
    ],
    "type": "verification_exception",
    "reason": "Found 1 problem\nline 1:1: Unknown index [my_ds-1::failures]"
  },
  "status": 400
}

Executing the query operation with error tracing enabled on the latest mainline version reveals the following stack traces:

org.elasticsearch.xpack.esql.VerificationException: Found 1 problem
line 1:1: Unknown index [missing*]
	at org.elasticsearch.xpack.esql.analysis.Analyzer.verify(Analyzer.java:196)
	at org.elasticsearch.xpack.esql.analysis.Analyzer.analyze(Analyzer.java:190)
	at org.elasticsearch.xpack.esql.session.EsqlSession.lambda$analyzedPlan$12(EsqlSession.java:342)
	at org.elasticsearch.xpack.esql.session.EsqlSession.analyzeAndMaybeRetry(EsqlSession.java:555)
	at org.elasticsearch.xpack.esql.session.EsqlSession.lambda$analyzedPlan$21(EsqlSession.java:387)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:394)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:314)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:343)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)
	at org.elasticsearch.xpack.esql.session.EsqlSession.lambda$analyzedPlan$20(EsqlSession.java:384)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:394)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:314)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:343)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)
	at org.elasticsearch.xpack.esql.session.EsqlSession.lambda$preAnalyzeIndices$27(EsqlSession.java:489)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:233)
	at org.elasticsearch.xpack.esql.session.IndexResolver.lambda$resolveAsMergedMapping$0(IndexResolver.java:91)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:413)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:203)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:197)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:350)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:413)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:111)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecuteForked(TransportFieldCapabilitiesAction.java:165)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.lambda$executeRequest$1(TransportFieldCapabilitiesAction.java:136)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:34)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[alex-spies]

Thanks @jbaiera . I think we've discussed this in the team before and pondered whether this inconsistency with _search is okay or should be considered a bug. IMHO it isn't a bug.

If no index matches the FROM pattern, we cannot determine the types in a query like FROM missing* | EVAL x = 2*y; but because ESQL's results are tabular, there should be an x column with some type. I think it's not clear what to do in such cases, and it becomes less clear when the x column is used in other downstream commands. We probably shouldn't fail with "no such field [y]" either, because that'd be misleading - y could exist when using a wider index pattern and its absence is not the root cause of the failed query.

I agree, however, that the error message is misleading for FROM my_ds-1::failures if there were no failures yet - it suggests a typo in the index name rather than the index just being effectively empty.

[jbaiera]

Hmm, that is indeed a tricky problem to solve. If there were no EVAL x = 2 * y on the end of the query, I would assume this query would be feasible to run. Any eval statement could use variables that don't have definitions provided by the query up until that point. I don't think we should treat an empty wildcard pattern as the only problem with that kind of query.

Granted, an empty wildcard means no schema would be returned in the response either. Is the lack of any column list in the final result a problem?

[alex-spies]

Granted, an empty wildcard means no schema would be returned in the response either. Is the lack of any column list in the final result a problem?

An empty column list can work, although we have identified issues with our current approach that give rise to bugs in edge cases that we haven't solved yet.

In general, an empty column list will be treated as if the whole result was empty - meaning, no rows either. This can be seen when intentionally dropping all columns: FROM idx | KEEP field | DROP field. Unfortunately, there are situations where the number of rows matter (even when there are no columns), and we have bugs like this one: #120272

That said, I think we should be able to return an empty result for FROM my_ds-1::failures in case this index just doesn't exist yet, as that'd be a better UX than users not being able to distinguish between a typo in the index name and just no results yet.

Or can the schema of the failure store maybe be known even when it doesn't exist yet?

[jbaiera]

Or can the schema of the failure store maybe be known even when it doesn't exist yet?

Technically the failure store schema is controlled by Elasticsearch, so it's possible that we could render a schema even for the failure indices that do not exist yet.

This rendered schema would need to be only used if the failure store has no backing indices in it. Users can add arbitrary indices to the failure store for operational reasons, so we make no guarantees about the shape or collection of mappings in the failure store when reading it. We only ensure that we write failure store data to indices that Elasticsearch creates itself.