[ES|QL] Date nanos implicit casting in union types

[fang-xing-esql]

Resolves #110009

The goal of this PR is to support union typed fields that reference both date and date_nanos fields without explicit casting. Union typed(mixed with date and date_nanos) fields are casted to a common data type during Analyzer phase automatically. numeric or string typed fields are not within the scope of this PR.

Today, queries that reference union(mixed with date and date_nanos) typed fields like below either return nulls or fail, because they reference union typed fields without explicit casting. After this PR, they should not fail or return null.

mapping
curl -u elastic:password -X PUT "localhost:9200/sample_data_1?pretty" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "client.ip": {"type": "ip"},
      "message": {"type": "text"},
      "test_date": {"type": "date"},
      "event.duration": {"type": "long"}
    }
  }
}
'
curl -u elastic:password -X PUT "localhost:9200/sample_data_2?pretty" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "client.ip": {"type": "ip"},
      "message": {"type": "keyword"},
      "test_date": {"type": "date_nanos"},
      "event.duration": {"type": "integer"}
    }
  }
}
'
queries
+ curl -u elastic:password -X POST 'localhost:9200/_query?format=txt&pretty' -H 'Content-Type: application/json' '-d
{
  "query": "FROM sample_data*"
}
'
       @timestamp       |   client.ip   |event.duration |    message    |   test_date   
------------------------+---------------+---------------+---------------+---------------
2023-10-23T12:15:03.360Z|172.21.2.162   |null           |null           |null           
2023-10-23T12:27:28.948Z|172.21.2.113   |null           |null           |null           
2023-10-23T13:33:34.937Z|172.21.0.5     |null           |null           |null           
2023-10-23T13:51:54.732Z|172.21.3.15    |null           |null           |null           
2023-10-23T13:52:55.015Z|172.21.3.15    |null           |null           |null           
2023-10-23T13:53:55.832Z|172.21.3.15    |null           |null           |null           
2023-10-23T13:55:01.543Z|172.21.3.15    |null           |null           |null           
2023-10-23T12:15:03.360Z|172.22.2.162   |null           |null           |null           
2023-10-23T12:27:28.948Z|172.22.2.113   |null           |null           |null           
2023-10-23T13:33:34.937Z|172.22.0.5     |null           |null           |null           
2023-10-23T13:51:54.732Z|172.22.3.15    |null           |null           |null           
2023-10-23T13:52:55.015Z|172.22.3.15    |null           |null           |null           
2023-10-23T13:53:55.832Z|172.22.3.15    |null           |null           |null           
2023-10-23T13:55:01.543Z|172.22.3.15    |null           |null           |null           

+ curl -u elastic:password -X POST 'localhost:9200/_query?format=txt&pretty' -H 'Content-Type: application/json' '-d
{
  "query": "FROM sample_data* | sort test_date"
}
'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "verification_exception",
        "reason" : "Found 1 problem\nline 1:26: Cannot use field [test_date] due to ambiguities being mapped as [2] incompatible types: [date_nanos] in [sample_data_2], [datetime] in [sample_data_1]"
      }
    ],
    "type" : "verification_exception",
    "reason" : "Found 1 problem\nline 1:26: Cannot use field [test_date] due to ambiguities being mapped as [2] incompatible types: [date_nanos] in [sample_data_2], [datetime] in [sample_data_1]"
  },
  "status" : 400
}

A field can be referenced in nearly all the processing commands, and this PR should cover all the possible places a (union typed)field can appear:

• evalCommand
• whereCommand
• keepCommand
• sortCommand
• dropCommand
• renameCommand
• statsCommand
• dissectCommand - works for keyword and text only, out of the scope of this PR
• grokCommand - works for keyword and text only, out of the scope of this PR
• enrichCommand - enrich does not support date_nanos yet
• mvExpandCommand
• joinCommand - lookup join on date_nanos field does not work today, out of the scope of this PR

[elasticsearchmachine]

Hi @fang-xing-esql, I've created a changelog YAML for you.

[fang-xing-esql]

@elasticmachine update branch

[costin]

Fang, can you please double check the status of this PR? Is it ready for review or still working on it?

[fang-xing-esql]

Fang, can you please double check the status of this PR? Is it ready for review or still working on it?

I'm trying to make enrich work with union typed fields, and this is the last piece that I want to figure out before making this ready for review.

[alex-spies]

Because we inject an EVAL here, there's a chance that we'll accidentally inject EVALs multiple times. Which could lead to broken plans, when an upstream EVAL already replaced the multi-typed field attribute by a reference attribute of the same name. Or, put the other way around, since we're replacing a field by another one with the same name, downstream plans may have invalid references now.

This could happen in case that the reference resolution manages to resolve the whole query plan, with multiple references to multi-typed fields, before we start adding Evals.

[alex-spies]

Normally, the way to update the expressions of a plan is to transform the expressions and use AttributeMap.resolve with an AttributeMap<Attribute> that contains which attribute should be replaced by which other attribute.

[craigtaverner]

Ran out of time, so I'll post the comments so far.

[fang-xing-esql]

Thanks for reviewing and brainstorming @craigtaverner @alex-spies ! Most of the comments have been addressed, here is a summary of the list of things that have been discussed:

1. The scope of this PR is reduced to do implicit casting for union typed fields with date and date_nanos only. It makes sense to take numeric types out, as there are a lot of questions on date_nanos already, date_nanos can be a pilot, so that we can collect feedbacks before supporting the other data types.
2. We don't do double casting, if there is explicit casting on union typed fields with date and date_nanos coded in the query, they won't be cast to date_nanos implicitly.
3. The implicit casting for union typed fields with date and date_nanos is taken out of the existing ImplicitCasting rule, a new rule is added after ResolveUnionTypes, to do implicit casting for the union typed fields(without explicit casting) that are not resolved by ResolveUnionTypes. We are not always able to replace a union typed field with a conversion function in place, sometimes an eval child plan needs to be added for the conversion functions, we need to traverse the plan tree, we can do it inside the ResolveUnionTypes rule, if it is preferred, however it will add some complexity to it, that's the main reason that I keep it as a separate rule in ImplicitCastingForUnionTypedFields.
4. Enrich doesn't support date_nanos as a match field(checked in resolveEnrich), it is out of the scope of this PR.
5. Lookup join does not work with date_nanos join keys yet, it is out of the scope of this PR, join is also a binary plan, in the future if we want to support it, we need some special handlings for binary plans when traversing and transforming the plan.

There are some comments that haven't been addressed yet:

1. Add a warning message when we do implicit casting to date_nanos, I think this is a good idea, I haven't added it as I want to clarify with the other folks whether casting to date_nanos is the final decision, as date_nanos have a smaller range of valid values compared to date. I'll reach out to relevant folks to collect feedbacks on which data type to cast to.
2. Performance. Filter and aggregations are the most interesting ones. Adding implicit casting may affect performance as it may prevent predicates to be pushed down into Lucene, and adding conversion functions to union typed fields add another layer of processing for operators that are not pushed down into Lucene(today we don't push down aggregations in a lot of cases). Although these queries that reference union typed fields fail at Verifier today, and allowing them to run is a step forward, if we can improve their performance by pushing down more into Lucene and reduce an extra layer of processing, that will be ideal. I'll investigate more about the possibility of unwrapping the conversion functions and pushing down more at data nodes.

[craigtaverner]

This looks much nicer to me! I have a few comments and questions, but otherwise I think it is good.

[craigtaverner]

Nice solution!

[craigtaverner]

So we allow through UnaryPlan and LookupJoin, but later we do nothing for LookupJoin. Is this a placeholder for a further update supporting LookupJoin?

[fang-xing-esql]

I'll add a comment here for LookupJoin, and remove the instanceof check, I was attempting to make date_nanos fields work as join keys, however it didn't workout easily for me. Thank you for creating a separate issue for it!

[craigtaverner]

Nice as an example, but is effectively dead code. I wonder what are the chances we'll change our mind on this one?

[craigtaverner]

It feels like we're making a decision prioritizing the LogsDb plans. But could we be causing trouble for other users? Or is the thinking that other users need to add an explicit cast? That seems a reasonable assumption.

[craigtaverner]

This code is very similar to code above, but that also took into account previous/existing field mappings. Why is that not needed here?

[fang-xing-esql]

The goal of castInvalidMappedFieldInFinalOutput is to add implicit casting to mixed date/date_nanos fields, that are not referenced by the query(there was no chance to do implicit casting for them if they are not referenced in any commands), however they are in the returned resultsets. For example, from index*. It only checks the top level plan's output, it doesn't traverse the entire tree, so it doesn't have a map to keep track of the casted fields map across all plans/subplans.

[alex-spies]

Discussed offline with @fang-xing-esql and @craigtaverner.

I think the proposed solution should work, it's just a little complex given that it needs to replicate a) some logic present in InsertFieldExtraction (inserting EVALs for implicitly cast fields just before they're used) and b) for passing on fields that users never explicitly dropped or kept into the final output correctly.

I suggested one more approach which we agreed to try before going forward with the Eval-injection approach. The alternative would turn invalid mapped fields of date/nanos type into multi typed fields per default, before any resolution step (multi typed fields in field attributes trigger automatic conversion on field extraction). I threw together a draft as a commit on top of this PR, which seems to still run most tests fine (all existing union type tests + most tests from this PR).

This approach still needs to special case for the situation where users specifically don't want to convert into nanos, but dates, i.e. when the query has datetime_nanos_field::datetime in some expression and the classical union type mechanism should kick in.