[ES|QL] Add suggested_cast

[limotova]

Adds a suggested_cast for unsupported fields based on original_types
for specific cases:

• when a field is mapped to datetime and date_nanos, suggests date_nanos
• when a field is mapped to aggregate_metric_double and other numerics, suggests aggregate_metric_double
• when a field is unsupported anywhere, suggests nothing
• suggests keyword in all other cases

For example, a field that is mapped to aggregate_metric_double in one index, but double in another will have a response like:

    {
      "name" : "agg_metric",
      "type" : "unsupported",
      "original_types" : [
        "aggregate_metric_double",
        "double"
      ],
      "suggested_cast" : "aggregate_metric_double"
    }

A field that is completely unsupported will have a response like:

    {
      "name" : "binary_field",
      "type" : "unsupported",
      "original_types" : [
        "binary"
      ]
    }

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[elasticsearchmachine]

Hi @limotova, I've created a changelog YAML for you.

[limotova]

Would it make sense to also add a suggestion to cast in the error message?
Like here so we have a message that reads more like:
Cannot use field [agg_metric] due to ambiguities being mapped as [2] incompatible types: [aggregate_metric_double] in [my-index], [double] in [my-index2]. Consider casting field to [aggregate_metric_double] ?

[elasticsearchmachine]

Pinging @elastic/kibana-esql (ES|QL-ui)

[nik9000]

Could you add an example response in the PR description? It helps folks scanning these things.

[nik9000]

I was wondering to myself "should this return a list?" and I think the answer is "no". At this point there's always one best choice. keyword is almost always an option though. Maybe it'd be nice to say ["date", "keyword"] for dates? Like, you have two choices? I don't think so.... But maybe? It's nice to know for the truly unsupported cases that you don't have any choices.

[kkrik-es]

Why not DATE_NANOS? This is orthogonal to aggregate_metric_double, but I thought we want to auto-cast (not even suggest) to nanos and error out if the conversion is not possible.

[nik9000]

There's still some up in the air about this one. DATE_NANOS would leave a lot of stuff out of range. I feel like it's pretty low commitment to put this here though.

[kkrik-es]

Is it an option to use one of the underlying metrics (e.g. max) in a mixed query? Or, we'd have to convert the e.g. double to aggregate_metric_double first, and then use the max from all of them (original and converted)? That'd be one expensive operation to read all values - it may be fine, if there's no better way to mix.

[nik9000]

Not really. Casting is only something we can do with one value at a time but these underlying metrics are things you'd get from an agg. I think. Right?

[limotova]

Yeah the underlying metrics would be accessed in specific aggregations.
Maybe when we implement implicit casting of aggregate_metric_double in aggregations instead of going double -> aggregate_metric_double -> submetric (double/int), we can do aggregate_metric_double -> submetric (double/int), and have no conversion for doubles/ints (or if the other mapping is a long or something convert that to the submetric type)?
(The count submetric is an int but the others are doubles)

[stratoula]

How can I cast to aggregate_metric_double ? Do we have a function for that already??
In case we want to utilize it in autocomplete which is the logic?

Is going to be

TO_xxx where xxx the suggested_cast value?

[nik9000]

Would it make sense to also add a suggestion to cast in the error message?

I'm cool with it. I wish we generated that error message lazily, but otherwise it's cool.

[alex-spies]

when a field is mapped to datetime and date_nanos, suggests datetime

Sorry for the drive by, but wanted to let you know: there's ongoing work (#123678) that will implicitly cast to date_nanos in this case, contrary to the suggestion, which may also become obsolete.

[nik9000]

Is going to be

TO_xxx where xxx the suggested_cast value?

Yeah, that exists. I figure folks will do <field>::AGGREGATE_METRIC_DOUBLE though.

@limotova, what do you think of an integration test that sets up indices with the conflicting types, does a FROM a,b and gets this back and then does a FROM a,b,c | EVAL f = f::<UPPER_CASE_SUGGESTED_TYPE>. That's how we expect folks to use this and I know it works now, but this integration test would catch if we break it.

Not sure we could do it for x - but maybe. Parameterized testing would make it easy to write. It probably takes a half second per combo... We have about 30 types. That'd be 900 cases.... so 450 seconds which is way too slow. But maybe I'm off? Still worth doing for some known combos I think. Ooh - it's probably half that because we don't care about order! Still 225 seconds is a lot.

[stratoula]

I figure folks will do ::AGGREGATE_METRIC_DOUBLE though

Sure this works for me too! As long as the cast type is valid!

[kkrik-es]

lgtm.. delegating to Nik to approve, since this is really ES|QL territory.

[limotova]

there's ongoing work (#123678) that will implicitly cast to date_nanos in this case, contrary to the suggestion, which may also become obsolete.

It sounds like I should change the datetime + date_nanos combo to suggest date_nanos instead?

integration test that sets up indices with the conflicting types
...
We have about 30 types. That'd be 900 cases

I can set up the integration test but I think it might be worth mentioning too that we might have 3+ types in a single request, so if we really wanted to go crazy there could be a lot more cases...
But I can set up a test that does all possible pairs

[nik9000]

There's still some up in the air about this one. DATE_NANOS would leave a lot of stuff out of range. I feel like it's pretty low commitment to put this here though.