Add last_over_time

[dnhatn]

This PR introduces a time-series aggregation function that collects the last value of each time series within each grouping.

For example:

TS index 
| STATS sum(last_over_time(memory_usage)) BY cluster, bucket(@timestamp, 1minute)

This aggregate function should support all data types, but currently, it only supports int, long, float, and double. Support for additional data types will be added later. Another issue is the lack of unit test infra for time-series and time-series aggregators. I plan to address them soon.

[dnhatn]

I opened this PR for discussion. Perhaps we should implement last(field, @timestamp), where @timestamp is optional for regular aggregation, then delegate last_over_time to it, similar to how max_over_time delegates to max.

[kkrik-es]

Lemme see if I understand this. Should this be returning a boolean to indicate if we found the last value? Assuming we're scanning in reverse chrono order, the first match should stop iteration over the values in the bucket.

[dnhatn]

We don't have a mechanism to signal this. I renamed maybeCollect to collectValue to make it less confusing.

[kkrik-es]

Let's add a comment about this, seems like an important optimization - e.g. for aggs with TBUCKET(1 day), we may scan 1 value vs 1m values, iiuc?

[kkrik-es]

I opened this PR for discussion. Perhaps we should implement last(field, @timestamp), where @timestamp is optional for regular aggregation, then delegate last_over_time to it, similar to how max_over_time delegates to max.

Right, I wonder if the following query makes sense:

TS metrics | STATS last(last_over_time(foo)) BY TBUCKET(1hour), hostname 

which should be equivalent to

TS metrics | STATS last(foo) BY TBUCKET(1hour), hostname 

I can't think of other uses of last vs last_over_time, rate and the other {agg}_over_time functions return a value for the entire interval. But then, last_over_time would have to return the timestamp of the last value, to be used in the wrapping last function.

[dnhatn]

PromQL: last_over_time(range-vector): the most recent point value in the specified interval.

@kkrik-es Thank you for the feedback. I was considering implementing last_over_time to be equivalent to the last_over_time function in PromQL. I am not sure about the output of your proposal last(last_over_time).

[kkrik-es]

Right the last would calculate the last value of the per-tsid last values, e.g. per host. I don't see it in the listed functions, not sure if it's supported in PromQL.. @felixbarny too.

[kkrik-es]

Hey Nhat, is this blocked? Shall we just move fwd with what you have, maybe adding comments to revisit the logic and terminate early, if possible?

[dnhatn]

Hey Nhat, is this blocked? Shall we just move fwd with what you have, maybe adding comments to revisit the logic and terminate early, if possible?

@kkrik-es I added a comment in 9e57f6c. Can you take a look? Thanks.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[martijnvg]

LGTM 👍

[dnhatn]

@kkrik-es @martijnvg Thanks for reviews.