Skip to content

Avoid sorted source for time_series aggs without rates #127033

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 22, 2025

Conversation

dnhatn
Copy link
Member

@dnhatn dnhatn commented Apr 17, 2025

With this change, TS index | STATS ... will be translated to FROM index METADATA _tsid | STATS ... to avoid emitting docs in _tsid and timestamp order, which is expensive. For example, this reduces the execution time of the below query with tsdb track from 50 seconds to 4.5 seconds.

TS tsdb 
| STATS sum(max_over_time(kubernetes.container.memory.usage.bytes)) BY bucket(@timestamp, 5minute)

@dnhatn dnhatn changed the title Use unsorted source for time_series aggs without rates Avoid sorted source for time_series aggs without rates Apr 20, 2025
@dnhatn dnhatn requested review from kkrik-es and martijnvg April 20, 2025 00:43
@dnhatn dnhatn added :StorageEngine/TSDB You know, for Metrics >non-issue labels Apr 20, 2025
@dnhatn dnhatn marked this pull request as ready for review April 20, 2025 00:44
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

@kkrik-es
Copy link
Contributor

To make sure I understand this, with this change the inner agg is still evaluated per tsid, but we can push this down to segments since we don't need global ordering per shard?

@dnhatn
Copy link
Member Author

dnhatn commented Apr 21, 2025

Yes, that's correct. We only need the global ordering per shard for rate aggregations to avoid holding all data points.

Copy link
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@dnhatn
Copy link
Member Author

dnhatn commented Apr 22, 2025

@kkrik-es @martijnvg Thanks!

@dnhatn dnhatn merged commit 09541c5 into elastic:main Apr 22, 2025
17 checks passed
@dnhatn dnhatn deleted the from-no-rates branch April 22, 2025 18:47
@dnhatn dnhatn mentioned this pull request Apr 27, 2025
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants