Skip to content

Allow Setting a Minimum Shard Size for Rollover Policies #85054

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
original-brownbear opened this issue Mar 17, 2022 · 4 comments
Closed

Allow Setting a Minimum Shard Size for Rollover Policies #85054

original-brownbear opened this issue Mar 17, 2022 · 4 comments
Labels
:Data Management/ILM+SLM Index and Snapshot lifecycle management >enhancement Team:Data Management Meta label for data/management team

Comments

@original-brownbear
Copy link
Member

Description

Currently ILM rollover policies allow setting maximum values on age and size of indices/shards that trigger rollovers.
This works well as far as managing the lifecycle of indices that experience non-trivial write load goes.
When managing a large number of indices of varying write-load though, setting maximums only can lead to a lot of rolling over of trivially sized indices that contributes little to moving data volume to colder tiers but costs a lot in terms of the overhead of managing more indices.

Could we add a minimum size requirement for rollovers to the rollover policy as well to add a threshold below which we do not roll over? This way we could run the same policy for data streams that experience high and low write load (e.g. logging data streams for heterogeneous producers) in a way that allows for daily rollover for high volume producers that produce tens of GB a day and avoid the needless overhead of rolling over indices that only see an influx of tens of MB daily.

cc @jpountz

@original-brownbear original-brownbear added >enhancement :Data Management/ILM+SLM Index and Snapshot lifecycle management team-discuss needs:triage Requires assignment of a team area label labels Mar 17, 2022
@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Mar 17, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@andreidan
Copy link
Contributor

Thanks for opening this Armin. There's an in-progress PR on this #83345
// cc @joegallo

@Alphayeeeet
Copy link

@original-brownbear I believe this issue has been implemented by now? Could this issue then be closed?

@original-brownbear
Copy link
Member Author

Oh right this has been dealt with :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/ILM+SLM Index and Snapshot lifecycle management >enhancement Team:Data Management Meta label for data/management team
Projects
None yet
Development

No branches or pull requests

5 participants