Skip to content

[DOCS] Sizing shards rewrite #120346

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 5 commits into from
Closed

[DOCS] Sizing shards rewrite #120346

wants to merge 5 commits into from
+22 −11

Conversation

thekofimensah
Copy link
Contributor

This is a really important internal document and is a question that many people struggle with. I’ve tried to make it as clear and direct as possible while packing in useful information right from the start. I’d appreciate it if someone could review it for accuracy and share any thoughts or suggestions on the approach.

@github-actions GitHub Actions
Copy link
Contributor

Documentation preview:

@elasticsearchmachine elasticsearchmachine added needs:triage Requires assignment of a team area label v9.0.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Jan 17, 2025
@tvernum tvernum added the >docs General docs changes label Jan 17, 2025
@elasticsearchmachine elasticsearchmachine added the Team:Docs Meta label for docs team label Jan 17, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-docs (Team:Docs)

@elasticsearchmachine elasticsearchmachine removed the needs:triage Requires assignment of a team area label label Jan 17, 2025
@tvernum tvernum added :Distributed Indexing/Distributed A catch all label for anything in the Distributed Indexing Area. Please avoid if you can. needs:triage Requires assignment of a team area label and removed Team:Docs Meta label for docs team needs:triage Requires assignment of a team area label labels Jan 17, 2025
@elasticsearchmachine elasticsearchmachine added Team:Docs Meta label for docs team Team:Distributed Indexing Meta label for Distributed Indexing team labels Jan 17, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

@leemthompo leemthompo changed the title Kofi sizing shards rewrite [DOCS] Sizing shards rewrite Jan 17, 2025
@leemthompo
Copy link
Contributor

A more detailed PR description would be helpful for reviewers. Something like:

Key Changes:

  • Added concrete shard sizing recommendations (10GB-50GB, <200M docs per shard)
  • Introduced new "What is a shard?" section explaining core concepts
  • Restructured introduction to emphasize balanced distribution and monitoring importance
  • Removed video links (why?)
  • Improved readability with clearer structure and bullet points

@thekofimensah
Copy link
Contributor Author

Thanks for the note @leemthompo . Will do that structure later on to make it easier for reviewers.

FYI, I'm confused about the youtube links. I didn't remove them and I couldn't find them in the original doc, so I don't know how I deleted them. Is there something I'm missing?

kingherc
kingherc reviewed Jan 20, 2025
View reviewed changes
Copy link
Contributor

@kingherc kingherc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Some comments.

@leemthompo
Copy link
Contributor

FYI, I'm confused about the youtube links. I didn't remove them and I couldn't find them in the original doc, so I don't know how I deleted them. Is there something I'm missing?

@thekofimensah You either removed them, or else you were working with an older version of the file somehow. Please restore those links unless you have a principled reason for removing them.

leemthompo
leemthompo reviewed Jan 29, 2025
View reviewed changes
Copy link
Contributor

@leemthompo leemthompo left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a nice update @thekofimensah ! Please address the following and we can get this merged today:

  • "What is a shard?" should be first section
  • Restore data streams context from original
  • Restore video URLs
  • Try make sizing paragraph a little less dense

@leemthompo leemthompo self-assigned this Jan 29, 2025
kingherc
kingherc reviewed Jan 29, 2025
View reviewed changes
Copy link
Contributor

@kingherc kingherc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes! A few more small comments now that I re-read it.

@thekofimensah
Changed order of sections to be more logical flow, readded the youtub…
4457385 
…e links, added data stream section

Added shard overhead and fixed replica description
@thekofimensah thekofimensah force-pushed the kofi-sizing-shards-rewrite branch from cb3709f to 4457385 Compare January 29, 2025 16:43
@thekofimensah thekofimensah force-pushed the kofi-sizing-shards-rewrite branch from d19c5c5 to 8d11421 Compare January 29, 2025 17:07
@kingherc
Copy link
Contributor

Awesome, thanks for the changes @thekofimensah ! It looks good to me! I'd like to add one more reviewer from the Coordination team, @DaveCTurner , who has contributed several of the parts of the file, mainly for proofreading. I'll then approve. Or I'll approve if I don't see more reviewing soon, as I'm happy with the current content.

@kingherc kingherc added the :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) label Jan 29, 2025
@kingherc kingherc requested a review from DaveCTurner January 29, 2025 17:40
@elasticsearchmachine elasticsearchmachine added the Team:Distributed Coordination Meta label for Distributed Coordination team label Jan 29, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

@thekofimensah
Copy link
Contributor Author

thekofimensah commented Jan 29, 2025 via email

@kingherc
Copy link
Contributor

Well I am not sure it will be so soon. Hopefully it's fine if we merge it after the freeze and/or backport it if needed. I was thinking of coming back to approve by Monday if I don't see anything in the meantime.

@leemthompo
Copy link
Contributor

Important

Elastic documentation is migrating to Markdown for version 9.0+. See the migration guide for details.

ℹ️ What's happening?

  • Starting January 29, we will start closing all unmerged documentation PRs targeting main/master
  • We're migrating from AsciiDoc to Markdown for 9.0+
  • 9.0 docs will be frozen from January 29 until February 20 2024
  • NOTE: PRs that include both code and documentation changes will remain open

What do I need to do?

For <=8.x docs:

  1. Rebase your PR to target the relevant 8.x branch instead
  2. The content can remain in AsciiDoc format

For 9.0+ docs:

Option 1:

  • Draft docs in Markdown
  • Once migration freeze ends, find the relevant page in the new docs system and use the edit options to submit your changes

Option 2:

💡 Need help?

  1. For Elasticians: Ask in #docs Slack channel
  2. For external contributors: Open an issue in elastic/docs-content

@thekofimensah
Copy link
Contributor Author

Hey @kingherc, any update on your approval of this?

kingherc
kingherc approved these changes Feb 28, 2025
View reviewed changes
Copy link
Contributor

@kingherc kingherc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

thekofimensah added a commit to thekofimensah/elasticsearch that referenced this pull request Mar 9, 2025
@thekofimensah
Edits copied over from elastic#120346
ba52258
thekofimensah added a commit to thekofimensah/elasticsearch that referenced this pull request Mar 9, 2025
@thekofimensah
Edits copied over from elastic#120346
b441133
thekofimensah added a commit to thekofimensah/elasticsearch that referenced this pull request Mar 9, 2025
@thekofimensah
Edits copied over from elastic#120346
5988536
thekofimensah added a commit to thekofimensah/elasticsearch that referenced this pull request Mar 9, 2025
@thekofimensah
Edits copied over from elastic#120346
fa21fc3
@thekofimensah thekofimensah mentioned this pull request Mar 9, 2025
Merged
georgewallace added a commit that referenced this pull request Apr 14, 2025
@thekofimensah @shainaraskas @georgewallace
[DOCS] Rewrite of sizing your shards-rebase (#124444)
a5abd1a 
* Edits copied over from #120346

* Update docs/reference/how-to/size-your-shards.asciidoc

Co-authored-by: shainaraskas <[email protected]>

* Improve Scanability

Co-authored-by: shainaraskas <[email protected]>

* Reduced what is a shard section for concision

* Adjusted title

* Add general and distribution sizing guidelines.

---------

Co-authored-by: shainaraskas <[email protected]>
Co-authored-by: George Wallace <[email protected]>
georgewallace added a commit to georgewallace/elasticsearch that referenced this pull request Apr 14, 2025
@thekofimensah @shainaraskas @georgewallace
[DOCS] Rewrite of sizing your shards-rebase (elastic#124444)
2e6dec0 
* Edits copied over from elastic#120346

* Update docs/reference/how-to/size-your-shards.asciidoc

Co-authored-by: shainaraskas <[email protected]>

* Improve Scanability

Co-authored-by: shainaraskas <[email protected]>

* Reduced what is a shard section for concision

* Adjusted title

* Add general and distribution sizing guidelines.

---------

Co-authored-by: shainaraskas <[email protected]>
Co-authored-by: George Wallace <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leemthompo leemthompo leemthompo left review comments

@kingherc kingherc kingherc approved these changes

@georgewallace georgewallace Awaiting requested review from georgewallace

@DaveCTurner DaveCTurner Awaiting requested review from DaveCTurner

Assignees

@leemthompo leemthompo

Labels
:Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) :Distributed Indexing/Distributed A catch all label for anything in the Distributed Indexing Area. Please avoid if you can. >docs General docs changes external-contributor Pull request authored by a developer outside the Elasticsearch team Team:Distributed Coordination Meta label for Distributed Coordination team Team:Distributed Indexing Meta label for Distributed Indexing team Team:Docs Meta label for docs team v9.1.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@thekofimensah @elasticsearchmachine @leemthompo @kingherc @tvernum