Add to allocation architecture guide #125328
Conversation
DiannaHohensee
added
>non-issue
:Distributed Coordination/Allocation
|
Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination) |
|
CC @JeremyDahlgren just fyi. |
| ### Indexes and Shards | ||
|
|
||
| Each index consists of a fixed number of primary shards. The number of primary shards cannot be changed for the lifetime of the index. Each | ||
| primary shard can have zero-to-many replicas used for data redundancy. The number of replicas per shard can be changed in runtime. Each |
There was a problem hiding this comment.
| primary shard can have zero-to-many replicas used for data redundancy. The number of replicas per shard can be changed in runtime. Each | |
| primary shard can have zero-to-many replicas used for data redundancy. The number of replicas per shard can be changed dynamically. Each |
(or maybe "at runtime")?
There was a problem hiding this comment.
Applied dynamically 👍
|
|
||
| Each index consists of a fixed number of primary shards. The number of primary shards cannot be changed for the lifetime of the index. Each | ||
| primary shard can have zero-to-many replicas used for data redundancy. The number of replicas per shard can be changed in runtime. Each | ||
| shard copy (primary or replica) can be in one of four states: |
There was a problem hiding this comment.
Maybe worth mentioning here that these states are org.elasticsearch.cluster.routing.ShardRoutingState the ones in the routing table (i.e. within the ClusterState) and the transitions between these states are part of the dance between data node and master node to reflect the lifecycle of a shard:
UNASSIGNED -> INITIALIZING happens when the master wants the data node to start creating this shard copy.
INITIALIZING -> STARTED happens when recovery is fully complete and the data node tells the master it's ready to serve requests.
STARTED -> RELOCATING happens when the master wants to initialize the node elsewhere.
A failure can take a shard in any state back to UNASSIGNED. Or the shard entry can be removed entirely from the cluster state. In either case, that tells the data node to stop whatever it is doing and shut down the shard.
Also IMO it'd be more discoverable to expand the Javadocs for ShardRoutingState with all this detail rather than hiding it away here.
There was a problem hiding this comment.
Thanks for the info. I was trying to shove Ievgen's brain dump info into here, but I wasn't that familiar with the relevant code. I agree, this information is too low-level. I pushed it down into ShardRoutingState in a separate PR because I want to add a reference to the new documentation in this PR.
I wasn't too sure where to push the state transitions. I guess we can prescribe how to transition states in the ShardRoutingState, too, but it's a little strange to describe how the class is used by other classes when ShardRoutingState doesn't enforce those transitions itself. Oh well 🤷♀️
| updated `RoutingTable`. The `RoutingTable` is part of the cluster state, so the master node updates the cluster state with the new | ||
| (incremental) desired shard allocation information. The updated cluster state is then published to the data nodes. Each data node will | ||
| observe any change in shard allocation related to that node and take action to achieve the new shard allocation by initiating creation of a | ||
| new empty shard, starting recovery (copying) of an existing shard from another data node, or remove a shard. When the data node finishes |
There was a problem hiding this comment.
grammar nit:
| new empty shard, starting recovery (copying) of an existing shard from another data node, or remove a shard. When the data node finishes | |
| new empty shard, starting recovery (copying) of an existing shard from another data node, or removing a shard. When the data node finishes |
but also I feel we should expand a bit on what "removing a shard" means in this context? It means actively shutting down the corresponding running (or recovering) IndexShard instance, releasing all its resources.
Again, this feels like detail that should be in a Javadoc somewhere, maybe org.elasticsearch.cluster.routing.GlobalRoutingTable, and linked to from RoutingTable, IndexRoutingTable, IndexShardRoutingTable and ShardRouting.
There was a problem hiding this comment.
Applied removing 👍
but also I feel we should expand a bit on what "removing a shard" means in this context? It means actively shutting down the corresponding running (or recovering) IndexShard instance, releasing all its resources.
The details of removal of a shard seem akin to the details of recovery of a shard, which is a separate component than allocation? It seems off topic to me: this section explains how the master and a data node communicate to change shard allocations. Though perhaps you're thinking about it from a some other angle?
There was a problem hiding this comment.
Again, this feels like detail that should be in a Javadoc somewhere, maybe org.elasticsearch.cluster.routing.GlobalRoutingTable, and linked to from RoutingTable, IndexRoutingTable, IndexShardRoutingTable and ShardRouting.
We should be able to drill down from the high level architecture guide to the package-info and down into a class file, with increasing amounts of detail. The presence of lower level documentation wouldn't eliminate the need for the architecture guide.
I can't find the classes that would contain additional documentation unless I know how to find them. I had not previously heard of GlobalRoutingTable. I consider this documentation's level of detail insufficient for a class level comment and too wide ranging across multiple classes.
There was a problem hiding this comment.
Thanks for the feedback, I've finally updated the text.
Info is all good but I think would be better placed in Javadocs, either duplicated here or else with pointers from here to the Javadocs.
I think it depends on the level of detail, and there's a discoverability concern. I've responded in the threads. I've also spun off a separate PR that should go in first.
| ### Indexes and Shards | ||
|
|
||
| Each index consists of a fixed number of primary shards. The number of primary shards cannot be changed for the lifetime of the index. Each | ||
| primary shard can have zero-to-many replicas used for data redundancy. The number of replicas per shard can be changed in runtime. Each |
There was a problem hiding this comment.
Applied dynamically 👍
| updated `RoutingTable`. The `RoutingTable` is part of the cluster state, so the master node updates the cluster state with the new | ||
| (incremental) desired shard allocation information. The updated cluster state is then published to the data nodes. Each data node will | ||
| observe any change in shard allocation related to that node and take action to achieve the new shard allocation by initiating creation of a | ||
| new empty shard, starting recovery (copying) of an existing shard from another data node, or remove a shard. When the data node finishes |
There was a problem hiding this comment.
Applied removing 👍
but also I feel we should expand a bit on what "removing a shard" means in this context? It means actively shutting down the corresponding running (or recovering) IndexShard instance, releasing all its resources.
The details of removal of a shard seem akin to the details of recovery of a shard, which is a separate component than allocation? It seems off topic to me: this section explains how the master and a data node communicate to change shard allocations. Though perhaps you're thinking about it from a some other angle?
| updated `RoutingTable`. The `RoutingTable` is part of the cluster state, so the master node updates the cluster state with the new | ||
| (incremental) desired shard allocation information. The updated cluster state is then published to the data nodes. Each data node will | ||
| observe any change in shard allocation related to that node and take action to achieve the new shard allocation by initiating creation of a | ||
| new empty shard, starting recovery (copying) of an existing shard from another data node, or remove a shard. When the data node finishes |
There was a problem hiding this comment.
Again, this feels like detail that should be in a Javadoc somewhere, maybe org.elasticsearch.cluster.routing.GlobalRoutingTable, and linked to from RoutingTable, IndexRoutingTable, IndexShardRoutingTable and ShardRouting.
We should be able to drill down from the high level architecture guide to the package-info and down into a class file, with increasing amounts of detail. The presence of lower level documentation wouldn't eliminate the need for the architecture guide.
I can't find the classes that would contain additional documentation unless I know how to find them. I had not previously heard of GlobalRoutingTable. I consider this documentation's level of detail insufficient for a class level comment and too wide ranging across multiple classes.
|
|
||
| Each index consists of a fixed number of primary shards. The number of primary shards cannot be changed for the lifetime of the index. Each | ||
| primary shard can have zero-to-many replicas used for data redundancy. The number of replicas per shard can be changed in runtime. Each | ||
| shard copy (primary or replica) can be in one of four states: |
There was a problem hiding this comment.
Thanks for the info. I was trying to shove Ievgen's brain dump info into here, but I wasn't that familiar with the relevant code. I agree, this information is too low-level. I pushed it down into ShardRoutingState in a separate PR because I want to add a reference to the new documentation in this PR.
I wasn't too sure where to push the state transitions. I guess we can prescribe how to transition states in the ShardRoutingState, too, but it's a little strange to describe how the class is used by other classes when ShardRoutingState doesn't enforce those transitions itself. Oh well 🤷♀️
|
#126875 has been committed and I've updated the text here to reference it. Ready for another review 👍 |
Co-authored-by: David Turner <[email protected]>
I was thinking that the java class level documentation was hard to find, not the architecture guide 🙃 I see expectations vary 😁 |
|
Links all round :) |
DiannaHohensee
added
the
auto-merge-without-approval
Adds discussion of index shards and their states, as well as
the communication flow between the master node and data
nodes for shard allocation changes.
Relates ES-7874
The first section is an attempt to move some of the allocation brain dump google document into the architecture guide.