Cluster Reroute | Allocate Replica | Allow fallback data tier

[stefnestor]

Description

👋 howdy, team!

In the situation of 2 hot and 1 warm node which can block ILM migrations for indices hot>warm, it would be nice if a Cluster Reroute > allocate_replica request allowed assigning the replica to data_hot even when its index.routing.allocation.include._tier_preference is data_warm,data_hot.

Example

• 2 hot nodes (instance-0000000009, instance-0000000008) & 1 warm node (instance-0000000010)

• index test is 1p1r with primary on warm#10 and replica unassigned

• attempt Cluster Reroute to send _tier_preference: data_warm,data_hot replica to data_hot and error that not data_warm

  POST _cluster/reroute
  {"commands":[{"allocate_replica":{ "index" : "test", "shard" : 0, "node" : "instance-0000000009" }}]}
  
  {
    "error": {
      "type": "illegal_argument_exception",
      "reason": "[allocate_replica] allocation of [test][0] on node {instance-0000000009}{...}{himrst}{8.9.2}{...} is not allowed, reason: [
      YES(shard has no previous failures)][
      YES(primary shard for this replica is already active)][
      YES(explicitly ignoring any disabling of allocation due to manual allocation commands via the reroute API)][
      YES(can allocate replica shard to a node with version [8.9.2] since this is equal-or-newer than the primary version [8.9.2])][
      YES(the shard is not being snapshotted)][
      YES(ignored as shard is not being recovered from a snapshot)][
      YES(no nodes are shutting down)][
      YES(there are no ongoing node replacements)][
      YES(node passes include/exclude/require filters)][
      YES(this node does not hold a copy of this shard)][
      YES(enough disk for shard on node, free: [321.3gb], used: [28.5%], shard size: [0b], free after allocating shard: [321.3gb])][
      YES(below shard recovery limit of outgoing: [0 < 2] incoming: [0 < 2])][
      YES(total shard limits are disabled: [index: -1, cluster: -1] <= 0)][
      YES(node meets all awareness attribute requirements)][
  -    NO(index has a preference for tiers [data_warm,data_hot] and node does not meet the required [data_warm] tier)][
      YES(shard is not a follower and is not under the purview of this decider)][
      YES(decider only applicable for indices backed by searchable snapshots)][
      YES(this decider only applies to indices backed by searchable snapshots)][
      YES(decider only applicable for indices backed by searchable snapshots)][
      YES(this node's data roles are not exactly [data_frozen] so it is not a dedicated frozen node)][
      YES(decider only applicable for indices backed by archive functionality)
      ]"
    },
    "status": 400
  }

TIA! 🙏

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DaveCTurner]

This is the expected and desired behaviour. If you only have one warm node then indices allocated to the warm tier must have zero replicas.

[stefnestor]

👋🏽 @DaveCTurner thanks for your feedback!

I don't find this to be desired behavior, but apparently some part of my understanding's off, sorry. Will you kindly help me resolve to documentation:

(ref) For example, if you set index.routing.allocation.include._tier_preference to data_warm,data_hot, the index is allocated to the warm tier if there are nodes with the data_warm role. If there are no nodes in the warm tier, but there are nodes with the data_hot role, the index is allocated to the hot tier.

(ref) In the warm phase, the migrate action sets index.routing.allocation.include._tier_preference to data_warm,data_hot. This moves the index to nodes in the warm tier. If there are no nodes in the warm tier, it falls back to the hot tier.

I had thought these interpreted as a shard-level decision that if there's no available warm tier then allocation would direct the shard to the hot tier. But your update and these literal quotes now sound to me as if it's a cluster-level decision that warm tier will never fallback to hot for any shard/index at all if warm nodes exist within the cluster. Is that a better interpretation?

Asking in case I need to correct my previous doc update #107981 to align (but it maybe also falls in the land of count be interpreted aligned but I didn't mean it that way):

(ref, v8, blame) This setting also accepts multiple tiers in order of preference. This prevents indices from remaining unallocated if no nodes are available in the preferred tier. For example, when index lifecycle management migrates an index to the cold phase, it sets the index _tier_preference to data_cold,data_warm,data_hot.

[DaveCTurner]

Yes that's right, the if there are no nodes in the warm tier clauses do not apply in this case, because there are nodes in the warm tier.

Your wording no nodes are available in the preferred tier is ambiguous, it depends whether "available" means "running" or "valid allocation targets". I think the simplest fix would be to delete that word.