Allow changing index settings of a searchable snapshot in the frozen tier

[gmarouli]

Description

When using searchable snapshots in cold and frozen phases the recovered index has the settings that index had just before it was snapshot. A user might have set a setting like total_shards_per_node to match their needs in the hot tier but this doesn't mean that this is relevant in the cold or frozen tier. In the cold tier a user can use the allocate action to set this setting to the desired value but this is not possible yet in the frozen tier.

This issue requests a way to reset or just set index settings in the frozen tier.

Example:
A user might have the following set-up:

• an index with 2 shards and the setting total_shards_per_node set to 1.
• a hot tier with 2 nodes and a cold and frozen tier with 1 node each.
• the following ILM policy related with the index:

{
  "hot": {
    "min_age": "0ms",
    "actions": {
      "rollover": {
        "max_primary_shard_size": "50gb"
      }
    }
  },
  "cold": {
    "min_age": "1h",
    "actions": {
      "allocate": {
        "total_shards_per_node": "2"
      },
      "searchable_snapshot": {
        "snapshot_repository": "found-snapshots",
        "force_merge_index": false
      }
    }
  },
  "frozen": {
    "min_age": "2h",
    "actions": {
      "searchable_snapshot": {
        "snapshot_repository": "found-snapshots",
        "force_merge_index": false
      }
    }
  }
}

The policy above will have the following effect per tier:

• hot: the index with total_shards_per_node will ensure that every node will have one shard of this index
• cold: the restored snapshot will be restored with total_shards_per_node=1 but because of the allocate action it will be set to 2, so both shards can be allocated in the single cold node.
• frozen: the partially restored snapshot will be restored with total_shards_per_node=1 and one shard will not be able to get allocated because we have 2 shards and only 1 node. Being able to remove or reset total_shards_per_node could solve this.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[passing]

We're having exactly this problem in our hot-frozen setup: for indexes in the hot zone we want to ensure that each of the 6 shards gets allocated to a diferent node. So we would need to set "total_shards_per_node": "1" in the index template, but then the 3 primary shards cannot be allocated when getting migrated to the 2 frozen nodes.

[passing]

We found yet another relevant use case for this: When using Slow Logs, thresholds that make sense on hot nodes will in most cases not make sense on frozen nodes and vice versa. The thresholds would need to be updated on migration to the frozen tier, to get usable results.

https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-slowlog.html

[dakrone]

@gmarouli rather than allowing setting arbitrary index settings for these indices, do you think we could instead only clean the total_shards_per_node setting?

[gmarouli]

@dakrone , I think so, we would have to check which other settings can cause issue though, like the slow logs mentioned before. In order to be complete.

Apologies for the late response, I totally missed it.

[andreidan]

We've discussed this within the team and decided we’ll remove total_shards_per_node automatically when partially mounting and index in frozen (if needed, we’ll have the option to later add an allocate action in the frozen phase)

[predogma]

Is there a target version this will be addressed in?

[gmarouli]

@predogma not yet. But it's on our radar.

[gmarouli]

Hi @predogma , #97979 has been merge and although this PR is not allowing to change the index setting of a searchable snapshot, it does address the situation that made us create this enhancement request. So, from now on, even if the original index did have total_shards_per_node set, this will be ignored in the frozen tier and the searchable snapshot will be successfully mounted.