Prevent deletion of snapshots that are mounted and ILM delete action

[tlrx]

Today nothing prevents the deletion of a snapshot that is used by a searchable snapshot index. If such a snapshot is deleted the mounted index will go RED in the future. We started to implement a check in #73821 to prevent the deletion of snapshots that are used by searchable snapshots indices within the same cluster and it made some ILM tests to fail.

ILM has a delete index action DeleteAction with a delete_searchable_snapshot flag that, when set to true (the default value), deletes the snapshot that is mounted before deleting the index. ILM deletes the snapshot before the index because the information about the snapshot are stored in the index's metadata (under the ILM custom metedata). If ILM was deleting the searchable snapshot index before the snapshot it would not know which snapshot to delete.

I still think it is important to prevent the deletion of mounted snapshots so we should investigate how to keep the ILM behavior while implementing a safety check in #73821.

I can see different solution and I'm opening this issue to discuss them:

• Adds a specific flag to the delete snapshot request that allows to bypass the safety check. This flag should not be exposed at the REST API level and can be kept as internal, only set by ILM delete action.
• Allow to delete a searchable snapshot index and its snapshot together. The index metadata should be removed from the cluster state at the same time the snapshot is added as a snapshot deletion in progress. It implies multiple verifications (if the snapshot is used by another index, if the snapshot is used by a restore etc) but it should be doable. Wether this should be implemented in the existing delete index action, the existing snapshot delete action or a new action (like Unmount API) is left to decide.
• ILM keeps the information of snapshots to delete somewhere else

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[tlrx]

@henningandersen suggested the idea to add a flag to the existing Mount API to indicate that the index "owns" the snapshot and that it should be deleted when the searchable snapshot index is deleted. I'm going to give it try.

[kunisen]

Hi @tlrx and @DaveCTurner hope you are doing well.

We have observed this issue happened from time to time, and users get a big surprise during the next ECH configuration (plan) change where frozen nodes restart and that wipes out the cache on disk, while before that, they could do searchable snapshot deletion and then search without any issue, which leads to a wrong understanding that deleting searchable snapshots is not a problem but actually it is.

Could we make at least the human deletion to be more restricted? e.g. adding a ?acknowledged=true parameter to the deletion API so that users have to specify this explicitly to make sure they understand what they are doing, and show some docs if they don't append this parameter.

And for ILM auto-managed ones, we apply the ?acknowledged=true automatically.

This is similar to what we have for license management: https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-license-post

---

By doing so, we can prevent inadvertent or accidental "manual deletion", and ILM things, we can improve the docs to make it more clear.