Support immutable repository when using snapshot and restore along with point in time recovery

[bradquarry]

Description

A customer has a requirement to use the Elastic snapshot capability to store data in an immutable repository endpoint.

The repository filesystem only allows for writes and does not allow deletes under any circumstances.

They are asking that we look at how we might adapt the current snapshot and restore capabilities to use this immutable filesystem endpoint.

They also need to be able to restore from any snapshot state.

Meaning, keeping all segments in the repository and if a restore is done from an older backup vs. current only restoring the older segments as a 'point in time'.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DaveCTurner]

They are asking that we look at how we might adapt the current snapshot and restore capabilities to use this immutable filesystem endpoint.

I believe ES will already work correctly against such a filesystem. If deletes are blocked by the underlying storage then the snapshot functionality will still work correctly. It'll log warnings about being unable to delete blobs but they can be ignored if this is expected. I expect we could find a way to quieten down any such logs.

They also need to be able to restore from any snapshot state.

This can be achieved by preventing users from deleting snapshots by removing the cluster:admin/snapshot/delete permission from all roles.

[ywangd]

If the deletion keeps failing and blobs accumulating, will the cluster still risk OOM or is it no longer an issue since #109848?

[DaveCTurner]

#109848 bounds the memory usage for the blobs-to-delete list to 2GiB, so as long as the master has more than this spare it'll be fine. But also this only applies if the user is deleting snapshots, which they won't be able to do if cluster:admin/snapshot/delete is forbidden.

[jmathew1279]

Even the initial repository connection fails for the first time with a immutability enabled. Because of this we are not able to create a repository since it tries to create an object and delete it as part of the initial repository creation.

[DaveCTurner]

Ah interesting, I'd forgotten about that. You can skip this step by setting ?verify=false when you register the repository.