Cleanup after the deleted cluster

[alexeyklyukin]

Right now it is not possible to delete an existing cluster and then create a new one with the same name. When stumbling upon the "/initialize" key from the previous cluster, a Patroni node decide to become a replica of an existing cluster instead of running initidb. Patroni does not delete all objects it creates, namely, non-expiring keys in the DCS, for instance, /initialize. There is simply no good point in the Patroni lifecycle to do this.

Therefore, the operator should take care of deleting those keys: it knows the intention of the user to delete the cluster and it can do this after all Patroni nodes are shut down.

The difficulty is the variety of DCS implementations supported by Patroni (Etcd, Consul, Zookeeper, Kubernetes) and the operator would have to know of all of them. So, as a first step, we can make sure we delete old endpoints created when running in Kubernetes. For the rest of the DCSs, it would be good to come up with a more generic solution that doesn't hard-code the knowledge about how to delete the keys stored there in the operator.

[avaczi]

@alexeyklyukin Wouldn't it make sense to implement the DCS cleanup in Patroni itself? It would then be completely transparent from the operator POV.

[alexeyklyukin]

There is no good point in the Patroni lifecycle to do it. Patroni manages individual PostgreSQL instances. Naturally, you want to delete keys when the instance is shut down; however, you don't know whether the shutdown is performed because the cluster is being deleted, or simply this particular instance should be stopped - this knowledge is completely outside of Patroni.

Ideally, we want to do it when all Patroni instances are stopped by the party that knows about the intention of the user to delete the cluster, but when all Patroni instances are shut down - how would you call Patroni to cleanup?

[alexeyklyukin]

Theoretically, it should be possible to have a shutdown procedure by writing a special /cluster/shutdown key to the DCS: when the nodes see that key, they delete everything but the shutdown and the /member key from the DCS and then stop gracefully, deleting their member keys, and the last member (according to the /member key) also deletes the shutdown key (which also can be set to expire after a certain large interval in case the last member dies).

[alexeyklyukin]

This is, however, error-prone, as determining the last /member will lead to race conditions; perhaps a better way would be for a single node to acquire the shutdown lock and waiting for the member keys other then the one belonging to the lock bearer to expire before removing the shutdown key and shutting down itself at last.

[alexeyklyukin]

Another idea is to make a /shutdown endpoint that simply shuts down this specific node and then add an option to wipe out the cluster contents at the discretion of the user (so that the user that runs /shutdown on the last node runs it with /shutdown wipe=true and it is the task of the user to make sure the node that wipes the cluster is the last one).

[alexeyklyukin]

I specifically like the last one, because it's rather safe from the point of accidentally destroying the cluster: the most damage one can get is to shutdown one node.

[CyberDem0n]

Either way or another it's not possible to implement it in Patroni, because there is always a race condition.

1. "The last" node can't know for sure that it is definitely the last one.
2. Even if it is a really the last one, it could crash at any moment, leaving some stuff uncleaned.

Moreover, the goal of Patroni is maintaining of a cluster in the healthy state, not destroying it!

[alexeyklyukin]

"The last" node can't know for sure that it is definitely the last one.
Even if it is a really the last one, it could crash at any moment, leaving some stuff uncleaned.

We can know it for sure in the operator by scaling the cluster statefulset to one pod and waiting until all other pods terminate.

The node that wipes the cluster on shut down (when asked) can make reasonable efforts, for instance, by deleting all keys before doing anything else (or making them expiring, which might not be compatible with every DCS, so deleting is better AFAIK). If it succeeds - it returns 200 to the caller. If we don't see 200 - we assume it has failed and inform the user that there has been an error.

[CyberDem0n]

We can know it for sure

You can, but Patroni can't

[alexeyklyukin]

Patroni doesn't need to know that, because it is a decision that is made on a cluster level, outside of the Patroni competence. What it needs to do is to follow the user request of shutting down itself and optionally wiping out keys in DCS. It is possible to assure there is only node in the cluster outside of Patroni.

The problem with external tools cleaning up after Patroni is that Patroni supports multiple DCSs and that mechanism is extensible. Making third-party tools learn about Patroni DCS layout and implement support for all the DCSs Patroni uses making those tools unnecessary complex, hindering the adoption of Patroni.

[alexeyklyukin]

That one has been resolved for the case of Patroni using Kubernetes as a DCS (see c4aab50). As we expect most people to use the operator with Patroni configured to use the Kubernetes API addressing the other DCS-es has much lower priority and should be done in a separate issue (ideas how to solve this include running patronictl remove or resuming the discussion of implementing this functionality in Patorni.

[kirillka415]

Dear @CyberDem0n @alexeyklyukin
I have to use ETCD instead of using Kubernetes as a DCS in Azure AKS due to unexpected restart when Patroni can't reach DCS for a while. (postgres-operator:v1.6.1)
And I have faced the same issue when operator do not delete all objects in ETCD.
Here is the list of keys
/initialize.
/config
/history
/optime/leader
I have to delete those keys manually to be able recreate a cluster with the same name.
Could you please implement a fix for this ?