Skip to content

Commit 3ffc8ac

Browse files
sdudoladovsdudoladov
authored
Sample logical backup image. (zalando#572)
* Sample logical backup image. Based on the earlier work by Dmitry Dolgov @erthalion
1 parent b619569 commit 3ffc8ac

File tree

4 files changed

+130
-3
lines changed

4 files changed

+130
-3
lines changed

‎docker/logical-backup/Dockerfile‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
FROM ubuntu:18.04
2+
LABEL maintainer="Team ACID @ Zalando <[email protected]>"
3+
4+
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
5+
RUN apt-get update \
6+
&& apt-get install --no-install-recommends -y \
7+
apt-utils \
8+
ca-certificates \
9+
lsb-release \
10+
pigz \
11+
python3-pip \
12+
python3-setuptools \
13+
curl \
14+
jq \
15+
gnupg \
16+
&& pip3 install --no-cache-dir awscli --upgrade \
17+
&& echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
18+
&& cat /etc/apt/sources.list.d/pgdg.list \
19+
&& curl --silent https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - \
20+
&& apt-get update \
21+
&& apt-get install --no-install-recommends -y \
22+
postgresql-client-11 \
23+
postgresql-client-10 \
24+
postgresql-client-9.6 \
25+
postgresql-client-9.5 \
26+
&& apt-get clean \
27+
&& rm -rf /var/lib/apt/lists/*
28+
29+
COPY dump.sh ./
30+
31+
ENV PG_DIR=/usr/lib/postgresql/
32+
33+
ENTRYPOINT ["/dump.sh"]

‎docker/logical-backup/dump.sh‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
#! /usr/bin/env bash
2+
3+
# enable unofficial bash strict mode
4+
set -o errexit
5+
set -o nounset
6+
set -o pipefail
7+
IFS=$'\n\t'
8+
9+
# make script trace visible via `kubectl logs`
10+
set -o xtrace
11+
12+
ALL_DB_SIZE_QUERY="select sum(pg_database_size(datname)::numeric) from pg_database;"
13+
PG_BIN=$PG_DIR/$PG_VERSION/bin
14+
DUMP_SIZE_COEFF=5
15+
16+
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
17+
K8S_API_URL=https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT/api/v1
18+
CERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
19+
20+
function estimate_size {
21+
"$PG_BIN"/psql -tqAc "${ALL_DB_SIZE_QUERY}"
22+
}
23+
24+
function dump {
25+
# settings are taken from the environment
26+
"$PG_BIN"/pg_dumpall
27+
}
28+
29+
function compress {
30+
pigz
31+
}
32+
33+
function aws_upload {
34+
declare -r EXPECTED_SIZE="$1"
35+
36+
# mimic bucket setup from Spilo
37+
# to keep logical backups at the same path as WAL
38+
# NB: $LOGICAL_BACKUP_S3_BUCKET_SCOPE_SUFFIX already contains the leading "/" when set by the Postgres operator
39+
PATH_TO_BACKUP=s3://$LOGICAL_BACKUP_S3_BUCKET"/spilo/"$SCOPE$LOGICAL_BACKUP_S3_BUCKET_SCOPE_SUFFIX"/logical_backups/"$(date +%s).sql.gz
40+
41+
if [ -z "$EXPECTED_SIZE" ]; then
42+
aws s3 cp - "$PATH_TO_BACKUP" --debug --sse="AES256"
43+
else
44+
aws s3 cp - "$PATH_TO_BACKUP" --debug --expected-size "$EXPECTED_SIZE" --sse="AES256"
45+
fi;
46+
}
47+
48+
function get_pods {
49+
declare -r SELECTOR="$1"
50+
51+
curl "${K8S_API_URL}/pods?$SELECTOR" \
52+
--cacert $CERT \
53+
-H "Authorization: Bearer ${TOKEN}" | jq .items[].status.podIP -r
54+
}
55+
56+
function get_current_pod {
57+
curl "${K8S_API_URL}/pods?fieldSelector=metadata.name%3D${HOSTNAME}" \
58+
--cacert $CERT \
59+
-H "Authorization: Bearer ${TOKEN}"
60+
}
61+
62+
declare -a search_strategy=(
63+
list_all_replica_pods_current_node
64+
list_all_replica_pods_any_node
65+
get_master_pod
66+
)
67+
68+
function list_all_replica_pods_current_node {
69+
get_pods "labelSelector=version%3D${SCOPE},spilo-role%3Dreplica&fieldSelector=spec.nodeName%3D${CURRENT_NODENAME}" | head -n 1
70+
}
71+
72+
function list_all_replica_pods_any_node {
73+
get_pods "labelSelector=version%3D${SCOPE},spilo-role%3Dreplica" | head -n 1
74+
}
75+
76+
function get_master_pod {
77+
get_pods "labelSelector=version%3D${SCOPE},spilo-role%3Dmaster" | head -n 1
78+
}
79+
80+
CURRENT_NODENAME=$(get_current_pod | jq .items[].spec.nodeName --raw-output)
81+
export CURRENT_NODENAME
82+
83+
for search in "${search_strategy[@]}"; do
84+
85+
PGHOST=$(eval "$search")
86+
export PGHOST
87+
88+
if [ -n "$PGHOST" ]; then
89+
break
90+
fi
91+
92+
done
93+
94+
dump | compress | aws_upload $(($(estimate_size) / DUMP_SIZE_COEFF))

‎docs/administrator.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -346,7 +346,7 @@ The operator logs reasons for a rolling update with the `info` level and a diff
346346

347347
The operator can manage k8s cron jobs to run logical backups of Postgres clusters. The cron job periodically spawns a batch job that runs a single pod. The backup script within this pod's container can connect to a DB for a logical backup. The operator updates cron jobs during Sync if the job schedule changes; the job name acts as the job identifier. These jobs are to be enabled for each indvidual Postgres cluster by setting `enableLogicalBackup: true` in its manifest. Notes:
348348

349-
1. The provided `registry.opensource.zalan.do/acid/logical-backup` image implements the backup via `pg_dumpall` and upload of (compressed) results to an S3 bucket; `pg_dumpall` requires a `superuser` access to a DB and runs on the replica when possible.
349+
1. The [example image](../docker/logical-backup/Dockerfile) implements the backup via `pg_dumpall` and upload of compressed and encrypted results to an S3 bucket; the default image ``registry.opensource.zalan.do/acid/logical-backup`` is the same image built with the Zalando-internal CI pipeline. `pg_dumpall` requires a `superuser` access to a DB and runs on the replica when possible.
350350

351351
2. Due to the [limitation of Kubernetes cron jobs](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cron-job-limitations) it is highly advisable to set up additional monitoring for this feature; such monitoring is outside of the scope of operator responsibilities.
352352

‎docs/reference/operator_parameters.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -485,9 +485,9 @@ scalyr sidecar. In the CRD-based configuration they are grouped under the
485485

486486
* **logical_backup_schedule**
487487
Backup schedule in the cron format. Please take [the reference schedule format](https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#schedule) into account. Default: "30 00 \* \* \*"
488-
488+
489489
* **logical_backup_docker_image**
490-
Docker image for the pods of the cron job. Must implement backup logic and correctly handle pod and job restarts. The default image runs `pg_dumpall` (on a replica if possible) and uploads compressed results to an S3 bucket under the key `/spilo/pg_cluster_name/cluster_k8s_uuid/logical_backups` Default: "registry.opensource.zalan.do/acid/logical-backup"
490+
An image for pods of the logical backup job. The [example image](../../docker/logical-backup/Dockerfile) runs `pg_dumpall` on a replica if possible and uploads compressed results to an S3 bucket under the key `/spilo/pg_cluster_name/cluster_k8s_uuid/logical_backups`. The default image is the same image built with the Zalando-internal CI pipeline. Default: "registry.opensource.zalan.do/acid/logical-backup"
491491

492492
* **logical_backup_s3_bucket**
493493
S3 bucket to store backup results. The bucket has to be present and accessible by Postgres pods. Default: empty.

0 commit comments

Comments
 (0)