Skip to content

Commit f13b3a0

Browse files
authored
docs: playbook for AWS RDS to Timescale (timescale#2840)
* docs: add playbooks for RDS to Timescale migration with pg_dump & live migration Signed-off-by: Harkishen-Singh <[email protected]>
1 parent 6dec291 commit f13b3a0

File tree

8 files changed

+810
-20
lines changed

8 files changed

+810
-20
lines changed
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
import ExplainPgDumpFlags from "versionContent/_partials/_migrate_explain_pg_dump_flags.mdx";
2+
3+
```sh
4+
pg_dump -d "$SOURCE" \
5+
--format=plain \
6+
--quote-all-identifiers \
7+
--no-tablespaces \
8+
--no-owner \
9+
--no-privileges \
10+
--schema-only \
11+
--file=dump.sql \
12+
--snapshot=$(cat /tmp/pgcopydb/snapshot)
13+
```
14+
15+
- `--schema-only` is used to dump only the object definitions (schema), not
16+
data.
17+
18+
- `--snapshot` is used to specified the synchronized [snapshot][snapshot] when
19+
making a dump of the database.
20+
21+
<ExplainPgDumpFlags />
22+
23+
[snapshot]: https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-SNAPSHOT-SYNCHRONIZATION
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
```bash
2+
pg_dumpall -d "$SOURCE" \
3+
--quote-all-identifiers \
4+
--roles-only \
5+
--no-role-passwords \
6+
--file=roles.sql
7+
```
8+
9+
<Highlight type="important">
10+
AWS RDS does not permit dumping of roles with passwords, which
11+
is why the above command is executed with the `--no-role-passwords`. However,
12+
when the migration of roles to your Timescale instance is complete, you
13+
need to manually assign passwords to the necessary roles using the following
14+
command: `ALTER ROLE name WITH PASSWORD 'password';`
15+
</Highlight>
16+
17+
Timescale services do not support roles with superuser access. If your SQL
18+
dump includes roles that have such permissions, you'll need to modify the file
19+
to be compliant with the security model.
20+
21+
You can use the following `sed` command to remove unsupported statements and
22+
permissions from your roles.sql file:
23+
24+
```bash
25+
sed -i -E \
26+
-e '/CREATE ROLE "postgres";/d' \
27+
-e '/ALTER ROLE "postgres"/d' \
28+
-e '/CREATE ROLE "rds/d' \
29+
-e '/ALTER ROLE "rds/d' \
30+
-e '/TO "rds/d' \
31+
-e '/GRANT "rds/d' \
32+
-e 's/(NO)*SUPERUSER//g' \
33+
-e 's/(NO)*REPLICATION//g' \
34+
-e 's/(NO)*BYPASSRLS//g' \
35+
-e 's/GRANTED BY "[^"]*"//g' \
36+
roles.sql
37+
```
38+
39+
<Highlight type="note">
40+
This command works only with the GNU implementation of sed (sometimes referred
41+
to as gsed). For the BSD implementation (the default on macOS), you need to
42+
add an extra argument to change the `-i` flag to `-i ''`.
43+
44+
To check the sed version, you can use the command `sed --version`. While the
45+
GNU version explicitly identifies itself as GNU, the BSD version of sed
46+
generally doesn't provide a straightforward --version flag and simply outputs
47+
an "illegal option" error.
48+
</Highlight>
49+
50+
A brief explanation of this script is:
51+
52+
- `CREATE ROLE "postgres"`; and `ALTER ROLE "postgres"`: These statements are
53+
removed because they require superuser access, which is not supported
54+
by Timescale.
55+
56+
- `(NO)SUPERUSER` | `(NO)REPLICATION` | `(NO)BYPASSRLS`: These are permissions
57+
that require superuser access.
58+
59+
- `CREATE ROLE "rds`, `ALTER ROLE “rds`, `TO "rds`, `GRANT "rds`: Any creation
60+
or alteration of rds prefixed roles are removed because of their lack of any use
61+
in a Timescale instance. Similarly, any grants to or from "rds" prefixed roles
62+
are ignored as well.
63+
64+
- `GRANTED BY role_specification`: The GRANTED BY clause can also have permissions that
65+
require superuser access and should therefore be removed. Note: Per the
66+
TimescaleDB documentation, the GRANTOR in the GRANTED BY clause must be the
67+
current user, and this clause mainly serves the purpose of SQL compatibility.
68+
Therefore, it's safe to remove it.

migrate/index.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,8 @@ and importing it with [timescaledb-parallel-copy][parallel-copy].
7575

7676
For other ingestion methods, see the [data ingest section][data-ingest].
7777

78+
For a detailed, step-by-step guide on migrating from various databases to Timescale, please refer to the migration [playbooks].
79+
7880
If you encounter any difficulties while migrating, consult the
7981
[troubleshooting] documentation, open a support requestl, or take
8082
your issue to the `#migration` channel in the [community slack](https://slack.timescale.com/),
@@ -89,3 +91,4 @@ where the developers of this migration method are there to help.
8991
[troubleshooting]: /migrate/:currentVersion:/troubleshooting/
9092
[live-migration]: /migrate/:currentVersion:/live-migration/
9193
[pgcopydb]: https://github.com/dimitri/pgcopydb
94+
[playbooks]: /migrate/:currentVersion:/playbooks/

migrate/live-migration/live-migration-from-postgres.md

Lines changed: 2 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ import GettingHelp from "versionContent/_partials/_migrate_dual_write_backfill_g
1010
import SourceTargetNote from "versionContent/_partials/_migrate_source_target_note.mdx";
1111
import StepOne from "versionContent/_partials/_migrate_dual_write_step1.mdx";
1212
import DumpDatabaseRoles from "versionContent/_partials/_migrate_dual_write_dump_database_roles.mdx";
13-
import ExplainPgDumpFlags from "versionContent/_partials/_migrate_explain_pg_dump_flags.mdx";
13+
import DumpSourceSchema from "versionContent/_partials/_migrate_dump_source_schema.mdx";
1414

1515
# Live migration from PostgreSQL database with pgcopydb
1616

@@ -208,25 +208,7 @@ longer the buffered files need to be stored.
208208

209209
### 4b. Dump the database schema from the source database
210210

211-
```sh
212-
pg_dump -d "$SOURCE" \
213-
--format=plain \
214-
--quote-all-identifiers \
215-
--no-tablespaces \
216-
--no-owner \
217-
--no-privileges \
218-
--schema-only \
219-
--file=dump.sql \
220-
--snapshot=$(cat /tmp/pgcopydb/snapshot)
221-
```
222-
223-
- `--schema-only` is used to dump only the object definitions (schema), not
224-
data.
225-
226-
- `--snapshot` is used to specified the synchronized [snapshot][snapshot] when
227-
making a dump of the database.
228-
229-
<ExplainPgDumpFlags />
211+
<DumpSourceSchema />
230212

231213
### 4c. Load the roles and schema into the target database
232214

migrate/page-index/page-index.js

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,25 @@ module.exports = [
7474
},
7575
],
7676
},
77+
{
78+
title: "Playbooks",
79+
href: "playbooks",
80+
excerpt: "Step-by-step migration playbook to Timescale",
81+
children: [
82+
{
83+
title: "From AWS RDS using pg_dump",
84+
href: "rds-timescale-pg-dump",
85+
excerpt:
86+
"Migrate from RDS to Timescale using pg_dump",
87+
},
88+
{
89+
title: "From AWS RDS using live migration",
90+
href: "rds-timescale-live-migration",
91+
excerpt:
92+
"Migrate from RDS to Timescale using live migration",
93+
},
94+
],
95+
},
7796
{
7897
title: "Troubleshooting",
7998
href: "troubleshooting",

migrate/playbooks/index.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
---
2+
title: Playbooks
3+
excerpt: Step-by-step playbooks on migrating your database to Timescale
4+
products: [cloud]
5+
keywords: [migration, low-downtime, pg_dump, tutorial]
6+
tags: [recovery, logical backup, replication]
7+
---
8+
9+
# Playbooks
10+
11+
Migration playbooks serve as comprehensive tutorials, providing step-by-step instructions on how to migrate your database to Timescale. The available migration guides are as follows:
12+
13+
1. [With downtime: AWS RDS PostgreSQL database to Timescale using pg_dump/pg_restore][rds-timescale-downtime]
14+
1. [With low-downtime: AWS RDS PostgreSQL database to Timescale using live migration][rds-timescale-low-downtime]
15+
16+
[rds-timescale-downtime]: /migrate/:currentVersion:/playbooks/rds-timescale-pg-dump/
17+
[rds-timescale-low-downtime]: /migrate/:currentVersion:/playbooks/rds-timescale-live-migration/

0 commit comments

Comments
 (0)