Skip to content

retention WAL purge #69

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
gsmolk opened this issue May 18, 2019 · 4 comments
Closed

retention WAL purge #69

gsmolk opened this issue May 18, 2019 · 4 comments
Assignees
@gsmolk
Labels
bug
Milestone
2.2.1

Comments

@gsmolk
Copy link
Contributor

gsmolk commented May 18, 2019

Setup

            /-------B1------            timeline b
A1----------------A2----                timeline a

If we run pg_probackup delete -i A1 --wal then timeline b will be wiped out.

Test:

======================================================================
ERROR: test_delete_wal_between_multiple_timelines (tests.delete.DeleteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/delete.py", line 296, in test_delete_wal_between_multiple_timelines
    self.validate_pb(backup_dir)
  File "tests/helpers/ptrack_helpers.py", line 930, in validate_pb
    return self.run_pb(cmd_list + options, old_binary=old_binary, gdb=gdb)
  File "tests/helpers/ptrack_helpers.py", line 650, in run_pb
    raise ProbackupException(e.output.decode('utf-8'), self.cmd)
ProbackupException: 
 ERROR: u'INFO: Validate backups of the instance \'node\'\nINFO: Validating backup PRPIF7\nINFO: Backup PRPIF7 data files are valid\nWARNING: Thread [1]: Could not read WAL record at 0/7000028\nERROR: Thread [1]: WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines/backup/wal/node/000000020000000000000007" is absent\nWARNING: There are not enough WAL records to consistenly restore backup PRPIF7 from START LSN: 0/7000028 to STOP LSN: 0/7000160\nWARNING: Backup PRPIF7 WAL segments are corrupted\nINFO: Validating backup PRPIF1\nINFO: Backup PRPIF1 data files are valid\nINFO: Backup PRPIF1 WAL segments are valid\nWARNING: Some backups are not valid\n'
 CMD: ['/home/gsmol/git/postgres/contrib/pg_probackup/pg_probackup validate -B /home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines/backup']
@gsmolk gsmolk added the bug label May 18, 2019
@gsmolk
Copy link
Contributor Author

gsmolk commented May 18, 2019

Current algorithm of wal purge is flawed, I think we should calculate oldest backup for every timeline, and check every segment against them. Tricky part is that oldest backup for some timeline can be located in different timeline:

         /-------------            timeline c
--B1------------------             timeline b
------------A1-------              timeline a

In this example B1 is oldest backup for timeline b and timeline c. A1 is oldest backup for timeline a

@gsmolk gsmolk added this to the 2.2.0 milestone Sep 18, 2019
@gsmolk gsmolk self-assigned this Sep 19, 2019
@gsmolk
Copy link
Contributor Author

gsmolk commented Sep 20, 2019

Also add support for dry-run flag as in #122

@gsmolk gsmolk mentioned this issue Sep 20, 2019
Closed
gsmolk added a commit that referenced this issue Sep 20, 2019
@gsmolk
[Issue #69] New algorithm of WAL purge, based on timelineInfo infrast…
eb9b234 
…ructure
@gsmolk
Copy link
Contributor Author

gsmolk commented Sep 20, 2019

Sample output:

['/home/gsmol/git/postgres/contrib/pg_probackup/pg_probackup delete -B /home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup --instance=replica --delete-wal --log-level-console=verbose']
VERBOSE: backup history file "000000010000000000000002.00000028.backup"
VERBOSE: backup history file "000000010000000000000004.00000028.backup"
VERBOSE: backup history file "00000002000000000000000C.00000028.backup"
VERBOSE: backup history file "000000020000000000000011.00000028.backup"
VERBOSE: backup history file "000000030000000000000011.00000028.backup"
VERBOSE: backup history file "000000030000000000000014.00000028.backup"
VERBOSE: backup history file "000000030000000000000017.00000028.backup"
INFO: All files on timeline 1 will be removed
INFO: WAL segments between 0000000000000001 and 000000000000000A on timeline 1 will be removed
INFO: WAL size to remove on timeline 1: 144MB
INFO: Resident data size to free on timeline 1: 7584kB
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000010000000000000001.gz"
VERBOSE: Removed backup history file "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000010000000000000002.00000028.backup"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000010000000000000002.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000010000000000000003.gz"
VERBOSE: Removed backup history file "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000010000000000000004.00000028.backup"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000010000000000000004.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000010000000000000005.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000010000000000000006.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000010000000000000007.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000010000000000000008.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000010000000000000009.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/00000001000000000000000A.gz"
INFO: WAL segments between 000000000000000B and 0000000000000011 on timeline 2 will be removed
INFO: WAL size to remove on timeline 2: 96MB
INFO: Resident data size to free on timeline 2: 4162kB
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/00000002000000000000000B.gz"
VERBOSE: Removed backup history file "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/00000002000000000000000C.00000028.backup"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/00000002000000000000000C.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/00000002000000000000000D.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/00000002000000000000000E.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/00000002000000000000000F.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000020000000000000010.gz"
INFO: WAL segments between 0000000000000010 and 0000000000000017 on timeline 3 will be removed
INFO: WAL size to remove on timeline 3: 112MB
INFO: Resident data size to free on timeline 3: 5833kB
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000030000000000000010.gz"
VERBOSE: Removed backup history file "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000030000000000000011.00000028.backup"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000030000000000000011.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000030000000000000012.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000030000000000000013.gz"
VERBOSE: Removed backup history file "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000030000000000000014.00000028.backup"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000030000000000000014.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000030000000000000015.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000030000000000000016.gz"
INFO: All files on timeline 4 will be removed
INFO: WAL segments between 0000000000000016 and 0000000000000018 on timeline 4 will be removed
INFO: WAL size to remove on timeline 4: 32MB
INFO: Resident data size to free on timeline 4: 2327kB
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000040000000000000016.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000040000000000000017.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000040000000000000018.gz"
INFO: All files on timeline 5 will be removed
INFO: WAL segments between 0000000000000017 and 0000000000000018 on timeline 5 will be removed
INFO: WAL size to remove on timeline 5: 16MB
INFO: Resident data size to free on timeline 5: 797kB
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000050000000000000017.gz"
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/000000050000000000000018.gz"
INFO: All files on timeline 6 will be removed
INFO: Resident data size to free on timeline 6: 586kB
VERBOSE: Removed WAL segment "/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/delete/test_delete_wal_between_multiple_timelines_1/backup/wal/replica/00000006000000000000000C.gz"
INFO: There are no backups to merge by retention policy
INFO: There are no backups to delete by retention policy

@gsmolk
Copy link
Contributor Author

gsmolk commented Sep 23, 2019

Fixed.

@gsmolk gsmolk closed this as completed Sep 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gsmolk gsmolk

Labels
bug
Projects
None yet
Milestone
2.2.1
Development

No branches or pull requests
1 participant
@gsmolk