PostgreSQL has the reputation of being rock solid from its beginnings, and over the years has accumulated a set of impressive features. However the peace of mind that your on-disk data is ACID compliant — if not complemented by an equivalent well thought backup strategy — can be easily shattered.

Backup Types

Before diving into the available tools, let’s look at the available PostgreSQL backup types and what their characteristics are:

SQL dumps (or logical)

• Does not block readers or writers.
• Geared towards small sets of data because of the negative impact on system load and the long time required for both backup and restore operations. The performance may be increased with the –no-sync flag, but refer to the man page for the risks associated with disabling the wait for writes.
• A post-restore ANALYZE is required in order to optimize the statistics.
• Global objects such as roles and tablespaces can only be backed up using pg_dumpall utility. Note that tablespace directories must be manually created prior to starting the restore.
• Supports parallelism at the expense of increased system load. Read man pg_dump for its caveats and special requirements e.g. synchronized snapshots.
• Dumps can be loaded in newer versions of PostgreSQL, or even another machine architecture, however they are not guaranteed to be backwards compatible between major versions so some manual editing of the dump file may be required.

Filesystem (or physical)

• Requires the database to be shut down.
• Faster than logical backups.
• Includes cluster data.
• Can only be restored on the same major version of PostgreSQL.

Continuous archiving (or Point In Time Recovery or PITR)

• Suitable for very large databases where logical or physical backups would take too long.
• Some directories inside the data directory can be excluded to speed up the process.

Snapshots

• Requires operating system support — for example LVM works quite well which is also confirmed by NetBackup for PostgreSQL Agent.
• Suitable for applications where both data directory and the database must be in sync e.g. LAMP applications, provided that the two snapshots are synchronized.
• Not recommended when the database files are stored across multiple filesystems (must snapshot all filesystems simultaneously).

Cloud

All cloud providers implement backups in their PostgreSQL offering. Logical backups can be performed as usual, while physical backups and PITR are available through the cloud service offerings since access to the data store is not available (see for example Amazon Aurora for PostgreSQL). Therefore, backing up PostgreSQL in the cloud will need to be a topic for another blog.

Agent base

• Requires an agent installed on targets.
• Can do block-level backups e.g. COMMVAULT (installation supported on Windows only).

Features

While PostgreSQL provides out of the box the tools required to perform logical, physical, and PITR backups, specialized backup applications rely on the native PostgreSQL and operating system tools to fill the need of implementing a backup strategy that addresses the following points:

• automation
• frequency
• retention period
• integrity
• ease of use

Additionally, PostgreSQL backup tools attempt to provide features common to generic backup tools such as:

• incremental backups for saving storage space
• backup catalogs
• ability to store backups on premise or in the cloud
• alerting and notification
• comprehensive reporting
• access control
• encryption
• graphical interface and dashboards
• backups of remote hosts
• adaptive throughput in order to minimize load on the targets
• handling multiple hosts in parallel
• backup orchestration e.g. jobs chaining
• REST APIs

Lab Setup

For this exercise I’ve setup a command-and-control host that where I’ll be installing the backup tools, that also runs two PostgreSQL instances — 9.6 and 10 — installed from PGDG repositories:

[root@cc ~]# ps -o user,pid,ppid,args --forest -U postgres
USER       PID  PPID COMMAND
postgres  4535     1 /usr/pgsql-10/bin/postmaster -D /var/lib/pgsql/10/data/
postgres  4538  4535  \_ postgres: logger process
postgres  4540  4535  \_ postgres: checkpointer process
postgres  4541  4535  \_ postgres: writer process
postgres  4542  4535  \_ postgres: wal writer process
postgres  4543  4535  \_ postgres: autovacuum launcher process
postgres  4544  4535  \_ postgres: stats collector process
postgres  4545  4535  \_ postgres: bgworker: logical replication launcher
postgres  4481     1 /usr/pgsql-9.6/bin/postmaster -D /var/lib/pgsql/9.6/data/
postgres  4483  4481  \_ postgres: logger process
postgres  4485  4481  \_ postgres: checkpointer process
postgres  4486  4481  \_ postgres: writer process
postgres  4487  4481  \_ postgres: wal writer process
postgres  4488  4481  \_ postgres: autovacuum launcher process
postgres  4489  4481  \_ postgres: stats collector process

[root@cc ~]# netstat -npeelt | grep :543
tcp   0  0  127.0.0.1:5432  0.0.0.0:*  LISTEN  26  79972  4481/postmaster
tcp   0  0  127.0.0.1:5433  0.0.0.0:*  LISTEN  26  81801  4535/postmaster
tcp6  0  0  ::1:5432        :::*       LISTEN  26  79971  4481/postmaster
tcp6  0  0  ::1:5433        :::*       LISTEN  26  81800  4535/postmaster

I’ve also setup two remote PostgreSQL instances running the same versions 9.6 and respectively 10:

[root@db-1 ~]# ps -o user,pid,ppid,args --forest -U postgres
USER       PID  PPID COMMAND
postgres 10972     1 /usr/pgsql-9.6/bin/postmaster -D /var/lib/pgsql/9.6/data/
postgres 10975 10972  \_ postgres: logger process
postgres 10977 10972  \_ postgres: checkpointer process
postgres 10978 10972  \_ postgres: writer process
postgres 10979 10972  \_ postgres: wal writer process
postgres 10980 10972  \_ postgres: autovacuum launcher process
postgres 10981 10972  \_ postgres: stats collector process

[root@db-1 ~]# netstat -npeelt | grep :5432
tcp   0  0  0.0.0.0:5432  0.0.0.0:*  LISTEN  26  34864  10972/postmaster
tcp6  0  0  :::5432       :::*       LISTEN  26  34865  10972/postmaster


[root@db-2 ~]# ps -o user,pid,ppid,args --forest -U postgres
USER       PID  PPID COMMAND
postgres 10829     1 /usr/pgsql-10/bin/postmaster -D /var/lib/pgsql/10/data/
postgres 10831 10829  \_ postgres: logger process
postgres 10833 10829  \_ postgres: checkpointer process
postgres 10834 10829  \_ postgres: writer process
postgres 10835 10829  \_ postgres: wal writer process
postgres 10836 10829  \_ postgres: autovacuum launcher process
postgres 10837 10829  \_ postgres: stats collector process
postgres 10838 10829  \_ postgres: bgworker: logical replication launcher

[root@db-2 ~]# netstat -npeelt | grep :5432
tcp   0  0  0.0.0.0:5432  0.0.0.0:*  LISTEN  26  34242  10829/postmaster
tcp6  0  0  :::5432       :::*       LISTEN  26  34243  10829/postmaster

Next, use pgbench to create a data set:

pgbench=# \dt+
                          List of relations
 Schema |       Name       | Type  |  Owner   |  Size   | Description
--------+------------------+-------+----------+---------+-------------
 public | pgbench_accounts | table | postgres | 128 MB  |
 public | pgbench_branches | table | postgres | 40 kB   |
 public | pgbench_history  | table | postgres | 0 bytes |
 public | pgbench_tellers  | table | postgres | 40 kB   |
(4 rows)

Tools

A list of common backup tools can be found in the PostgreSQL Wiki — Backup section. I’ve augmented the list with products I’ve come across over the years and from a recent Internet search.

Amanda

Amanda is agent based, open source, and supports PostgreSQL out of the box via the ampgsql API. As of this writing, the version 3.5.1 does not support tablespaces (see man ampgsql).

Zmanda provides an enterprise version which is also open source, however not directly available for download as a trial.

Amanda requires a dedicated backup host as the server and client packages exclude each other:

[root@cc ~]# rpm -qp --conflicts ./amanda-backup_client-3.5.1-1.rhel7.x86_64.rpm
amanda-backup_server
[root@cc ~]# rpm -qp --conflicts ./amanda-backup_server-3.5.1-1.rhel7.x86_64.rpm
amanda-backup_client

Follow the basic configuration guide to setup the server and client then configure the PostgreSQL API.

Here’s a git diff from my lab:

• Server:

  • increase the server backup space:

    --- a/etc/amanda/omiday/amanda.conf
    				+++ b/etc/amanda/omiday/amanda.conf
    				@@ -13,7 +13,7 @@ amrecover_changer "changer"
    
    				tapetype "TEST-TAPE"
    				define tapetype TEST-TAPE {
    				1.  length 100 mbytes
    				2.  length 500 mbytes
    					filemark 4 kbytes
    				}

    • define the PostgreSQL target (and disable sample backup):

      --- a/etc/amanda/omiday/disklist
      +++ b/etc/amanda/omiday/disklist
      @@ -1,3 +1,2 @@
      -localhost /etc simple-gnutar-local
      +#localhost /etc simple-gnutar-local
      +10.1.9.243 /var/lib/pgsql/9.6/data dt_ampgsql

• Client:

  • config:

    --- /dev/null
    +++ b/etc/amanda/omiday/amanda-client.conf
    @@ -0,0 +1,5 @@
    +property "PG-DATADIR" "/var/lib/pgsql/9.6/data"
    +property "PG-ARCHIVEDIR" "/var/lib/pgsql/9.6/archive"
    +property "PG-HOST" "/tmp"
    +property "PG-USER" "amandabackup"
    +property "PG-PASSFILE" "/etc/amanda/pg_passfile"

    • authentication file:

      --- /dev/null
      +++ b/etc/amanda/pg_passfile
      @@ -0,0 +1 @@
      +/tmp:*:*:amandabackup:pass

  • authorize the server:

    --- a/var/lib/amanda/.amandahosts
    +++ b/var/lib/amanda/.amandahosts
    @@ -1,2 +1,3 @@
    localhost amandabackup amdump
    localhost.localdomain amandabackup amdump
    +10.1.9.231 amandabackup amdump

  • PostgreSQL authentication:

    --- a/var/lib/pgsql/9.6/data/pg_hba.conf
    +++ b/var/lib/pgsql/9.6/data/pg_hba.conf
    @@ -79,7 +79,8 @@
    # "local" is for Unix domain socket connections only
    local   all             all                                     trust
    # IPv4 local connections:
    -host    all             all             127.0.0.1/32            ident
    +host    all             all             127.0.0.1/32            trust
    +host    all             amandabackup    10.1.9.243/32           trust
    # IPv6 local connections:
    host    all             all             ::1/128                 ident
    # Allow replication connections from localhost, by a user with the

  • PostgreSQL config:

    --- a/var/lib/pgsql/9.6/data/postgresql.conf
    +++ b/var/lib/pgsql/9.6/data/postgresql.conf
    @@ -178,6 +178,7 @@ dynamic_shared_memory_type = posix  # the default is the first option
    
    #wal_level = minimal                   # minimal, replica, or logical
                                           # (change requires restart)
    +wal_level = replica
    #fsync = on                            # flush data to disk for crash safety
                                                    # (turning this off can cause
                                                    # unrecoverable data corruption)
    @@ -215,10 +216,12 @@ dynamic_shared_memory_type = posix        # the default is the first option
    
    #archive_mode = off            # enables archiving; off, on, or always
                                  # (change requires restart)
    +archive_mode = on
    #archive_command = ''          # command to use to archive a logfile segment
                                  # placeholders: %p = path of file to archive
                                  #               %f = file name only
                                  # e.g. 'test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f'
    +archive_command = 'test ! -f /var/lib/pgsql/9.6/archive/%f && cp %p /var/lib/pgsql/9.6/archive/%f'
    #archive_timeout = 0           # force a logfile segment switch after this
                                  # number of seconds; 0 disables

Once completed the above configuration, run the backup:

[amandabackup@cc ~]$ amdump omiday

And verify:

[amandabackup@cc ~]$ amreport omiday
Hostname: cc
Org     : omiday
Config  : omiday
Date    : April 14, 2018

These dumps were to tape MyData01.
The next tape Amanda expects to use is: MyData02.


STATISTICS:
                        Total       Full      Incr.   Level:#
                        --------   --------   --------  --------
Estimate Time (hrs:min)     0:00
Run Time (hrs:min)          0:00
Dump Time (hrs:min)         0:00       0:00       0:00
Output Size (meg)            0.1        0.0        0.1
Original Size (meg)         16.0        0.0       16.0
Avg Compressed Size (%)      0.5        --         0.5
DLEs Dumped                    1          0          1  1:1
Avg Dump Rate (k/s)         33.7        --        33.7

Tape Time (hrs:min)         0:00       0:00       0:00
Tape Size (meg)              0.1        0.0        0.1
Tape Used (%)                0.0        0.0        0.0
DLEs Taped                     1          0          1  1:1
Parts Taped                    1          0          1  1:1
Avg Tp Write Rate (k/s)    830.0        --       830.0


USAGE BY TAPE:
Label                 Time         Size      %  DLEs Parts
MyData01              0:00          83K    0.0     1     1


NOTES:
planner: tapecycle (3) <= runspercycle (3)
planner: Last full dump of 10.1.9.243:/var/lib/pgsql/9.6/data on tape MyData04 overwritten in 3 runs.
taper: tape MyData01 kb 83 fm 1 [OK]


DUMP SUMMARY:
                                                               DUMPER STATS   TAPER STATS
HOSTNAME     DISK                    L ORIG-KB  OUT-KB  COMP%  MMM:SS   KB/s MMM:SS   KB/s
-------------------------------------- ---------------------- -------------- -------------
10.1.9.243   /var/lib/pgsql/9.6/data 1   16416      83    0.5    0:02   33.7   0:00  830.0

(brought to you by Amanda version 3.5.1)

Restoring from backup involves more manual steps as explained in the restore section.

According to the Amanda Enterprise FAQ the following enhancement would apply to our PostgreSQL example:

• management console for automation of backup, retention policies, and schedules
• backup to Amazon S3 cloud storage

Barman

Barman is a disaster recovery solution for PostgreSQL maintained by 2ndQuadrant. It is designed to manage backups for multiple databases and has the ability to restore to a previous point in time using the PITR feature of PostgreSQL.

Barman’s features at a glance:

• handles multiple targets
• support for different PostgreSQL versions
• zero data loss
• streaming and/or standard archiving of WALs
• local or remote recovery
• simplified point in time recovery

As noted in the Barman Manual, support for incremental backups, parallel jobs, data deduplication, and network compression is available only when using the rsync option. Also, streaming WALs from a standby using the archive_command isn’t currently supported.

After following the instructions in the manual for setting up the environment we can verify:

-bash-4.2$ barman list-server
db1 - master
db2 - replica

-bash-4.2$ barman check db1
Server db1:
      PostgreSQL: OK
      is_superuser: OK
      PostgreSQL streaming: OK
      wal_level: OK
      replication slot: OK
      directories: OK
      retention policy settings: OK
      backup maximum age: OK (no last_backup_maximum_age provided)
      compression settings: OK
      failed backups: OK (there are 0 failed backups)
      minimum redundancy requirements: OK (have 0 backups, expected at least 0)
      pg_basebackup: OK
      pg_basebackup compatible: OK
      pg_basebackup supports tablespaces mapping: OK
      archive_mode: OK
      archive_command: OK
      continuous archiving: OK
      pg_receivexlog: OK
      pg_receivexlog compatible: OK
      receive-wal running: OK
      archiver errors: OK

-bash-4.2$ barman check db2
Server db2:
      PostgreSQL: OK
      is_superuser: OK
      PostgreSQL streaming: OK
      wal_level: OK
      replication slot: OK
      directories: OK
      retention policy settings: OK
      backup maximum age: OK (no last_backup_maximum_age provided)
      compression settings: OK
      failed backups: OK (there are 0 failed backups)
      minimum redundancy requirements: OK (have 0 backups, expected at least 0)
      pg_basebackup: OK
      pg_basebackup compatible: OK
      pg_basebackup supports tablespaces mapping: OK
      archive_mode: OK
      archive_command: OK
      continuous archiving: OK
      pg_receivexlog: OK
      pg_receivexlog compatible: OK
      receive-wal running: OK
      archiver errors: OK

Everything checks OK, so we can test by backing up the two hosts:

-bash-4.2$ barman backup db1
Starting backup using postgres method for server db1 in /var/lib/barman/db1/base/20180414T091155
Backup start at LSN: 0/240001B0 (000000010000000000000024, 000001B0)
Starting backup copy via pg_basebackup for 20180414T091155
Copy done (time: 2 seconds)
Finalising the backup.
This is the first backup for server db1
WAL segments preceding the current backup have been found:
      000000010000000000000023 from server db1 has been removed
Backup size: 201.9 MiB
Backup end at LSN: 0/26000000 (000000010000000000000025, 00000000)
Backup completed (start time: 2018-04-14 09:11:55.783708, elapsed time: 2 seconds)
Processing xlog segments from file archival for db1
      000000010000000000000023
      000000010000000000000024
      000000010000000000000025.00000028.backup
Processing xlog segments from streaming for db1
      000000010000000000000024

-bash-4.2$ barman backup db2
Starting backup using postgres method for server db2 in /var/lib/barman/db2/base/20180414T091225
Backup start at LSN: 0/B0000D0 (00000001000000000000000B, 000000D0)
Starting backup copy via pg_basebackup for 20180414T091225
Copy done (time: 3 seconds)
Finalising the backup.
This is the first backup for server db2
WAL segments preceding the current backup have been found:
      000000010000000000000009 from server db2 has been removed
      00000001000000000000000A from server db2 has been removed
Backup size: 196.8 MiB
Backup end at LSN: 0/D000000 (00000001000000000000000C, 00000000)
Backup completed (start time: 2018-04-14 09:12:25.619005, elapsed time: 3 seconds)
Processing xlog segments from file archival for db2
      00000001000000000000000B
      00000001000000000000000C.00000028.backup
Processing xlog segments from streaming for db2
      00000001000000000000000B

List the backup catalog:

-bash-4.2$ barman list-backup all
db1 20180414T091155 - Sat Apr 14 09:11:58 2018 - Size: 217.9 MiB - WAL Size: 0 B
db2 20180414T091225 - Sat Apr 14 09:12:28 2018 - Size: 212.8 MiB - WAL Size: 0 B

Displaying the contents for a particular backup:

-bash-4.2$ barman list-files db1 20180414T091155 | head
/var/lib/barman/db1/base/20180414T091155/backup.info
/var/lib/barman/db1/base/20180414T091155/data/backup_label
/var/lib/barman/db1/base/20180414T091155/data/PG_VERSION
/var/lib/barman/db1/base/20180414T091155/data/postgresql.auto.conf
/var/lib/barman/db1/base/20180414T091155/data/pg_ident.conf
/var/lib/barman/db1/base/20180414T091155/data/postgresql.conf
/var/lib/barman/db1/base/20180414T091155/data/pg_hba.conf

When Barman was configured for synchronous WAL streaming we can verify the replication status:

-bash-4.2$ barman replication-status db1
Status of streaming clients for server 'db1':
Current LSN on master: 0/26000528
Number of streaming clients: 1

1. Async WAL streamer
   Application name: barman_receive_wal
   Sync stage      : 3/3 Remote write
   Communication   : TCP/IP
   IP Address      : 10.1.9.231 / Port: 37278 / Host: -
   User name       : streaming_barman
   Current state   : streaming (async)
   Replication slot: barman
   WAL sender PID  : 2046
   Started at      : 2018-04-14 09:04:03.019323+00:00
   Sent LSN   : 0/26000528 (diff: 0 B)
   Write LSN  : 0/26000528 (diff: 0 B)
   Flush LSN  : 0/26000000 (diff: -1.3 KiB)

Further enhancements can be added using the provided hook scripts.

Finally, for command line lovers, Barman comes with full TAB completion.

EDB Backup and Recovery Tool (BART)

EDB BART is a closed source proprietary application provided by EnterpriseDB. It combines the PostgreSQL native Filesystem Level Backup and PITR into an easy to use tool providing the following features:

• retention policies
• incremental backups
• complete, hot, physical backups of multiple Postgres Plus Advanced Server and PostgreSQL database servers
• backup and recovery management of the database servers on local or remote hosts
• centralized catalog for backup data
• store backup data in compressed format
• checksum verification

While the trial version for the latest version v2.1 is not freely available, the article Data Backup Made Easy and the product documentation guide offer some information for those curious to learn more.

pgBackRest

pgBackRest implements a full system backup that doesn’t rely on the common tools tar and rsync. It is currently hosted and made available by CrunchyData under an MIT license. See Recognition for details on its origins.

It offers all the features one would expect from a PostgreSQL centric tool:

• high backup/restore throughput
• full, incremental, and differential backups
• retention policies
• backup and restore integrity verification through file checksums and integration with PostgreSQL page checksums.
• ability to resume backups
• streaming compression and checksums
• Amazon S3 cloud storage support
• Encryption

..and much more. Refer to the project page for details.

The installation requires a 64-bit Linux/Unix system and it is outlined in the user guide. The guide also introduces the reader to the main concepts, very useful to those new to PostgreSQL or storage technology.

Although the guide uses command examples for Debian/Ubuntu the pgBackRest is available in the PGDG yum repository, and the installer will pull in all the dependencies:

Installing:

pgbackrest       x86_64  2.01-1.rhel7     pgdg10  36k

Installing       for     dependencies:
perl-DBD-Pg      x86_64  2.19.3-4.el7     base    195k
perl-DBI         x86_64  1.627-4.el7      base    802k
perl-Digest-SHA  x86_64  1:5.85-4.el7     base    58k
perl-JSON-PP     noarch  2.27202-2.el7    base    55k
perl-Net-Daemon  noarch  0.48-5.el7       base    51k
perl-PlRPC       noarch  0.2020-14.el7    base    36k
perl-XML-LibXML  x86_64  1:2.0018-5.el7   base    373k
perl-version     x86_64  3:0.99.07-2.el7  base    84k

Let’s setup two clusters, pg96 and pg10, each having one node:

• control node (“repository” in the guide):

  [root@cc ~]# cat /etc/pgbackrest.conf
  [global]
  repo1-path=/var/lib/pgbackrest
  repo1-retention-full=2
  start-fast=y
  
  [pg96]
  pg1-path=/var/lib/pgsql/9.6/data
  pg1-host=db1
  pg1-host-user=postgres
  
  [pg10]
  pg1-path=/var/lib/pgsql/10/data
  pg1-host=db2
  pg1-host-user=postgres

• cluster #1:

  [root@db-1 ~]# cat /etc/pgbackrest.conf
  [global]
  log-level-file=detail
  repo1-host=repository
  
  [pg96]
  pg1-path=/var/lib/pgsql/9.6/data

• cluster #2:

  [root@db-2 ~]# cat /etc/pgbackrest.conf
  [global]
  log-level-file=detail
  repo1-host=repository
  
  [pg10]
  pg1-path=/var/lib/pgsql/10/data

Next, run backups and display the backup catalog:

-bash-4.2$ pgbackrest --stanza=pg96 info
stanza: pg96
   status: ok

   db (current)
      wal archive min/max (9.6-1): 00000001000000000000003D / 00000001000000000000003D

      full backup: 20180414-120727F
            timestamp start/stop: 2018-04-14 12:07:27 / 2018-04-14 12:08:01
            wal start/stop: 00000001000000000000003D / 00000001000000000000003D
            database size: 185.6MB, backup size: 185.6MB
            repository size: 12.1MB, repository backup size: 12.1MB
-bash-4.2$ pgbackrest --stanza=pg10 info
stanza: pg10
   status: ok

   db (current)
      wal archive min/max (10-1): 000000010000000000000012 / 000000010000000000000012

      full backup: 20180414-120810F
            timestamp start/stop: 2018-04-14 12:08:10 / 2018-04-14 12:08:38
            wal start/stop: 000000010000000000000012 / 000000010000000000000012
            database size: 180.5MB, backup size: 180.5MB
            repository size: 11.6MB, repository backup size: 11.6MB

pgBackRest supports parallelizing of backup and restore — following the example in the guide, we are backing with one CPU and then update the config to use 2 CPUs:

--- a/etc/pgbackrest.conf
+++ b/etc/pgbackrest.conf
@@ -2,6 +2,7 @@
repo1-path=/var/lib/pgbackrest
repo1-retention-full=2
start-fast=y
+process-max=2

[pg96]
pg1-host=db1

The result:

-bash-4.2$ pgbackrest --stanza=pg96 info
stanza: pg96
    status: ok

    db (current)
        wal archive min/max (9.6-1): 00000001000000000000003D / 000000010000000000000041

        full backup: 20180414-120727F
            timestamp start/stop: 2018-04-14 12:07:27 / 2018-04-14 12:08:01
            wal start/stop: 00000001000000000000003D / 00000001000000000000003D
            database size: 185.6MB, backup size: 185.6MB
            repository size: 12.1MB, repository backup size: 12.1MB

        incr backup: 20180414-120727F_20180414-121434I
            timestamp start/stop: 2018-04-14 12:14:34 / 2018-04-14 12:14:52
            wal start/stop: 00000001000000000000003F / 00000001000000000000003F
            database size: 185.6MB, backup size: 8.2KB
            repository size: 12.1MB, repository backup size: 431B
            backup reference list: 20180414-120727F

        incr backup: 20180414-120727F_20180414-121853I
            timestamp start/stop: 2018-04-14 12:18:53 / 2018-04-14 12:19:08
            wal start/stop: 000000010000000000000041 / 000000010000000000000041
            database size: 185.6MB, backup size: 8.2KB
            repository size: 12.1MB, repository backup size: 429B
            backup reference list: 20180414-120727F

With 2 CPUs the backup ran almost 20% faster which can make a big difference when running against a large data set.

Conclusion

PostgreSQL centric backup tools offer, as expected, more options than general purpose tools. Most PostgreSQL backup tools offer the same core functionality, but their implementation introduces limitations that can only be discovered by carefully following the documentation to test drive the product.

In addition, ClusterControl offers an array of backup and restore features that you can use as part of your database management setup.

Pgpool is less actual today, than it used to be 10 years ago, when it was the default part of a production PostgreSQL set up. Often when somebody was talking about PostgreSQL cluster, they were referring to postgreSQL behind pgpool and not to the PostgreSQL instance itself (which is the right term). Pgpool is recognised between most influential Postgres players: postgresql community, commandprompt, 2ndquadrant, EDB, citusdata, postgrespro (ordered by age, not influence). I realize the level of recognition in my links is very different - I just want to emphasize the overall impact of pgpool in the postgres world. Some of the most known current postgres “vendors” were found after the pgpool was already famous. So what makes it so famous?

Just the list of most in-demand offered features makes it look great:

• native replication
• connection pooling
• load balancing for read scalability
• high availability (watchdog with virtual IP, online recovery & failover)

Well, let’s make a sandbox and play. My sample setup is master slave mode. I would assume it is the most popular today, because you typically use streaming replication together with load balancing. Replication mode is barely used these days. Most DBAs skip it in favour to streaming replication and pglogical, and previously to slony.

The replication mode has many interesting settings and surely interesting functionality. But most DBAs have master/multi slave setup by the time they get to pgpool. So they are looking for automatic failover and load balancer, and pgpool offers it out of the box for existing master/multi slave environments. Not to mention that as from Postgres 9.4, streaming replication works with no major bugs and from 10 hash indexes replication is supported, so there are barely anything to stop you from using it. Also streaming replication is asynchronous by default (configurable to synchronous and even not “linear” synchronization complicated setups, while native pgpool replication is synchronous (which means slower data changes) with no choice option. Also additional limitations apply. Pgpool manual itself suggests to prefer when possible streaming replication over pgpool native one). And so this is my choice here.

Ah, but first we need to install it - right?

Installation (of higher version on ubuntu).

First checking the ubuntu version with lsb_release -a. For me repo is:

root@u:~# sudo add-apt-repository 'deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | \
>   sudo apt-key add -
OK
root@u:~# sudo apt-get update

Lastly installation itself:

sudo apt-get install pgpool2=3.7.2-1.pgdg16.04+1

Config:

I user default config from recommended mode:

zcat /usr/share/doc/pgpool2/examples/pgpool.conf.sample-stream.gz > /etc/pgpool2/pgpool.conf

Starting:

If you missed config, you see:

2018-03-22 13:52:53.284 GMT [13866] FATAL:  role "nobody" does not exist

Ah true - my bad, but easily fixable (doable blindly with one liner if you want the same user for all healthchecks and recovery):

root@u:~# sed -i s/'nobody'/'pgpool'/g /etc/pgpool2/pgpool.conf

And before we go any further, let’s create database pgpool and user pgpool in all clusters (In my sandbox they are master, failover and slave, so I need to run it on master only):

t=# create database pgpool;
CREATE DATABASE
t=# create user pgpool;
CREATE ROLE

At last - starting:

postgres@u:~$ /usr/sbin/service pgpool2 start
postgres@u:~$ /usr/sbin/service pgpool2 status
pgpool2.service - pgpool-II
   Loaded: loaded (/lib/systemd/system/pgpool2.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-04-09 10:25:16 IST; 4h 14min ago
     Docs: man:pgpool(8)
  Process: 19231 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
 Main PID: 8770 (pgpool)
    Tasks: 10
   Memory: 5.5M
      CPU: 18.250s
   CGroup: /system.slice/pgpool2.service
           ├─ 7658 pgpool: wait for connection reques
           ├─ 7659 pgpool: wait for connection reques
           ├─ 7660 pgpool: wait for connection reques
           ├─ 8770 /usr/sbin/pgpool -n
           ├─ 8887 pgpool: PCP: wait for connection reques
           ├─ 8889 pgpool: health check process(0
           ├─ 8890 pgpool: health check process(1
           ├─ 8891 pgpool: health check process(2
           ├─19915 pgpool: postgres t ::1(58766) idl
           └─23730 pgpool: worker proces

Great - so we can proceed to the first feature - let’s check load balancing. It has some requirements to be used, supports hints (e.g. to balance in same session), has black-and-white-listed functions, has regular expressions based redirect preference list. It is sophisticated. Alas goingf thoroughly over all that functionality would be out of the scope of this blog, thus we will check the simplest demos:

First, something very simple will show which node is used for select (in my setup, master spins on 5400, slave on 5402 and failover on 5401, while pgpool itself is on 5433, as I have another cluster running and did not want to interfere with it):

vao@u:~$ psql -h localhost -p 5433 t -c "select current_setting('port') from ts limit 1"
 current_setting
-----------------
 5400
(1 row)

Then in loop:

vao@u:~$ (for i in $(seq 1 99); do psql -h localhost -p 5433 t -c "select current_setting('port') from ts limit 1" -XAt; done) | sort| uniq -c
      9 5400
     30 5401
     60 5402

Great. It definitely balances load between nodes, but seems to balance not equally - maybe it’s so smart it knows the weight of each statement? Let’s check the distribution with expected results:

t=# show pool_nodes;
 node_id | hostname  | port | status | lb_weight |  role   | select_cnt | load_balance_node | replication_delay
---------+-----------+------+--------+-----------+---------+------------+-------------------+-------------------
 0       | localhost | 5400 | up     | 0.125000  | primary | 122        | false             | 0
 1       | localhost | 5401 | up     | 0.312500  | standby | 169        | false             | 0
 2       | localhost | 5402 | up     | 0.562500  | standby | 299        | true              | 0
(3 rows)

No - pgpool does not analyze the weight of statements - it was a DBA with her settings again! The settings (see the lb_weight attribute) reconciles with actual query destination targets. You can easily change it (as we did here) by changing the corresponding setting, eg:

root@u:~$ grep weight /etc/pgpool2/pgpool.conf
backend_weight0 =0.2
backend_weight1 = 0.5
backend_weight2 = 0.9
root@u:~# sed -i s/'backend_weight2 = 0.9'/'backend_weight2 = 0.2'/ /etc/pgpool2/pgpool.conf
root@u:~# grep backend_weight2 /etc/pgpool2/pgpool.conf
backend_weight2 = 0.2
root@u:~# pgpool reload
root@u:~$ (for i in $(seq 1 9); do psql -h localhost -p 5433 t -c "select current_setting('port') from ts limit 1" -XAt; done) | sort| uniq -c
      6 5401
      3 5402

Great! The next great feature offered is connection pooling. With 3.5 the “thundering herd problem” is solved by serializing accept() calls, greatly speeding up “client connection” time. And yet this feature is pretty straightforward. It does not offer several levels of pooling or several pools configured for the same database (pgpool lets you to choose where to run selects with database_redirect_preference_list of load balancing though), or other flexible features offered by pgBouncer.

So short demo:

t=# select pid,usename,backend_type, state, left(query,33) from pg_stat_activity where usename='vao' and pid <> pg_backend_pid();
 pid  | usename |  backend_type  | state |     left
------+---------+----------------+-------+--------------
 8911 | vao     | client backend | idle  |  DISCARD ALL
 8901 | vao     | client backend | idle  |  DISCARD ALL
 7828 | vao     | client backend | idle  |  DISCARD ALL
 8966 | vao     | client backend | idle  |  DISCARD ALL
(4 rows)
Hm - did I set up this little number of children?
t=# pgpool show num_init_children;
 num_init_children
-------------------
 4
(1 row)

Ah, true, I changed them lower than default 32, so the output would not take several pages. Well then, let’s try exceeding the number of sessions (below I open postgres sessions async in loop, so the 6 sessions would be requested at more or less the same time):

vao@u:~$ for i in $(seq 1 6); do (psql -h localhost -p 5433 t -U vao -c "select pg_backend_pid(), pg_sleep(1), current_setting('port'), clock_timestamp()" &);  done
vao@u:~$  pg_backend_pid | pg_sleep | current_setting |        clock_timestamp
----------------+----------+-----------------+-------------------------------
           8904 |          | 5402            | 2018-04-10 12:46:55.626206+01
(1 row)

 pg_backend_pid | pg_sleep | current_setting |        clock_timestamp
----------------+----------+-----------------+-------------------------------
           9391 |          | 5401            | 2018-04-10 12:46:55.630175+01
(1 row)

 pg_backend_pid | pg_sleep | current_setting |       clock_timestamp
----------------+----------+-----------------+------------------------------
           8911 |          | 5400            | 2018-04-10 12:46:55.64933+01
(1 row)

 pg_backend_pid | pg_sleep | current_setting |        clock_timestamp
----------------+----------+-----------------+-------------------------------
           8904 |          | 5402            | 2018-04-10 12:46:56.629555+01
(1 row)

 pg_backend_pid | pg_sleep | current_setting |        clock_timestamp
----------------+----------+-----------------+-------------------------------
           9392 |          | 5402            | 2018-04-10 12:46:56.633092+01
(1 row)

 pg_backend_pid | pg_sleep | current_setting |       clock_timestamp
----------------+----------+-----------------+------------------------------
           8910 |          | 5402            | 2018-04-10 12:46:56.65543+01
(1 row)

It lets sessions to come by three - expected, as one is taken by the above session (selecting from pg_stat_activity) so 4-1=3. As soon as pg_sleep finishes its one second nap and session is closed by postgres, the next one is let in. So after the first three ends, the next three step in. What happens to the rest? They are queued until the next connection slot frees up. Then the process described next to serialize_accept happens and client gets connected.

Huh? Just session pooling in session mode? Is it all?.. No, here the caching steps in! Look.:

postgres=# /*NO LOAD BALANCE*/ select 1;
 ?column?
----------
        1
(1 row)

Checking the pg_stat_activity:

postgres=# select pid, datname, state, left(query,33),state_change::time(0), now()::time(0) from pg_stat_activity where usename='vao' and query not like '%DISCARD%';
  pid  | datname  | state |               left                | state_change |   now
-------+----------+-------+-----------------------------------+--------------+----------
 15506 | postgres | idle  | /*NO LOAD BALANCE*/ select 1, now | 13:35:44     | 13:37:19
(1 row)

Then run the first statement again and observe state_change not changing, which means you don’t even get to the database to get a known result! Of course if you put some mutable function, results won’t be cached. Experiment with:

postgres=# /*NO LOAD BALANCE*/ select 1, now();
 ?column? |             now
----------+------------------------------
        1 | 2018-04-10 13:35:44.41823+01
(1 row)

You will find that state_change changes as does the result.

Last point here - why /*NO LOAD BALANCE*/ ?.. to be sure we check pg_stat_activity on master and run query on master as well. Same you can use /*NO QUERY CACHE*/ hint to avoid getting a cached result.

Already much for a short review? But we did not even touch the HA part! And many users look towards pgpool specifically for this feature. Well, this is not the end of the story, this is the end of part one. Part two is coming, where we will briefly cover HA and some other tips on using pgpool...

What is Slony?

Slony-I (referred to as just ‘Slony’ from here on out) is a third-party replication system for PostgreSQL that dates back to before version 8.0, making it one of the older options for replication available. It operates as a trigger-based replication method that is a ‘master to multiple slaves’ solution.

Slony operates by installing triggers on each table to be replicated, on both master and slaves, and every time the table gets an INSERT, UPDATE, or DELETE, it logs which record gets changed, and what the change is. Outside processes, called the ‘slon daemons’, connect to the databases as any other client and fetch the changes from the master, then replay them on all slave nodes subscribed to the master. In a well performing replication setup, this asynchronous replication can be expected to be anywhere 1 to 20 seconds lagged behind the master.

As of this writing, the latest version of Slony is at version 2.2.6, and supports PostgreSQL 8.3 and above. Support continues to this day with minor updates, however if a future version of PostgreSQL changes fundamental functionality of transactions, functions, triggers, or other core features, the Slony project may decide to discontinue large updates to support such drastic new approaches.

PostgreSQL’s mascot is an elephant known as ‘Slonik’, which is Russian for ‘little elephant’. Since this replication project is about many PostgreSQL databases replicating with each other, the Russian word for elephants (plural) is used: Slony.

Concepts

• Cluster: An instance of Slony replication.
• Node: A specific PostgreSQL database as Slony replication node, which operates as either a master or slave for a replication set.
• Replication Set: A group of tables and / or sequences to be replicated.
• Subscribers: A subscriber is a node that is subscribed to a replication set, and receives replication events for all tables and sequences within that set from the master node.
• Slony Daemons: The main workers that execute replication, a Slony daemon is kicked off for every node in the replication set and establishes various connections to the node it manages, as well as the master node.

How it is Used

Slony is installed either by source or through the PGDG (PostgreSQL Global Development Group) repositories which are available for Red Hat and Debian based linux distributions. These binaries should be installed on all hosts that will contain either a master or slave node in the replication system.

After installation, a Slony replication cluster is set up by issuing a few commands using the ‘slonik’ binary. ‘slonik’ is a command with a simple, yet unique syntax of its own to initialize and maintain a slony cluster. It is the main interface for issuing commands to the running Slony cluster that is in charge of replication.

Interfacing with Slony can be done by either writing custom slonik commands, or compiling slony with the --with-perltools flag, which provides the ‘altperl’ scripts that help generate these slonik scripts needed.

Creating a Slony Replication Cluster

A ‘Replication Cluster’ is a collection of databases that are part of replication. When creating a replication cluster, an init script needs to be written that defines the following:

• The name of the Slony cluster desired
• The connection information for each node part of replication, each with an immutable node number.
• Listing all tables and sequences to be replicated as part of a ‘replication set’.

An example script can be found in Slony’s official documentation.

When executed, slonik will connect to all nodes defined and create the Slony schema on each. When the Slony daemons are kicked off, they will then clear out all data in the replicated tables on the slave (if there is any), and copy over all data from the master to the slave(s). From that point on, the daemons will continually replicate changes recorded on the master to all subscribed slaves.

Clever configurations

While Slony is initially a Master-to-Multiple-Slave replication system, and has mainly been used in that way, there are several other features and clever usages that make Slony more useful than a simple replication solution. The highly customizable nature of Slony keeps it relevant for a variety of situations for administrators that can think outside of the box.

Cascading Replication

Slony nodes can be set up to cascade replication down a chain of different nodes. If the master node is known to take an extremely heavy load, each additional slave will increase that load slightly. With cascading replication, a single slave node connected to the master can be configured as a ‘forwarding node’, which will then be responsible to sending replication events to more slaves, keeping the load on the master node to a minimum.

Cascading Replication with Slony

Data processing on a slave node

Unlike PostgreSQL’s built in replication, the slave nodes are not actually ‘read only’, only the tables that are being replicated are locked down as ‘read only’. This means on a slave node, data processing can take place by creating new tables not part of replication to house processed data. Tables part of replication can also have custom indexes created depending on the access patterns that may differ from the slave and the master.

Read only tables on the slaves can still have custom trigger based functions executed on data changes, allowing more customization with the data.

Data Processing on a Slony Slave Node

Minimal downtime Upgrades

Upgrading major versions of PostgreSQL can be extremely time consuming. Depending on data size and table count, an upgrade including the ‘analyze’ process post-upgrade could take several days even. Since Slony can replicate data between PostgreSQL clusters of different versions, it can be used to set up replication between an older version as the master and a newer version as the slave. When the upgrade is to happen, simply perform a ‘switchover’, making the slave the new master, and the old master becomes the slave. When the upgrade is marked a success, decommission the Slony replication cluster and shut down the old database.

Upgrade PostgreSQL with Minimal Downtime using Slony

High Availability with frequent server maintenance

Like the minimal downtime for upgrades, server maintenance can be done easily with no downtime by performing a ‘switchover’ between two or more nodes, allowing a slave to be rebooted with updates / other maintenance. When the slave comes back online, it will re-connect to the replication cluster and catch up on all the replicated data. End users connecting to the database may have long transactions disrupted, but downtime itself would be seconds as the switchover happens, rather than however long it takes to reboot / update the host.

Log Shipping

Though not likely to be popular solution, a Slony slave can be set up as a ‘log shipping’ node, where all data it receives through replication can be written to SQL files, and shipped. This can be used for a variety of reasons, such as writing to an external drive and transporting to a slave database manually, and not over a network, compressed and kept archived for future backups, or even have an external program parse the SQL files and modify the contents.

Multiple database data sharing

Since any number of tables can be replicated at will, Slony replication sets can be set up to share specific tables between databases. While similar access can be achieved through Foreign Data Wrappers (which have improved in recent PostgreSQL releases), it may be a better solution to use Slony depending on the usage. If a large amount of data is needed to be fetched from one host to another, having Slony replicate that data means the needed data will already exist on the requesting node, eliminating long transfer time.

Delayed Replication

Usually replication is desired to be as quick as possible, but there can be some scenarios where a delay is desired. The slon daemon for a slave node can be configured with a lag_interval, meaning it won’t receive any replication data until the data is old as specified. This can be useful for quick access to lost data if something goes wrong, for example if a row is deleted, it will exist on the slave for 1 hour for quick retrieval.

Things to Know:

• Any DDL changes to tables that are part of replication must be executed using the slonik execute command.
• Any table to be replicated must have either a primary key, or a UNIQUE index without nullable columns.
• Data that is replicated from the master node is replicated after any data may have been functionally generated. Meaning if data was generated using something like ‘random()’, the resulting number is stored and replicated on the slaves, rather than ‘random()’ being run again on the slave returning a different result.
• Adding Slony replication will increase server load slightly. While efficiently written, each table will have a trigger that logs each INSERT, UPDATE, and DELETE to a Slony table, expect about 2-10% server load increase, depending on database size and workload.

Tips and Tricks:

• Slony daemons can run on any host that has access to all other hosts, however the highest performing configuration is to have the daemons run on the nodes they are managing. For example, the master daemon running on the master node, the slave daemon running on the slave node.
• If setting up a replication cluster with a very large amount of data, the initial copy can take quite a long time, meaning all changes that happen from kickoff till the copy is done could mean even longer to catch up and be in sync. This can be solved by either adding a few tables at a time to replication (very time consuming), or by creating a data directory copy of the master database to the slave, then doing a ‘subscribe set’ with the OMIT COPY option set to true. With this option, Slony will assume that the slave table is 100% identical to the master, and not clear it out and copy data over.
• The best scenario for this is to create a Hot Standby using the built in tools for PostgreSQL , and during a maintenance window with zero connections modifying data, bring the standby online as a master, validate data matches between the two, initiate the slony replication cluster with OMIT COPY = true, and finally re-enable client connections. This may take time to do the setup for the Hot Standby, but the process won’t cause huge negative impact to clients.

Community and Documentation

The community for Slony can found in the mailing lists, located at http://lists.slony.info/mailman/listinfo/slony1-general, which also includes archives.

Documentation is available on the official website, http://slony.info/documentation/, and provides help with log analysis, and syntax specification for interfacing with slony.