If you maintain some PostgreSQL extensions which rely on PGXS, a build infrastructure for PostgreSQL, the following commit added to Postgres 12 will be likely something interesting, because it adds new options to control more types of regression tests:

commit: d3c09b9b1307e022883801000ae36bcb5eef71e8
author: Michael Paquier <michael@paquier.xyz>
date: Mon, 3 Dec 2018 09:27:35 +0900
committer: Michael Paquier <michael@paquier.xyz>
date: Mon, 3 Dec 2018 09:27:35 +0900
Add PGXS options to control TAP and isolation tests, take two

The following options are added for extensions:
- TAP_TESTS, to allow an extention to run TAP tests which are the ones
present in t/*.pl.  A subset of tests can always be run with the
existing PROVE_TESTS for developers.
- ISOLATION, to define a list of isolation tests.
- ISOLATION_OPTS, to pass custom options to isolation_tester.

A couple of custom Makefile rules have been accumulated across the tree
to cover the lack of facility in PGXS for a couple of releases when
using those test suites, which are all now replaced with the new flags,
without reducing the test coverage.  Note that tests of contrib/bloom/
are not enabled yet, as those are proving unstable in the buildfarm.

Author: Michael Paquier
Reviewed-by: Adam Berlin, Álvaro Herrera, Tom Lane, Nikolay Shaplov,
Arthur Zakirov
Discussion: https://postgr.es/m/20180906014849.GG2726@paquier.xyz

This is similar rather to the existing REGRESS and REGRESS_OPTS which allow to respectively list a set of regression tests and pass down additional options to pg_regress (like a custom configuration file). When it comes to REGRESS, input files need to be listed in sql/ and expected output files are present in expected/, with items listed without a dedicated “.sql” suffix.

The new options ISOLATION and ISOLATION_OPTS added in PostgreSQL 12 are similar to REGRESS and REGRESS_OPTS, except that they can be used to define a set of tests to stress the behavior of concurrent sessions, for example for locking checks across commands, etc. PostgreSQL includes in its core tree the main set of isolation tests in src/test/isolation/, and has also a couple of modules with their own custom set:

• contrib/test_decoding/
• src/test/modules/snapshot_too_old/

To add such tests, a set of tests suffixed with “.spec” need to be added to a subdirectory of the module called specs/, matching with entries listed in ISOLATION without the suffix. Output files are listed in an subdirectory called expected/, suffixed with “.out” similarly to tests listed in REGRESS.

Another new option available is called TAP_TESTS, which allows a module to define if TAP tests need to be run. This is described extensively in the documentation as a facility which uses the Perl TAP tools. All the tests present in the subdirectory t/ and using “.pl” as suffix will be run. Note that a subset of tests can be enforced with the variable PROVE_FLAGS, which is useful for development purposes. Some modules use it in the core code:

• contrib/oid2name/
• contrib/vacuumlo/
• src/test/modules/brin/
• src/test/modules/commit_ts/
• src/test/modules/test_pg_dump/

The main advantage of those switches is to avoid custom rules in one’s extension Makefile. Here is for example what has been used by many modules, including a couple of tools that I have developed and/or now maintain:

check:
    $(prove_check)

installcheck:
    $(prove_installcheck)

One of the main benefits is a huge cleanup of all the makefiles included in PostgreSQL core code which have accumulated custom rules across many releases, as well as better long-term maintenance of all in-core as well as out-of-core modules which depend on PGXS, so that is really a nice addition for the project.

There are three resources that affect query performance: CPU, memory, and storage. Allocating CPU and storage for optimal query performance is straightforward, or at least linear. For CPU, for each query you must decide the optimal number of CPUs to allocate for parallel query execution. Of course, only certain queries can benefit from parallelism, and the optimal number of CPU changes based on the other queries being executed. So, it isn't simple, but it is linear.

For storage, it is more of a binary decision — should the table or index be stored on fast media (SSDs) or slow media (magnetic disks), particularly fast random access. (Some NAS servers have even more finely-grained tiered storage.) It takes analysis to decide the optimal storage type for each table or index, but the Postgres statistic views can help to identify which objects would benefit.

Unfortunately, for memory, resource allocation is more complex. Rather than being a linear or binary problem, it is a multi-dimensional problem — let me explain. As stated above, for CPUs you have to decide how many CPUs to use, and for storage, what type. For memory, you have to decide how much memory to allocate to shared_buffers at database server start, and then decide how much memory to allocate to each query for sorting and hashing via work_mem. What memory that is not allocated gets used as kernel cache, which Postgres relies on for consistent performance (since all reads and writes go through that cache). So, to optimize memory allocation, you have to choose the best sizes for:

1. shared buffers (fixed size at server start)
2. work_mem (potentially optimized per query based on workload)
3. kernel buffers (the remainder)

Continue Reading »

How we evaluated several third-party tools and ultimately selected Patroni as our preferred method.

If you are working with a database, you already know that high availability (HA) is a requirement for any reliable system. A proper HA setup eliminates the problem of a single point-of-failure by providing multiple places to get the same data, and an automated way of selecting a proper location with which to read and write data. PostgreSQL typically achieves HA by replicating the data on the primary database to one or more read-only replicas.

Streaming replication is the primary method of replication supported by TimescaleDB. It works by having the primary database server stream its write-ahead log (WAL) entries to its database replicas. Each replica then replays these log entries against its own database to reach a state consistent with the primary database.

Unfortunately, such streaming replication alone does not ensure that users don’t encounter some loss of uptime (“availability”) when a server crashes. If the primary node fails or becomes unreachable, there needs to be a method to promote one of the read replicas to become the new primary. And the faster that this failover occurs, the smaller the window of unavailability a user will see. One simple approach to such promotion is by manually performing it through a few PostgreSQL commands. On the other hand, automatic failover refers to a mechanism by which the system detects when a primary has failed, fully removes it from rotation (sometimes called fencing), promotes one read replica to primary status, and ensures the rest of the system is aware of the new state.

While PostgreSQL supports HA through its various replication modes, automatic failover is not something it supports out of the box. There are a number of third-party solutions that provide HA and automatic failover for PostgreSQL, many of which work with TimescaleDB because they also leverage streaming replication.

In this post, we will discuss how we evaluated third-party failover solutions and share deployment information about the one that best fits with our requirements and constraints. We will also provide a series of HA/failover scenarios that users may encounter and how to deal with these scenarios.

Requirements and constraints

When looking for a failover solution to adopt, we decided that it should meet the following requirements (in no particular order):

• Does not require manual intervention for failover
• Supports streaming replication (TimescaleDB is not currently compatible with logical replication, so methods relying on this or other custom protocols did not make the cut)
• Supports different modes of replication (i.e. async vs synchronous commit)
• Handles network partitions (commonly called “split brain,” where multiple instances think they are the primary databases and different clients may write to different servers, leading to inconsistencies)
• Provides operational simplicity
• Supports load balancing the read replicas
• Plays nicely with cloud providers and containerized deployments
• Uses a known/understood algorithm for establishing consensus (This is needed to ensure that the entire system agrees on which node the primary is. Many solutions will offload the consensus work to a system that already has a robust implementation of a known consensus algorithm. Patroni and Stolon, for example, use an etcd cluster, which implements the Raft consensus algorithm, to manage the system’s state.)
• Supports the primary rejoining the cluster as a replica after being demoted and can become the primary again (failback)
• Can be managed manually through some kind of interface (CLI, HTTP, GUI, etc.)
• Has meaningful community adoption
• Released under a permissive license

Evaluation of solutions

To begin, we evaluated several popular HA/failover solutions and their support for our requirements. Some solutions were closed source and not free, so they were not considered. Others were disqualified because they did not support streaming replication or different replication modes.

A handful of the requirements (community adoption, interface for management, no manual intervention for failover) were met by all options considered. We decided to rule out solutions that involve determining the master by configuring a floating/virtual IP, which would not work well in many modern cloud environments without jumping through some operational hoops. Additionally, we eliminated Corosync and Pacemaker since they were older products that required extra complexity and specialized knowledge.

We concluded that Stolon and Patroni both were good solutions. Their architectures are both easy to understand which is in no small part thanks to the fact that they offload all of the consensus heavy-lifting to etcd/consul. Adoption wise, they appear to be on equal footing, at least by the metric of Github stars, followers, and forks. There is also a good deal of documentation and sample configurations/scripts for both of them.

Ultimately we decided to choose Patroni because of its slightly simpler architecture and more modular approach to load balancing. Where Stolon requires you to use its own stolon-proxy nodes for load balancing, Patroni exposes health checks on the PostgreSQL nodes and lets you use Kubernetes, AWS ELB, HA Proxy, or another proxy solution to deal with load balancing. Where Stolon requires an additional layer of standalone sentinel and proxy services, Patroni is able to offer the same HA guarantees with “bots” on the PostgreSQL instances communicating with an etcd cluster.

Getting started with Patroni

The core architecture of Patroni looks like this:

Each PostgreSQL node has a Patroni bot deployed on it. The bots are capable both of managing the PostgreSQL database and updating the distributed consensus system (etcd in this case although Zookeeper, Consul, and the Kubernetes API, which is backed by etcd, are also perfectly fine options). Etcd must be deployed in an HA fashion that allows its individual nodes to reach a quorum about the state of the cluster. This requires a minimum deployment of 3 etcd nodes.

Leader election is handled by attempting to set an expiring key in etcd. The first PostgreSQL instance to set the etcd key via its bot becomes the primary. Etcd ensures against race conditions with a Raft-based consensus algorithm. When a bot receives acknowledgement that it has the key, it sets up the PostgreSQL instance as a primary. All other nodes will see that a primary has been elected and their bots will set their PostgreSQL instances up as replicas.

The primary key has a short TTLattached to it. The primary’s bot must continually health check its PostgreSQL instance and update the primary key. If this doesn’t happen because (a) the node loses connectivity, (b) the database fails, or (c) the bot dies, the key will expire. This will cause the primary to be fully removed from the system. When the replicas see that there is no primary key, each replica will attempt to become the primary unless it is obvious through exchanging information between bots that one of the primaries should be elected over the other. The bots will then either choose an obvious candidate for promotion or both race to be promoted to the primary.

You can find this flow illustrated below:

Common failover scenarios

There are a number of failover scenarios we made sure to test when fully evaluating failover functionality. Some of the key ones to keep in mind are:

• The primary database crashes or is shut down
• The server with the primary database crashes, is shut down, or is unreachable
• A failed primary thinks it is a primary after failover
• Replica databases crash or shut down
• The Patroni bot on the primary goes down
• The Patroni bot on the replica goes down
• A replica is added to the cluster

Next steps

Above, we shared what we believe to be the best automatic failover solution for TimescaleDB users. While there are many third party solutions available, ultimately we went with Patroni because it combined robust and reliable failover with a simple architecture and easy-to-use interface.

If you are new to TimescaleDB and ready to get started, follow the installation instructions. If you have questions, we encourage you to join 1500+ member-strong Slack community. If you are looking for enterprise-grade support and assistance, please let us know.

Like this post? Interested in learning more? Follow us on Twitter or sign up for the community mailing list below!

Evaluating High Availability Solutions for TimescaleDB + PostgreSQL was originally published in Timescale on Medium, where people are continuing the conversation by highlighting and responding to this story.

PostGIS 2.5.1 was released on November 18th 2018 and I finished off packaging the PostGIS 2.5.1 windows builds and installers targeted for PostgreSQL EDB distribution this weekend and pushing them up to stackbuilder. This covers PostgreSQL 9.4-11 64-bit and PostgreSQL 95-10 (32bit).

Note that PostGIS 2.5 series will be the last of the PostGIS 2s. Goodbye PostGIS 2.* and start playing with the in-development version of PostGIS 3. Snapshot binaries for PostGIS 3.0 windows development are also available on the PostGIS windows download page. These should work for both BigSQL and EDB distributions.

I have already discussed the complexities of allocating shared_buffers,work_mem, and the remainder as kernel cache. However, even if these could be optimally configured, work_mem (and its related setting maintenance_work_mem) are configured per query, and potentially can be allocated multiple times if multiple query nodes require it. So, even if you know the optimal amount of work_mem to allocate for the entire cluster, you still have to figure out how to allocate it among sessions.

This detailed email thread explores some possible ways to simplify this problem, but comes up with few answers. As stated, it is a hard problem, and to do it right is going to take a lot of work. Even DB2's solution to the problem was criticized, though it was our initial approach to solving the problem. With additional parallelism, configuring work_mem is getting even more complex for users, to say nothing of the complexity of allocating parallel workers itself.

This is eventually going to have to be addressed, and doing it in pieces is not going to be fruitful. It is going to need an entirely new subsystem, perhaps with dynamic characteristics unseen in any other Postgres modules.

Interested in running PostgreSQL natively on Kubernetes? Let's look at a few quick steps to get up and running with the open source Crunchy PostgreSQL Operator for Kubernetes on your choice of Kubernetes deployment.

The Crunchy PostgreSQL Operator (aka "pgo") provides a quickstart script to automate the deployment of the Crunchy PostgreSQL Operator to a number of popular Kubernetes environments, including Google Kubernetes Engine (GKE), OpenShift Container Platform (OCP) and Pivotal Container Service (PKS). You can download the quickstart script here:

• quickstart.sh

The script will deploy the operator to a GKE Kubernetes cluster or an OpenShift Container Platform cluster. The script assumes you have a StorageClass defined for persistence.

It's weird to do performance analysis of a database you run on your laptop. When testing some app, your local instance probably has 1/1000 the amount of realistic data compared to a production server. Or, you're running a bunch of end-to-end integration tests whose PostgreSQL performance doesn't make sense to measure.

Anyway, if you are doing some performance testing of an app that uses PostgreSQL one great tool to use is pghero. I use it for my side-projects and it gives me such nice insights into slow queries that I'm willing to live with the cost that it is to run it on a production database.

This is more of a brain dump of how I run it locally:

First, you need to edit your postgresql.conf. Even if you used Homebrew to install it, it's not clear where the right config file is. Start psql (on any database) and type this to find out which file is the one:

$ psql kintobench

kintobench=# show config_file;
               config_file
-----------------------------------------
 /usr/local/var/postgres/postgresql.conf
(1 row)

Now, open /usr/local/var/postgres/postgresql.conf and add the following lines:

# Peterbe: From Pghero's configuration help.
shared_preload_libraries = 'pg_stat_statements'
pg_stat_statements.track = all

Now, to restart the server use:

▶ brew services restart postgresql
Stopping `postgresql`... (might take a while)==> Successfully stopped `postgresql`(label: homebrew.mxcl.postgresql)==> Successfully started `postgresql`(label: homebrew.mxcl.postgresql)

The next thing you need is pghero itself and it's easy to run in docker. So to start, you need Docker for mac installed. You also need to know the database URL. Here's how I ran it:

docker run -ti -e DATABASE_URL=postgres://peterbe:@host.docker.internal:5432/kintobench -p 8080:8080 ankane/pghero

Note the trick of peterbe:@host.docker.internal because I don't use a password but inside the Docker container it doesn't know my terminal username. And the host.docker.internal is so the Docker container can reach the PostgreSQL installed on the host.

Once that starts up you can go to http://localhost:8080 in a browser and see a listing of all the cumulatively slowest queries. There are other cool features in pghero too that you can immediately benefit from such as hints about unused/redundent database indices.

Hope it helps!

Prepared transactions are disabled in PostgreSQL by default, since the parameter max_prepared_transactions has the default value 0.

You don’t need prepared transactions in most cases. However, they can cause nasty problems, so I think that everybody who runs a PostgreSQL database should understand them.

To illustrate these problems, I’ll show you how to use prepared transactions to get a PostgreSQL into an inaccessible state.

What are prepared transactions?

Normally, a database transaction that spans multiple statements is ended with COMMIT or ROLLBACK. With prepared transactions, another step is added:

1. BEGIN or START TRANSACTION: starts a transaction as usual.
2. PREPARE TRANSACTION 'name': prepares the transaction for commit or rollback and assigns a name to it.
3. { COMMIT | ROLLBACK } PREPARED 'name': commits or rolls back a previously prepared transaction.

The PREPARE TRANSACTION step performs all actions that may fail during COMMIT. That way, both COMMIT PREPARED and ROLLBACK PREPARED are guaranteed to succeed once a transaction is prepared. Moreover, PREPARE TRANSACTION persists the still open transaction, so that it will survive a crash or server restart.

Once a transaction is prepared, it is complete. Subsequent SQL statements belong to different transactions. You cannot do anything with a prepared transaction except COMMIT PREPARED and ROLLBACK PREPARED.

What is the use of prepared transactions?

Prepared transactions are used to implement “distributed transactions”.
Distributed transactions are transactions that affect more than one data source.
The protocol is as follows:

1. Start a transaction on all data sources involved in the transaction.
2. Modify data in the data sources. If there is a problem, ROLLBACK all involved transactions.
3. Once you are done, PREPARE all involved transactions.
4. If the PREPARE step fails in any of the transactions, issue ROLLBACK PREPARED everywhere.
5. If the PREPARE step succeeds everywhere, COMMIT PREPARED all involved transactions.

This so-called “two-phase commit protocol” guarantees that the distributed transaction either succeeds or is rolled back everywhere, leaving the whole system consistent.

To make that works reliably, you need a “distributed transaction manager”.
That is software that keeps track of all distributed transactions, persisting their state to survive crashes and other interruptions.
That way it can complete all interrupted distributed transactions as soon as operation is resumed.

Problems caused by prepared transactions

Normally, no transaction should be in the prepared state for longer than a split second. But software bugs and other disruptions can cause a transaction to remain in the prepared state for a longer time. This causes the problems associated with long running transactions in general:

• locks that are held for a long time, blocking other sessions and increasing the risk of a deadlock
• VACUUM cannot clean up dead tuples created after the the start of the transaction

These problems are exacerbated by the fact that prepared transactions and their locks stay around even after the database server is restarted.

Implementation details

Preparing a transaction will write a WAL record, so the prepared transaction can be restored during crash recovery. This requires forcing the WAL to disk, just like a normal commit does.

During a checkpoint, the state of the prepared transaction is persisted in a file in the pg_twophase subdirectory of the data directory. The name of the file is the hexadecimal transaction ID.

On startup, all prepared transactions are restored from pg_twophase.

The file is deleted when the prepared transaction is committed or rolled back.

Getting rid of “orphaned” prepared transactions

You can examine all prepared transactions in the PostgreSQL database cluster using the view pg_prepared_xacts.

If a prepared transaction is “orphaned” because the transaction manager failed to close it, you will have to do that manually. Connect to the correct database and run COMMIT PREPARED or ROLLBACK PREPARED.

Locking up a database with prepared transactions

Warning: Don’t try this on your production database!

As a database superuser, run the following:

BEGIN;

LOCK pg_catalog.pg_authid;

PREPARE TRANSACTION 'locked';

Then disconnect from the database.

pg_authid, the table that contains the database users, is required to authenticate a database session. Since this table is locked by the prepared transaction, all future connection attempts will hang.

Restarting the database won’t help, because the prepared transaction will be retained.

Before you read on to the next part that contains the solution, let me invite you to try and get out of this dilemma yourself.

Enter Houdini!

Your first reaction will probably be the same as mine: Start PostgreSQL in single user mode. Alas, no luck:

$ pg_ctl stop
waiting for server to shut down.... done
server stopped
$ postgres --single postgres
LOG:  recovering prepared transaction 571 from shared memory
^C
FATAL:  canceling statement due to user request

Single user mode just hangs until I send it a SIGINT by pressing Ctrl+C, which shuts down the server.

But we can easily find a way to recover by reviewing the implementation details above:
571 in hexadecimal is 23b, so while PostgreSQL is shut down, we can remove the prepared transaction as follows:

$ rm $PGDATA/pg_twophase/0000023B

This will essentially roll back the transaction, and its effects will be undone when PostgreSQL is restarted.

The post Be prepared for prepared transactions appeared first on Cybertec.

Register here:

A quick start guide on implementing and deploying code to PostgreSQL.

PostgreSQL 11 Server Side Programming - Now Available!

Near the end of November, Packt published the book PostgreSQL 11 Server Side Programming Quick Start Guide, authored by me.

*This post has the only aim of describing the book contents and the reason behind choices related to it.**

Following a consolidated tradition, Packt is producing more and more books on PostgreSQL and related technologies, and this is the first one that covers aspects about the freshly released PostgreSQL 11 version.

Nevertheless, this does not mean that the book is only for PostgreSQL 11 users and administrators: it covers topics, concepts and provide examples that can be use as-is or ported to older versions of PostgreSQL, as well as probably to newer ones. In fact, while the book code examples have been tested against a PostgreSQL 11 cluster, only the examples related to the new object PROCEDURE, introduced by PostgreSQL 11, are strongly tied to such a version.

This book is a Quick Start Guide, and therefore it has a very practical approach to a limited scope, and only that. Therefore the book assumes you are able to install, manage and run a PostgreSQL 11 cluster, that you know how to connect and how to handle basic SQL statements. A basic knowledge in general programming is also required.

The book consists of 10 chapters, each focusing on a particular aspect of developing and deploying code within a PostgreSQL cluster. The main programming language used in the book is PL/pgSQL, the default procedural language for PostgreSQL; however several examples are...

You might know that Postgres has optimizer hints listed as a feature we do not want to implement. I have covered this topic in the past.

This presentation from Tatsuro Yamada covers the trade-offs of optimizer hints in more detail than I have ever seen before. Starting on slide 29, Yamada-san explains the number-one reason to use optimizer hints — as a short-term fix for inefficient query plans. He uses pg_hint_plan (produced by NTT) to fix inefficient plans in his use case. He then goes on to consider possible non-hint solutions to inefficient plans, such as recording data distributions found during execution for use in the optimization of later queries.

Most interesting to me was his reproduction of the optimizer hints discussion on the Postgres wiki, including his analysis of how pg_hint_plan fits that criteria. There are certainly environments where optimizer hints are helpful, and it seems Yamada-san has found one. The reason the community does not plan to support hints is that it is considered likely that optimizer hints would cause more problems for users than they solve. While Postgres has some crude manual optimizer controls, it would certainly be good if Postgres could come up with additional solutions that further minimize the occurrence of significantly inefficient plans.

PostgreSQL streaming physical replication with slots simplifies setup and maintenance procedures. Usually, you should estimate disk usage for the Write Ahead Log (WAL) and provide appropriate limitation to the number of segments and setup of the WAL archive procedure. In this article, you will see how to use replication with slots and understand what problems it could solve.

Introduction

PostgreSQL physical replication is based on WAL. Th Write Ahead Log contains all database changes, saved in 16MB segment files. Normally postgres tries to keep segments between checkpoints. So with default settings, just 1GB of WAL segment files is available.

Replication requires all WAL files created after backup and up until the current time. Previously, it was necessary to keep a huge archive directory (usually mounted by NFS to all slave servers). The slots feature introduced in 9.4 allows Postgres to track the latest segment downloaded by a slave server. Now, PostgreSQL can keep all segments on disk, even without archiving, if a slave is seriously behind its master due to downtime or networking issues. The drawback: the disk space could be consumed infinitely in the case of configuration error. Before continuing, if you need a better understanding of physical replication and streaming replication, I recommend you read “Streaming Replication with PostgreSQL“.

Create a sandbox with two PostgreSQL servers

To setup replication, you need at least two PostgreSQL servers. I’m using pgcli (pgc) to setup both servers on the same host. It’s easy to install on Linux, Windows, and OS X, and provides the ability to download and run any version of PostgreSQL on your staging server or even on your laptop.

python -c "$(curl -fsSL https://s3.amazonaws.com/pgcentral/install.py)"
mv bigsql master
cp -r master slave
$ cd master
master$ ./pgc install pg10
master$ ./pgc start pg10
$ cd ../slave
slave$ ./pgc install pg10
slave$ ./pgc start pg10

First of all you should allow the replication user to connect:

master$ echo "host replication replicator 127.0.0.1/32 md5" >> ./data/pg10/pg_hba.conf

If you are running master and slave on different servers, please replace 127.0.0.1 with the slave’s address.

Next pgc creates a shell environment file with PATH and all the other variables required for PostgreSQL:

master$ source ./pg10/pg10.env

Allow connections from the remote host, and create a replication user and slot on master:

master$ psql
postgres=# CREATE USER replicator WITH REPLICATION ENCRYPTED PASSWORD 'replicator';
CREATE ROLE
postgres=# ALTER SYSTEM SET listen_addresses TO '*';
ALTER SYSTEM
postgres=# SELECT pg_create_physical_replication_slot('slot1');
pg_create_physical_replication_slot
-------------------------------------
(slot1,)

To apply system variables changes and hba.conf, restart the Postgres server:

master$ ./pgc stop ; ./pgc start
pg10 stopping
pg10 starting on port 5432

Test table

Create a table with lots of padding on the master:

master$ psql psql (10.6) Type "help" for help.
postgres=# CREATE TABLE t(id INT, pad CHAR(200));
postgres=# CREATE INDEX t_id ON t (id);
postgres=# INSERT INTO t SELECT generate_series(1,1000000) AS id, md5((random()*1000000)::text) AS pad;

Filling WAL with random data

To see the benefits of slots, we should fill the WAL with some data by running transactions. Repeat the update statement below to generate a huge amount of WAL data:

UPDATE t SET pad = md5((random()*1000000)::text);

Checking the current WAL size

You can check total size for all WAL segments from the shell or from psql:

master$ du -sh data/pg10/pg_wal
17M data/pg10/pg_wal
master$ source ./pg10/pg10.env
master$ psql
postgres=# \! du -sh data/pg10/pg_wal
17M data/pg10/pg_wal

Check maximum WAL size without slots activated

Before replication configuration, we can fill the WAL with random data and find that after 1.1G, the data/pg10/pg_wal directory size does not increase regardless of the number of update queries.

postgres=# UPDATE t SET pad = md5((random()*1000000)::text); -- repeat 4 times
postgres=# \! du -sh data/pg10/pg_wal
1.1G data/pg10/pg_wal
postgres=# UPDATE t SET pad = md5((random()*1000000)::text);
postgres=# \! du -sh data/pg10/pg_wal
1.1G data/pg10/pg_wal

Backup master from the slave server

Next, let’s make a backup for our slot1:

slave$ source ./pg10/pg10.env
slave$ ./pgc stop pg10
slave$ rm -rf data/pg10/*
# If you are running master and slave on different servers, replace 127.0.0.1 with master's IP address.
slave$ PGPASSWORD=replicator pg_basebackup -S slot1 -h 127.0.0.1 -U replicator -p 5432 -D $PGDATA -Fp -P -Xs -Rv

Unfortunately pg_basebackup hangs with: initiating base backup, waiting for checkpoint to complete.
We can wait for the next checkpoint, or force the checkpoint on the master. Checkpoint happens every checkpoint_timeout seconds, and is set to five minutes by default.

Forcing checkpoint on master:

master$ psql
postgres=# CHECKPOINT;

The backup continues on the slave side:

pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/92000148 on timeline 1
pg_basebackup: starting background WAL receiver
1073986/1073986 kB (100%), 1/1 tablespace
pg_basebackup: write-ahead log end point: 0/927FDDE8
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: base backup completed

The backup copies settings from the master, including its TCP port value. I’m running both master and slave on the same host, so I should change the port in the slave .conf file:

slave$ vim data/pg10/postgresql.conf
# old value port = 5432
port = 5433

Now we can return to the master and run some queries:

slave$ cd ../master
master$ source pg10/pg10.env
master$ psql
postgres=# UPDATE t SET pad = md5((random()*1000000)::text);
UPDATE t SET pad = md5((random()*1000000)::text);

By running these queries, the WAL size is now 1.4G, and it’s bigger than 1.1G! Repeat this update query three times and the WAL grows to 2.8GB:

master$ du -sh data/pg10/pg_wal
2.8G data/pg10/pg_wal

Certainly, the WAL could grow infinitely until whole disk space is consumed.
How do we find out the reason for this?

postgres=# SELECT redo_lsn, slot_name,restart_lsn,
round((redo_lsn-restart_lsn) / 1024 / 1024 / 1024, 2) AS GB_behind
FROM pg_control_checkpoint(), pg_replication_slots;
redo_lsn    | slot_name | restart_lsn | gb_behind
------------+-----------+-------------+-----------
1/2A400630  | slot1     |  0/92000000 | 2.38

We have one slot behind the master of 2.38GB.

Let’s repeat the update and check again. The gap has increased:

postgres=# postgres=# SELECT redo_lsn, slot_name,restart_lsn,
round((redo_lsn-restart_lsn) / 1024 / 1024 / 1024, 2) AS GB_behind
FROM pg_control_checkpoint(), pg_replication_slots;
redo_lsn    | slot_name | restart_lsn | gb_behind
------------+-----------+-------------+-----------
1/8D400238  |     slot1 | 0/92000000  | 3.93

Wait, though: we have already used slot1 for backup! Let’s start the slave:

master$ cd ../slave
slave$ ./pgc start pg10

Replication started without any additional change to recovery.conf:

slave$ cat data/pg10/recovery.conf
standby_mode = 'on'
primary_conninfo = 'user=replicator password=replicator passfile=''/home/pguser/.pgpass'' host=127.0.0.1 port=5432 sslmode=prefer sslcompression=1 krbsrvname=postgres target_session_attrs=any'
primary_slot_name = 'slot1'

pg_basebackup -R option instructs backup to write to the recovery.conf file with all required options, including primary_slot_name.

WAL size, all slots connected

The gap reduced several seconds after the slave started:

postgres=# SELECT redo_lsn, slot_name,restart_lsn,
round((redo_lsn-restart_lsn) / 1024 / 1024 / 1024, 2) AS GB_behind
FROM pg_control_checkpoint(), pg_replication_slots;
redo_lsn    | slot_name | restart_lsn | gb_behind
------------+-----------+-------------+-----------
 1/8D400238 |     slot1 |  0/9A000000 | 3.80

And a few minutes later:

postgres=# SELECT redo_lsn, slot_name,restart_lsn,
round((redo_lsn-restart_lsn) / 1024 / 1024 / 1024, 2) AS GB_behind
FROM pg_control_checkpoint(), pg_replication_slots;
redo_lsn    | slot_name | restart_lsn | gb_behind
------------+-----------+-------------+-----------
 1/9E5DECE0 |     slot1 |  1/9EB17250 | -0.01
postgres=# \!du -sh data/pg10/pg_wal
1.3G data/pg10/pg_wal

Slave server maintenance

Let’s simulate slave server maintenance with ./pgc stop pg10 executed on the slave. We’ll push some data onto the master again (execute the UPDATE query 4 times).

Now, “slot1” is again 2.36GB behind.

Removing unused slots

By now, you might realize that a problematic slot is not in use. In such cases, you can drop it to allow retention for segments:

master$ psql
postgres=# SELECT pg_drop_replication_slot('slot1');

Finally the disk space is released:

master$ du -sh data/pg10/pg_wal
1.1G data/pg10/pg_wal

Important system variables

• archive_mode is not required for streaming replication with slots.
• wal_level – is replica by default
• max_wal_senders – set to 10 by default, a minimum of three for one slave, plus two for each additional slave
• wal_keep_segments – 32 by default, not important because PostgreSQL will keep all segments required by slot
• archive_command – not important for streaming replication with slots
• listen_addresses – the only option that it’s necessary to change, to allow remote slaves to connect
• hot_standby – set to on by default, important to enable reads on slave
• max_replication_slots – 10 by default https://www.postgresql.org/docs/10/static/runtime-config-replication.html

Summary

• Physical replication setup is really easy with slots. By default in pg10, all settings are already prepared for replication setup.
• Be careful with orphaned slots. PostgreSQL will not remove WAL segments for inactive slots with initialized restart_lsn.
• Check pg_replication_slots restart_lsn value and compare it with current redo_lsn.
• Avoid long downtime for slave servers with slots configured.
• Please use meaningful names for slots, as that will simplify debug.

References

• https://www.openscg.com/bigsql/package-manager/
• https://www.openscg.com/bigsql/docs/pgcli/pgcli/
• https://www.postgresql.org/docs/current/static/wal-configuration.html
• https://paquier.xyz/postgresql-2/postgres-9-4-feature-highlight-replication-phydical-slots/
• https://www.postgresql.org/docs/current/static/protocol-replication.html
• https://www.postgresql.org/docs/9.4/static/functions-admin.html

This email thread discusses the tradeoffs of adding optimization improvements that affect only a small percentage of queries, i.e., micro-optimizations. The big sticking point is that we don't know if the time required to check for these optimizations is worth it. For short-running queries, it probably isn't, but for long-running queries, it probably is. The problem is that we don't know the final cost of the query until the end the optimization stage — this makes it impossible to decide if checks for micro-optimizations are worthwhile during optimization.

During the email discussion, optimizing X = X clauses was considered to be a win for all queries, so was applied. Optimization to convert OR queries to use UNION is still being considered. Figuring out a way to estimate the cost before optimization starts was recently discussed.

I recently had the honor of speaking at the last Keep Ruby Weird. A good part of the talk dealt with Postgres and since Citus is a Postgres company I figured sharing those parts on the blog would be a good idea. If you’d like to see it in talk form, or you’d also like to know how to watch movies rendered as emojis in your terminal, I encourge you to watch the talk.

So, obviously we want to watch Star Wars in psql. The first step though is getting all of the frames into a format that makes sense. http://asciimation.co.nz by Simon Jansen is an ASCII version of Star Wars, and is perfect for our needs. Simon hand drew all of the frames and it looks like this:

                                        ====
          Help me,                      o o~~
      Obi-Wan Kenobi!                  _\O /_
                           ___        / \ /  \
                          /() \      //| |  |\\
                        _|_____|_   // | |  |//
            ,@         | | === | | //  | |  //
            /=-        |_|  O  |_|('   |===(|
            ||          ||  O  ||      | || |
            ||          ||__*__||      (_)(_)
          --~~---      |~ \___/ ~|     |_||_|
          |     |      /=\ /=\ /=\     |_||_|
 _________|_____|______[_]_[_]_[_]____/__][__\______________________

We can get all of the data using curl as long as we get it using gzip, and then we can extract the frames out of the single, very long javacript line that starts with var film =. The format is a number representing how many times this particular frame should be repeated and is followed by fourteen lines separated by new line characters.

raw = File.readlines("site.html")
  .find {|line| line.start_with? "var film"}
  .encode('UTF-8','binary',invalid: :replace,undef: :replace,replace: '')
  .gsub(%q{\'},"'")[12..-16]
  .split('\n')
  .each_slice(14)

Once we have that we can store it in a postgres table

CREATE TABLE film (
  i     serial PRIMARY KEY,
  count int    NOT NULL,
  frame text   NOT NULL
);

require 'sequel'
db = Sequel.connect("postgres:///")

raw.each do |data|
  count, *frame = data
  db[:film] << {count: count.to_i, frame: frame.join("\n")}
end

To watch this, we’re going to want some way that will show the frame then sleep for the appropriate amount of time. Postgres functions are perfect for this. This one will actually sleep for the duration requried for the previous frame, then display the current frame.

CREATE OR REPLACE FUNCTION go(speed numeric, ctr bigint)
RETURNS text AS $$
  begin
    PERFORM pg_sleep(count*speed) FROM film WHERE i=ctr-1; -- 😴
    RETURN (SELECT frame FROM film WHERE i=ctr);           -- 👀
  end
$$ LANGUAGE plpgsql;

We’re almost have all the pieces together, but we need a way to keep track of the current frame. If you create a table with an auto-incrementing column, behind the scenes, Postgres creates a sequence to keep track of the current value. You can create unattached sequences however, and control where they start.

CREATE SEQUENCE ctr;
SELECT nextval('ctr'); -- 1
SELECT nextval('ctr'); -- 2
SELECT nextval('ctr'); -- 3

ALTER SEQUENCE ctr RESTART 200;
SELECT nextval('ctr'); -- 200
SELECT nextval('ctr'); -- 201

And with that, SELECT go(0.01, nextval('ctr')); can be run over and over again to watch Star Wars inside psql. But it’s awfully annoying to have to hit up and enter over and over again, ins’t it? Well let me introduce you to the best feature in all of Postgres, and possibally the history of computer science. \watch was added in 9.3. Much like it’s Unix counterpart, it reruns the previous query for you over and over and over again. So if we run the select go command from earlier and add \watch 0.01 to the end, psql will show us the movie as a movie! Run it for yourself, or check out the video earlier to see what it looks like.

The following commit has happened in Postgres 12, adding a feature which allows to control and potentially enforce the protocol SSL connections can use when connecting to the server:

commit: e73e67c719593c1c16139cc6c516d8379f22f182
author: Peter Eisentraut <peter_e@gmx.net>
date: Tue, 20 Nov 2018 21:49:01 +0100
Add settings to control SSL/TLS protocol version

For example:

ssl_min_protocol_version = 'TLSv1.1'
ssl_max_protocol_version = 'TLSv1.2'

Reviewed-by: Steve Singer <steve@ssinger.info>
Discussion: https://www.postgresql.org/message-id/flat/1822da87-b862-041a-9fc2-d0310c3da173@2ndquadrant.com

As mentioned in the commit message, this commit introduces two new GUC parameters:

• ssl_min_protocol_version, to control the minimal version used as communication protocol.
• ssl_max_protocol_version, to control the maximum version used as communication protocol.

Those can also take different values, which defer depending on what the version of OpenSSL PostgreSQL is compiled with is able to support or not, with values going from TLS 1.0 to 1.3: TLSv1, TLSv1.1, TLSv1.2, TLSv1.3. An empty string can also be used for the maximum, to mean that anything is supported, which gives more flexibility for upgrades. Note that within a given rank, the latest protocol will be the one used by default.

Personally, I find the possibility to enforce that quite useful, as up to Postgres 11 the backend has been taking automatically the newest protocol available with SSLv2 and SSLv3 disabled by being hardcoded in the code. However sometimes there are requirements which pop up, telling to make sure that at least a given TLS protocol needs to be enforced. Such things would not matter for most users but for some large organizations sometimes it makes sense to enforce some control. This is also useful for testing a protocol when doing development on a specific patch, which can happen when working on things like SSL-specific things for authentication. Another area where this can be useful is if a flaw is found in a specific protocol to make sure that connections are able to fallback to a safer default, so flexibility is nice to have from all those angles.

From an implementation point of view, this makes use of a set of specific OpenSSL APIs to control the minimum and maximum protocols:

• SSL_CTX_set_min_proto_version
• SSL_CTX_set_max_proto_version

These have been added in OpenSSL 1.1.0, still PostgreSQL provides a set of compatibility wrappers which make use of SSL_CTX_set_options for older versions of OpenSSL, so this is not actually a problem when compiling with other versions, especially since OpenSSL 1.0.2 is the current LTS (Long-Time-Supported) version of upstream at this point.

Perhaps one overlooked change in Postgres 10 was the removal of the recommendation of smaller shared_buffers on Windows. This email thread discussed its removal. So, if you have been minimizing the size of shared_buffers on Windows, you can stop now.