1. Introduction

This is part 3 of the blog “Understanding Security Features in PostgreSQL”, in which I will be discussing how to apply TLS in both PostgreSQL server and client using the principles we have learned in part 2 of the blog. In the end, I will also briefly talk about Transparent Data Encryption (TDE) and security vulnerability.

Here is the overview of the security topics that will be covered in all parts of the blog:

Part 1:

• PostgreSQL Server Listen Address
• Host-Based Authentication
• Authentication with LDAP Server
• Authentication with PAM
• Role-Based Access Control
• Assign Table and Column Level Privileges to Users
• Assign User Level Privileges as Roles
• Assign and Column Level Privileges via Roles
• Role Inheritance

Part 2:

• Security Concepts around TLS
• Symmetrical Encryption
• Asymmetrical Encryption (a.k.a Public Key Cryptography)
• Block Cipher Mode of Operation (a.k.a Stream Cipher)
• Key Exchange Algorithm
• TLS Certificate and Chain of Trust
• Data Integrity Check / Data Authentication
• TLS Cipher Suite and TLS handshake
• TLS versions

Part 3:

• Preparing TLS Certificates
• Enabling Transport Layer Security (TLS) to PostgreSQL Server
• Enabling Transport Layer Security (TLS) to PostgreSQL Client
• TLS Connect Examples
• Transparent Data Encryption (TDE)
• Security Vulnerability

2. Preparing TLS Certificates

Before we can utilize TLS to secure both the server and the client, we must prepare a set of TLS certificates to ensure mutual trust. Normally the CA (Certificate Authority) certificates can be purchased from a trusted organization and used it to create more CA-Signed certificates for services and applications. In this section, I will show you how to create your own CA Certificate and CA-Signed certificates using OpenSSL command line tool for both PostgreSQL server and client.

You may also have heard the term self-signed certificate. This type of certificate is not signed by a trusted CA and is normally considered insecured in many applications. We will not go over the self-signed certificate generation in this blog.

2.1 Generate a Private Key for CA Certificate

Remember in last blog we mention that each certificate contains organization information and public key, which is paired with a private key file. Let’s generate a private key file for our CA first.

$ openssl genrsa -des3 -out cacert.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
...............+++++
........................+++++
e is 65537 (0x010001)
Enter pass phrase for cacert.key:
Verifying - Enter pass phrase for cacert.key:

Your will be prompted with pass phrase, which is recommended to provide as it will prevent someone else from generating more root CA certificate from this key.

2.2 Generate CA Certificate Using the Private key

Now, let’s generate the CA Certificate with the private key

$ openssl req -x509 -new -nodes -key cacert.key -sha256 -days 3650 -out cacert.pem
Enter pass phrase for cacert.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:BC
Locality Name (eg, city) []:Vancouver
Organization Name (eg, company) [Internet Widgits Pty Ltd]:HighGo
Organizational Unit Name (eg, section) []:Software
Common Name (e.g. server FQDN or YOUR name) []:va.highgo.com
Email Address []:cary.huang@highgo.ca

Please note that OpenSSL will prompt you to enter several pieces of organizational information that identifies the CA certificate. You should enter these information suited to your organization. The most important field is Common Name, which is commonly checked against the hostname or domain name of the service. Depending on the seurity policy, some server will enforce the rule that common name must equal its host / domain name; some servers do not have this restriction.

2.3 Generate a private key for CA-Signed certificate

Like in the CA case, CA-signed certificate is also paired with a private key file

$ openssl genrsa -out server.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
...................+++++
............................................................................+++++
e is 65537 (0x010001)

2.4 Generate a Certificate Signing Request for CA-Signed certificate

Then we create a Certificate Signing Request (CSR), which contains a list of organizational information to be presented to the CA server for verification. The CA server then decide if the CSR should be granted a new certificate according to the security policy configured. Since we are using OpenSSL for certificate generation, the CA server here refers to OpenSSL itself, and the security policy configuration is located in openssl.cnf, which is commonly located in /usr/local/ssl/openssl.cnf. In an enterprise environment where Public Key Infrastructure (PKI) is deployed, the CA Server could refer to an actual service whose sole purpose is to verify incoming CSRs and renew or issue new certificates to requesting clients.

$ openssl req -new -key server.key -out server.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:BC
Locality Name (eg, city) []:Vancouver
Organization Name (eg, company) [Internet Widgits Pty Ltd]:HighGo
Organizational Unit Name (eg, section) []:Software
Common Name (e.g. server FQDN or YOUR name) []:va.highgo.ca
Email Address []:cary.huang@highgo.ca

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:HighGo Canada

2.5 Generate a CA-Signed certificate

Since we are generating CA-signed certificate with OpenSSL locally, we can configure how the certificate should be generated using openssl.cnf file. We will just be using the default policty set in openssl.cnf. Here’s a snapshot of the default settings:

[ usr_cert ]

# These extensions are added when 'ca' signs a request.

# This goes against PKIX guidelines but some CAs do it and some software
# requires this to avoid interpreting an end user certificate as a CA.

basicConstraints=CA:FALSE

# Here are some examples of the usage of nsCertType. If it is omitted
# the certificate can be used for anything *except* object signing.

# This is OK for an SSL server.
# nsCertType            = server

# For an object signing certificate this would be used.
# nsCertType = objsign

# For normal client use this is typical
# nsCertType = client, email

# and for everything including object signing:
# nsCertType = client, email, objsign

# This is typical in keyUsage for a client certificate.
# keyUsage = nonRepudiation, digitalSignature, keyEncipherment

Let’s generate the CA-signed certificate. Note that the command will take cacert.pem, cacert.key and server.csr as inputs, in which we have already generated from previous steps. server.pem will be the output.

$ openssl x509 -req -in server.csr -CA cacert.pem -CAkey cacert.key -CAcreateserial -out server.pem -days 3650 -sha256
Signature ok
subject=C = CA, ST = BC, L = Vancouver, O = HighGo, OU = Software, CN = va.highgo.ca, emailAddress = cary.huang@highgo.ca
Getting CA Private Key
Enter pass phrase for cacert.key:

We can repeat from step 2.3 to 2.5 to generate a new pair for the client application.

To conclude, we have the following files generated:

• cacert.pem – Root CA certificate that is at the top of the chain of trust. We use it to sign and create other certificates
• cacert.key – key for the Root CA Certificate – must keep it secured.
• server.pem – CA-signed certificate for server application
• server.key – key for the server certificate
• client.pem – CA-signed certificate for client application
• client.key – key for the client certificate

3. Enabling Transport Layer Security (TLS) to PostgreSQL Server

PostgreSQL has native support for TLS to secure connection between client and server. The TLS support has to be enabled during build time and requires OpenSSL libraries. Depending on the versions of OpenSSL that the client or server is built with, TLS versions and ciphersuites may differ as well. This does not mean that both client and server must be linked with the same version of OpenSSL. It is possible that a client with older OpenSSL can connect to a server with newer OpenSSL if the server is configured to accept it. The TLS handshake process is initiated when a client first connects to the server in which they will evaluate TLS version used and negotiate ciphersuite that both ends are able to support. In this case, the server may use less secured ciphersuite and TLS version to communiate with the client, which may not be ideal.

The TLS support for a PostgreSQL server can be enabled in postgresql.conf.

ssl = on
ssl_ca_file = '~/cert/cacert.pem'
ssl_cert_file = '~/cert/server.pem'
ssl_crl_file = ''
ssl_key_file = '~/cert/server.key'
ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers
ssl_prefer_server_ciphers = on
ssl_ecdh_curve = 'prime256v1'
ssl_min_protocol_version = 'TLSv1.2'
ssl_max_protocol_version = ''
ssl_dh_params_file = ''
ssl_passphrase_command = ''
ssl_passphrase_command_supports_reload = off

Let’s examine the configuration parameters.

ssl = on

This line turns on the TLS support. Please note that even if TLS is turned on, the server will still be able to accept connections that do not use TLS. Normally, the client is the entity that decides if TLS should be used or not. The server can also enforce the incoming connections to use TLS by modifying the pg_hba.conf file like this, where the connections from 172.16.30.0/24 must be TLS, otherwise the server will deny.

hostssl sales_team       all             172.16.30.0/24          trust

ssl_ca_file = '~/cert/cacert.pem'
ssl_cert_file = '~/cert/server.pem'
ssl_crl_file = ''
ssl_key_file = '~/cert/server.key'

These 4 lines tell PostgreSQL where to load the X509 certificate, the CA certificate, server private key and the certificate revokation list. These certificates must be pre-generated by OpenSSL command or purchased from a trusted organization. For TLS to work, ssl_ca_file, ssl_cert_file and ssl_key_file must be provided. We will use the certificates we have generated for server in the previous section.

The file pointed by ssl_ca_file will be used to determined if the certificate can be trusted by deriving the chain of trust.
The file pointed by ssl_cert_file will be sent to the connecting client during TLS handshake for authentication purposes.
The file pointed by ssl_key_file will be used for asymmetrical encryption during authentication

The file pointed by ssl_crl_file is optional and it contains a list of certificates that cannot be trusted (or revoked). Distributing revoked certificates using this file is not the most ideal but still being practice today. It may have performance impact if the list is very large and it introduces a problem of when the list should be renewed and how often. Online Certificate Status Protocol (OCSP. ref:https://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol) is a newer protocol designed for Public Key Infrastructure (PKI) for querying certificate revokation status that addresses some of the issues with revokation file. Feel free to give a read on OCSP in the link above.

ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers
ssl_prefer_server_ciphers = on
ssl_ecdh_curve = 'prime256v1'

During TLS handshake, both client and server will present to each other a list of desired ciphersuites ordered by preference. Handshake process will go through both lists and find a common ciphersuite supported by both sides or abord if there is nothing in common. The ssl_ciphers configuration is used to configure the size of the ciphersuite lists to be presented to the client during handshake.

ssl_ciphers is a string list consisting of one or more cipher strings separated by colons ( ref: https://www.openssl.org/docs/man1.1.1/man1/ciphers.html) and defaults to HIGH:MEDIUM:+3DES:!aNULL which translates to:

• allows high strength ciphersuites (HIGH)
• allows medium strength ciphersuites (MEDIUM)
• move any ciphersuite using 3DES algorithm to the end of the list (+3DES)
• remove any ciphersuite that does not have authentication algorithm (!aNULL)

For example, “HIGH:!ADH:!MD5:!RC4:!SRP:!PSK:!DSS:!ECDHE:!ECDSA:!EDH:!DH:!ECDH:!CAMELLIA256” will use high strength ciphersuites while removing any ciphersuites containing ADH, MD5, RC4…etc.

Before applying the cipher string to PostgreSQL, it is recommended to check the output cipher list after tuning the cipher string using Openssl client tool.

$ openssl ciphers -v 'HIGH:!ADH:!MD5:!RC4:!SRP:!PSK:!DSS:!ECDHE:!ECDSA:!EDH:!DH:!ECDH:!CAMELLIA256'
TLS_AES_256_GCM_SHA384  TLSv1.3 Kx=any      Au=any  Enc=AESGCM(256) Mac=AEAD
TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 Kx=any      Au=any  Enc=CHACHA20/POLY1305(256) Mac=AEAD
TLS_AES_128_GCM_SHA256  TLSv1.3 Kx=any      Au=any  Enc=AESGCM(128) Mac=AEAD
AES256-GCM-SHA384       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(256) Mac=AEAD
AES256-CCM8             TLSv1.2 Kx=RSA      Au=RSA  Enc=AESCCM8(256) Mac=AEAD
AES256-CCM              TLSv1.2 Kx=RSA      Au=RSA  Enc=AESCCM(256) Mac=AEAD
ARIA256-GCM-SHA384      TLSv1.2 Kx=RSA      Au=RSA  Enc=ARIAGCM(256) Mac=AEAD
AES128-GCM-SHA256       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(128) Mac=AEAD
AES128-CCM8             TLSv1.2 Kx=RSA      Au=RSA  Enc=AESCCM8(128) Mac=AEAD
AES128-CCM              TLSv1.2 Kx=RSA      Au=RSA  Enc=AESCCM(128) Mac=AEAD
ARIA128-GCM-SHA256      TLSv1.2 Kx=RSA      Au=RSA  Enc=ARIAGCM(128) Mac=AEAD
AES256-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA256
AES128-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA256
CAMELLIA128-SHA256      TLSv1.2 Kx=RSA      Au=RSA  Enc=Camellia(128) Mac=SHA256
AES256-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA1
AES128-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA1
CAMELLIA128-SHA         SSLv3 Kx=RSA      Au=RSA  Enc=Camellia(128) Mac=SHA1

ssl_prefer_server_ciphers specifies whether to use the server’s SSL cipher preferences, rather than the client’s. It should always be on for more control in terms of ciphersuite selection.

ssl_ecdh_curve specifies the name of the curve to use in ECDH key exchange algorithms and is useful only if the ciphersuite uses ECDHE key exchange algorithm. The most common curves are : prime256v1, secp384r1 and secp521r1 and normally leaving it default should suffice.

ssl_min_protocol_version = 'TLSv1.2'
ssl_max_protocol_version = ''

“`
These 2 lines configure the minimum and maximun TLS versions to accept. By default the server will only serve the TLS client using TLSv1.2 and above. TLSv1.2 is a very secured TLS version and it is widely used in the world. Normally, we only change the minimum TLS version with assumption that all future versions will be more secured and for this reason, we normally don’t put restriction on the max version.

TLSv1.3 is recently introduced that has new ciphersuite support and has more improvement in the handshake process. To enforce TLSv1.3 to be used, set the ssl_min_protocol_version to ‘TLSv1.3’ will suffice.

#ssl_dh_params_file = ''
#ssl_passphrase_command = ''
#ssl_passphrase_command_supports_reload = off

ssl_dh_params_file points to a file that contains custom diffie-hellman key exchange algorithm parameter. This is an optional parameter and is only useful if the ciphersuite uses DHE key exchange algorithm. If left empty, compiled-in defaults will be used. Custom DH parameters can be generated using command openssl dhparam -out dhparams.pem 2048 and will normally reduce the attack exposure as attacker will have hardtime cracking the key exchange process using custom parameter instead of the well-known default.

ssl_passphrase_command is the command to obtain the password for the private key file specified by ssl_key_file. There is an option to add a password to a private key file during its generation and if password is used, ssl_passphrase_command must be set with the system command that will retrieve such password. Otherwise, TLS handshake will abord as PostgreSQL will not be able to access private key without password.

ssl_passphrase_command_supports_reload configures if the ssl_passphrase_command should be re-run at every reload (ie. SIGHUP). It is default to off, so the ssl_passphrase_command will not be run at every reload.

4. Enabling Transport Layer Security (TLS) to PostgreSQL Client

Now that we have a PostgreSQL server with TLS setup, we can use psql client to connect to the server also using TLS. Depending on the client connect parameters given, we can utilize TLS in different security levels. I will show the most common usages here:

# Case 1: connect to server in TLS mode
$ psql -U user -h localhost -d "sslmode=require dbname=postgres"

# Case 2: connect to server in TLS mode if server supports it
$ psql -U user -h localhost -d "sslmode=prefer dbname=postgres"

# Case 3: connect to server in TLS mode and verify server CA against client CA
$ psql -U user -h localhost -d "sslmode=verify-ca dbname=postgres sslrootcert=~/cert/cacert.pem"

# Case 4: connect to server in TLS mode and present client certificate. Verify all certificate details and trust chain. Check certificate revokation list does not contain server cert.
$ psql -U user -h localhost -d "sslmode=verify-full dbname=postgres sslrootcert=~/cert/cacert.pem sslcert=~/cert/client.pem sslkey=~/cert/client.key"

The usage in Case 4 is the most secured becuase both server and client will verify each other’s certificate and decide if both can be mutually trusted. The common name field in the certificate is checked against the server hostname; certificate validity period is checked, organization details are checked; certificate trust chain is checked; revokation list is checked.

Please note that PostgreSQL server with TLS enabled by default does not force the client to present a TLS certificate for verification. If client presents one like in Case 4 above, the server will verify and deny connection is certificate is bad. If client does not provide a certificate like in Case 1 ~ 3, the server will skip the client certificate verification as there is nothing to verify, which is less secure.

To enforce the connecting client to present a TLS certificate for verification, we will need to add a special clientcert=1 argument in existing authentication rules defined in pg_hba.conf.

# TYPE  DATABASE         USER                 ADDRESS                 METHOD
hostssl production_team     production_user      0.0.0.0/0            pam clientcert=1

The example above will enforce connecting client to present TLS certificate to access production_team database as production_user. If a TLS certificate is not provided by client, the connection will abort.

5. TLS Connect Examples

$ psql -U user -h localhost -d "sslmode=require dbname=postgres"
psql (13devel)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

postgres=# 

Please note that psql prints the TLS version used (TLSv1.2) and the cipher suite negotiated during handshake (ECDHE-RSA-AES256-GCM-SHA384). Below is the wireshark capture of the above TLS connection:

Another Example:

$ psql -U cary -h localhost -d "sslmode=verify-full dbname=postgres sslrootcert=~/cert/cacert.pem sslcert=~/cert/client.pem sslkey=~/cert/client.key"

psql: error: could not connect to server: server certificate for "va.highgo.ca" does not match host name "localhost"

Here, we have an error when we set sslmode to verify-full, where both server and client will verify each other with the most strict criteria. This error happens because the Common Name field in the certificate does not match the host name. Did I mention that Common Name is the most important field of a certificate? To resolve this error, we can either re-generate certificate with matching Common name, or change the host name.

I simply add an entry to /etc/hosts to resolve the error

127.0.0.1       localhost
127.0.0.1       va.highgo.ca

and the error will disappear when both Common Name and Hostname match

$ psql -U cary -h va.highgo.ca -d "sslmode=verify-full dbname=postgres sslrootcert=~/cert/cacert.pem sslcert=~/cert/client.pem sslkey=~/cert/client.key"

psql (13devel)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

Please note that this command also forces the client to submit a certificate to server as well as seen from the wireshark capture. We can tell by looking at the length field of the packet capture. There are 2 exchanges having lengths = 2675 and 2446. Those are the actual certificate contents being transmitted. Previous capture only has 1 exchanges having packet length = 2675; it means only server is providing certificate to client for verification.

6. Transparent Data Encryption (TDE)

Transparent Data Encryption refers to the process of protecting data at rest by encrypting database files on the hard disk level and decrypting them while reading from hard disk. This is to prevent physical storage media theft. This is called transparent because the encryption and decryption happen between PostgreSQL server and the physical hard disk and it is not visible to the client applications. TDE uses symmetrical encryption for securing blocks of database files such as shared buffer and WAL files, and it is designed to accompany with a internal Key Management System (KMS) to manage the lifecycle of the encryption keys.

TDE and KMS are still under development by the PostgreSQL community. The KMS feature is expected to be released in PG13 while the TDE feature to be in PG14. With its completion, it will add another layer of security feature on top of already security-rich PostgreSQL database.

7. Security Vulnerability

Security Vulnerability is a weakness which can be exploited by an attacker to perform unauthorized actions, sabotage a service, or inject malicious software or virus. These weaknesses are generally implementation mistakes, undiscovered bugs or a legacy problem that require an update to the server to resolve.

PostgreSQL also has a list of known security vulnerability that has been discovered and fixed by the community. The list can be found here: https://www.postgresql.org/support/security/. These vulnerability ranges from different severity levels, from simple memory leak to crash the server.

This is why doing regular PostgreSQL server upgrade is important because each minor release fixes some of the discovered security vulnerabilities and therefore reducing the attack surface on your server.

8. Summary

In part 3 of the blog, we have learned and understood what each TLS related configuration means in postgresql.conf and how to initiate TLS connection with psql client. We learned that keeping PostgreSQL server up-to-date can reduce the attack surface on some of the discovered vulnerabilities. We can ensure a fairly secured database network environment with TLS having adequate understanding of its fundamentals and practices. With the TDE feauture coming in near future, we can further secure the database envrionment in the disk level and prevent possible data loss due to disk theft.

Cary Huang

Cary is a Senior Software Developer in HighGo Software Canada with 8 years of industrial experience developing innovative software solutions in C/C++ in the field of smart grid & metering prior to joining HighGo. He holds a bachelor degree in Electrical Engineering from University of British Columnbia (UBC) in Vancouver in 2012 and has extensive hands-on experience in technologies such as: Advanced Networking, Network & Data security, Smart Metering Innovations, deployment management with Docker, Software Engineering Lifecycle, scalability, authentication, cryptography, PostgreSQL & non-relational database, web services, firewalls, embedded systems, RTOS, ARM, PKI, Cisco equipment, functional and Architecture Design.

Substituting a single string by another within a larger string is straightforward in SQL, with the replace function:

select replace('the string is bar', 'bar', 'foo');
     replace
-------------------
 the string is foo

But there isn’t a PostgreSQL core function to substitute multiple strings each by its own independent replacement, like for instance the strtr function in PHP, or the substitutions operators with regular expressions in Python or Perl (which we’re going to use in this post).

In PostgreSQL, regexp_replace does support alternations to search for multiple strings in a single pass, but it can’t do multiple replacements, that is regexp_replace(string, 'foo|bar', 'baz', 'g') would replace both foo and bar by baz, but it cannot be instructed to replace foo by a string and bar by another.

In some cases, nested or successive replace() calls might do the job, and it seems the popular answer when searching for a solution on the web, but in the general case, they produce flawed results if any inner replacement happens to create a match for other outer replace calls.

As an example, consider the necessary conversion of characters into HTML entities when creating an HTML fragment from a raw text. These five substitutions have to be applied:

> → >
 < → &lt;
 & → &amp;
 " → &apos;
 ' → &quot;

Say we apply the substitutions in that order, with a query like this:

select replace(replace(replace(replace(replace(
  rawtext,
  '>', '>'),
  '<', '&lt;'),
  '&', '&amp;'),
  '"', '&apos;'),
  '''', '&quot;');

If rawtext is <strong> AT&T </strong>, then the result obtained is:

<strong> AT&T </strong>

and this is clearly wrong, as what we want is:

<strong> AT&T </strong>

The problem with the query above is that once > has been replaced by > (or < by <), the next replacement stage has no way to distinguish the ampersand in AT&T (that must be replaced) from the ampersand in > that comes from a previous replacement and should be left unmodified.

In that particular case, an effective workaround is to rearrange the order of the replacements so that & is substituted by & first.

But what if we have circular references between substitutions, like when two strings need to be switched? In that case, no order will succeed to produce the desired result:

-- replace foo with bar and bar with foo.
-- wrong result, version 1
select replace(replace('foo and bar', 'foo', 'bar'), 'bar', 'foo');
   replace   
-------------
 foo and foo

-- wrong result, version 2
 select replace(replace('foo and bar', 'bar', 'foo'), 'foo', 'bar');
   replace   
-------------
 bar and bar

What is needed instead is an algorithm that scans and replaces in the string in a single pass. It also needs a rule for when several matches occur simultaneously at the same point in the string (typically the longest match wins, but another rule could be chosen).

The Perl substitution operator (s/pattern/replacement/flags) provides this functionality, since pattern can be an alternation and replacement can be a hash with key and values holding the substitutions. For instance:

my%subs=(foo=>bar,bar=>foo);my$string="foo et bar";$string=~s/(foo|bar)/$subs{$1}/g;print$string;

As a bonus, since Perl 5.10 the implementation of this construct has been specifically optimized:

Trie optimisation of literal string alternations

Alternations, where possible, are optimised into more efficient matching structures. String literal alternations are merged into a trie and are matched simultaneously. This means that instead of O(N) time for matching N alternations at a given point, the new code performs in O(1) time. A new special variable, ${^RE_TRIE_MAXBUF}, has been added to fine-tune this optimization. (Yves Orton)

PostgreSQL allows to write functions in Perl With the plperl extension. Using this, here’s a simple ready-to-use implementation of the multiple replacement:

CREATEFUNCTIONmulti_replace(stringtext,origtext[],repltext[])RETURNStextAS$BODY$my($string,$orig,$repl)=@_;my%subs;if(@$orig!=@$repl){elog(ERROR,"array sizes mismatch");}if(ref@$orig[0]eq'ARRAY'||ref@$repl[0]eq'ARRAY'){elog(ERROR,"array dimensions mismatch");}@subs{@$orig}=@$repl;my$re=join"|",mapquotemeta,sort{(length($b)<=>length($a))}keys%subs;$re=qr/($re)/;$string=~s/$re/$subs{$1}/g;return$string;$BODY$languageplperlstrictimmutable;

This Perl function appears to be very fast, even with many strings to replace (tested up to 1000, knowing that they get combined into a single regular expression) and many actual replacements occurring.

In a plpgsql implementation, searching with the alternation appears to be pretty fast, but iterating on the replacements is very costly. The code is a bit longer, I won’t put it in this post but it’s on the PostgreSQL wiki, see multi_replace in plpgsql. Aside from taking the input replacements through a jsonb parameter rather than arrays, which incidentally allows to have the two implementations to co-exist in the database with the same function name, the results should always be identical.

For instance, this invocation switches foo and bar while also leaving foobar unchanged by using the trick of replacing it by itself:

selectmulti_replace('foo and bar are not foobar','{foo,bar,foobar}','{bar,foo,foobar}');

       multi_replace        
----------------------------
 bar and foo are not foobar

Be it in plperl or plpgsql, this function avoids the problems with nesting replace(), including allowing dynamic lists of substitutions. But if you need that functionality with large numbers of substitutions, it’s a case where Perl is much more efficient than the equivalent with SQL built-in functions.

What is PostgresqlCO.NF?

PostgresqlCO.NF (CONF for short) is your postgresql.conf documentation and ultimate recommendations’ source. Our mission is to help you tune and optimize all of your PostgreSQL configuration. With around 290 configuration parameters in postgresql.conf (and counting), it is definitely a difficult task! We aim to make PostgreSQL configuration accessible for HUMANS.

CONF is, according to our users, “literally the best Postgres configuration reference ever created.”

Have you tried https://t.co/GuARgAZX1v by @ongresinc?

I use it daily, literally. It's the best Postgres configuration reference ever created.

Recently redesigned. I'm not a big fan of the new color schema, but I do like tiny improvements, details and information structure👍 pic.twitter.com/M2ncahQ4Dh

— Nikolay Samokhvalov (@postgresmen) January 13, 2020

Why you should be using CONF

There are almost 300 parameters and with each new Postgres version more are added. Parameters can be integers, real, string, enum or bool. Numeric values may have units (or unit-less) and some units can be confusing (like “blocks of 8kB”). It can be challenging to try to understand them all.

You can learn many things by reading the documentation reference, but CONF aggregates the official documentation, plus some recommendations and relevant links to StackOverflow and the -hackers mailing list in a direct and easy to read format that you can bookmark or share a link of an specific parameter to someone, and if you already are an expert you can comment on the site and add your thoughts on how to tune any given parameter.

Some parameters require restart. Having a quick and clear reference of all the parameters context can help you to plan ahead whether you need or not a maintenance window for your deployment. Consider archive_mode parameter as an example: it’s required for WAL archiving but it’s off by default and requires restart; the archive_command on the other hand does not require restart. Actually, one good recommendation is to set archive_mode to on and archive_command to /bin/true until you find the real archive_command you will use in production, but never have to restart for this reason. This is just the tip of the iceberg on what you can learn by just reading the docs.

However CONF it’s not just a documentation reference. Right now we are working hard on a fully featured application service where you can have a graphical configuration interface (or UI) with Drag & Drop of your postgresql.conf files with automatic validation, and a REST API where you can store and share your custom postgresql.conf configuration files. You will also be able to download your configurations in several formats, like the native postgresql.conf, YAML or JSON.

Subscribe to our extremely low traffic mailing list on https://postgresqlco.nf to become a beta tester of the configurator when the software will be ready.

But wait, there’s more. There is a little hidden feature that we want to make public today. It is a small JavaScript snippet that you can add to your web/blog that converts all the parameters found in the html and shows a nice tooltip with info about that parameter. Like magic!

Adding it to your website will take you 1 minute. Check the documentation page and grab it while it’s hot. We are improving it with more options in later versions, but we can’t wait to hear your comments and feedback (and of course bugs reports).

The redesign

On December 31, 2019, we deployed (yes, we deployed on 12/31, it wasn’t Friday but it’s pretty much the same thing!) a new version of the postgresqlCO.NF website, saying goodbye to both the old year and design. We had been working on the redesign of our Postgres Documentation page for a while.

First of all, we decided to give CONF a more modern look, changing the UI color scheme and unifying all fonts and sizes. Now, all pages have the same look and feel: clean, professional and most importantly, intuitive. We believe it is important that these services are not only useful, but also pleasant and easy to use. We kept the three-column structure, but we reorganised the content. This redesign also fixes many small bugs that existed in the previous version.

Last but not least, the whole infrastructure that runs CONF has changed. We migrated from a web application powered by Java that dynamically renders the pages served from Google Cloud, to a statically generated web site served from Amazon S3 using CloudFront. This drastically improves the performance of the whole site, giving a better user experience.

Feel free to give us your feedback about this new version of CONF and stay tuned to our updates, as we will be adding some new and highly requested features (like a dark mode 🕶) really soon!

Do you want to know where it all started?

pg_timetable is a brand-new job scheduler for PostgreSQL implemented completely from scratch (in Go). Traditional job scheduling is quite limited and does not cover all we needed in real life. Therefore, we thought about job execution in more detail and came up with a new concept that has not been seen in the PostgreSQL world before. pg_timetable does not simply execute some simple SQL statements periodically. It has been designed to execute entire sequences of operations at a time while offering some other important features we will discuss in this post.

Installing and running pg_timetable

You can run pg_timetable in 2 ways:

• Container
• Standalone application

Both methods are described on our Github page so it should be easy for most people to run pg_timetable and to give it a try.

For the sake of simplicity we assume that you have installed our new PostgreSQL job scheduler and everything is up and running. So let’s take a look and see how pg_timetable can be configured.

Base tasks and chains

The core idea is to allow users to execute more complicated sequences of operations. Now what does that mean? Suppose you want to …

• Start a transaction
• Write a log entry to the database
• Download a file
• Import the file
• Run aggregations
• Commit the transaction

A sequence like that is quite common. Let us take a look at those steps in pg_timetable terminology: Each step on the way is a “base task”. There are various types of base tasks:

• Built-in tasks (such as “send email”, start transaction, etc.)
• SQL tasks (some database instruction)
• Shell tasks (some external executables)

Those tasks can be combined and turned into chains. A base task can take parameters which are added at runtime. What does that mean? Suppose you want to send an email once a day. Who is going to receive this email? This kind of information can be passed to chains / base tasks scheduled for execution.

Configuring pg_timetable: An example

Let us take a look at an easy pg_timetable configuration example. The goal is just to make the database sleep periodically. We created a simple config script:

WITH noop(id) AS (
     SELECT task_id FROM timetable.base_task WHERE name = 'Sleep'
),
chain_insert(chain_id) AS (
     INSERT INTO timetable.task_chain
            (chain_id, parent_id, task_id, run_uid, 
             database_connection, ignore_error)
     VALUES
            (DEFAULT, NULL, (SELECT id FROM noop), NULL, NULL, TRUE)
     RETURNING chain_id
),
chain_config(id) AS (
     INSERT INTO timetable.chain_execution_config VALUES
     (
          DEFAULT, -- chain_execution_config,
          (SELECT chain_id FROM chain_insert),    -- chain_id,
          'sleep every minute',                   -- chain_name,
          NULL,   -- run_at_minute,
          NULL,   -- run_at_hour,
          NULL,   -- run_at_day,
          NULL,   -- run_at_month,
          NULL,   -- run_at_day_of_week,
          1,      -- max_instances,
          TRUE,   -- live,
          FALSE,  -- self_destruct,
          FALSE   -- exclusive_execution,
     )
     RETURNING chain_execution_config
)
INSERT INTO timetable.chain_execution_parameters
     (chain_execution_config, chain_id, order_id, value)
VALUES (
     (SELECT id FROM chain_config),
     (SELECT chain_id FROM chain_insert),
     1,
     '5' :: jsonb)
;

The WITH statement just selects the ID of a base task called “Sleep”. Each base task has a name and an ID. We will need the ID to schedule those events in the form of a simple chain. In my example, the chain has just one element, so there is no need for parent elements. Also: We simply want to sleep. There is no need for a database connect string.
To do the actual scheduling, we need to insert into timetable.chain_execution_config. The configuration is pretty similar to cron. However, there are a couple of interesting things here:

• max_instances tells us that this job must not run concurrently. One sleep is allowed at a time
• self_destruct = false means that the chain will survive its own execution. If you set this to true, a chain is only executed once (in case it succeeds).
• exclusive_execution means that other chains doing something might basically happen in parallel.

Finally, parameters are added to the chain.

Using pg_timetable concurrency protection

Concurrency protection is an important aspect. Suppose you want to backup a database daily. What is going to happen if the backup takes 60 hours to complete? At some point, your server is going to die because jobs pile up. Or maybe backup files will start to overwrite each other. In short: If you want to make sure that only one backup at a time is allowed, pg_timetable concurrency protection is what you want. The max_instances = 1 setting is the easiest way to do that. There is no more need for pid files, no need for anything else. pg_timetable does it all for you.

Utilizing self-destructive chains

More often than not, it happens that a job should only be done once. The idea is simple: Suppose somebody gets a 7-day free trial. After 7 days, you want to shut down the person’s account. Obviously, shutting down somebody’s account is not done daily – it happens once, and if the process was successful, that was basically it.
Self-destructive chains are especially useful for asynchronous execution. What do I mean by that? Suppose you have a website and you press a button to launch a job that runs for 2 hours. Clearly, nobody is going to sit in front of the browser window and wait. What you do instead is to tell pg_timetable to execute stuff as soon as possible and kill the job as soon as it is ended successfully. Asynchronous execution is the prime use case.

More simplistic jobs

Now I want to show you a more complex example. Let’s take a look at the following file: https://github.com/cybertec-postgresql/pg_timetable/blob/master/samples/ManyTasks.sql

In this case, 500 base tasks are created on the fly and assembled in a chain. The logic is the same as we had before. First, base tasks are selected (or created). Then, they are put into a chain and finally, execution is scheduled before we can at last add parameters to the job. The concept is always the same.

Making use of cron-style configuration

pg_timetable is really powerful, but sometimes you don’t need all those features. Sometimes all you want is to simply run some SQL and nothing more. To make configuration easier, we have introduced a cron-style configuration interface. The following example creates a dummy table and adds a row once in a while:

CREATE TABLE timetable.dummy_log (
    log_ID       BIGSERIAL,
    event_name   TEXT,
    timestmp     TIMESTAMPTZ        DEFAULT TRANSACTION_TIMESTAMP(),
    PRIMARY KEY (log_ID)
);

-- Parameters detail for timetable.job_add()
-- task_name: The name of the Task
-- task_function: The function which will be executed.
-- client_name: The name of worker under which this task will execute
-- task_type: Type of the function SQL,SHELL and BUILTIN
-- by_cron: Time Schedule in Cron Syntax
-- by_minute: This specifies the minutes on which the job is to run
-- by_hour: This specifies the hours on which the job is to run
-- by_day: This specifies the days on which the job is to run.
-- by_month: This specifies the month on which the job is to run
-- by_day_of_week: This specifies the day of week (0,7 is sunday) on which the job is to run
-- max_instances: The amount of instances that this chain may have running at the same time.
-- live: Control if the chain may be executed once it reaches its schedule.
-- self_destruct: Self destruct the chain.

----CRON-Style
-- * * * * * command to execute
-- ┬ ┬ ┬ ┬ ┬
-- │ │ │ │ │
-- │ │ │ │ └──── day of the week (0 - 7) (Sunday to Saturday)(0 and 7 is Sunday);
-- │ │ │ └────── month (1 - 12)
-- │ │ └──────── day of the month (1 - 31)
-- │ └────────── hour (0 - 23)
-- └──────────── minute (0 - 59)

SELECT timetable.job_add (
          task_name => 'cron_Job run after 40th minutes after 2 hour on 27th of every month ',
          task_function => $$INSERT INTO timetable.dummy_log (event_name) VALUES ('Cron test')$$,
          client_name => NULL, -- any worker may execute
          task_type => 'SQL',
          by_cron => '40 */2 27 * *',
          by_minute => NULL,
          by_hour => NULL,
          by_day => NULL,
          by_month => NULL,
          by_day_of_week => NULL,
          max_instances => 1,
          live => TRUE,
          self_destruct => FALSE
);

As you can see, all you have to do is to call the “timetable.job_add” function and pass a couple of parameters to the function. pg_timetable will automatically translate things into the internal configuration format for you.

Finally …

If you like pg_timetable, we are looking forward to receive you feedback, or maybe even feature requests, bug reports and contributions. We also want to point out that commercial support for pg_timetable is available internationally.

If you want to read something else right now, we can recommend our post about pg_permissions which is available here.

The post pg_timetable: Advanced PostgreSQL scheduling appeared first on Cybertec.

Prepared transactions are a key feature of PostgreSQL. Understanding what this feature offer and handling any potential pitfalls is critical to maintaining a system that is reliable. So let’s take a dive into what prepared transactions are all about. 

A Little Bit About Transactions

In a database system, transactions are a way of processing all or zero statements in a block that generally contains more than one statement. The results of the statements in the block are not visible to other transactions until the entire block is committed. If the transaction fails, or is rolled back, it has no affect on the database at all.

Transactions are attached to a session. However, there are needs when you want to perform a transaction that is session independent (and has other benefits as well). That’s where “prepared transactions” come in. 

Prepared Transactions

A prepared transaction is a session independent, crash resistant, state maintained transaction. The state of the transaction is stored on disk which allows the database server to reinstate the transaction even after restarting from a crash. A prepared transaction is maintained until a rollback or a commit action is performed on it.

The PostgreSQL documentation states that a prepared transaction may be created by issuing a PREPARE TRANSACTION ‘transaction_id’ command within an existing transaction block. It further states that this process prepares a transaction for a two-phase commit.

Furthermore, it is recommended that prepared transactions are not be used by applications or interactive sessions. Ideally, an external transactional manager should perform atomic global transactions across homogeneous or heterogeneous database resources. 

In PostgreSQL, by default, max_prepared_transactions is set to 0 in postgresql.conf file; i.e. it is disabled. If you want to use prepared transactions, it is recommended to set this value to max_connections. On the standby server in a synchronous streaming replication setup however, you’d want to be more generous and set the value a little higher than “max_connections” as otherwise you risk the standby not accepting any queries.

At any given time, you can view any active prepared transactions by simply querying the view pg_prepared_xacts. The view shows five columns:

1. Transaction; xid (transaction ID)
2. GID; text (a user defined name for the prepared transaction)
3. Prepared date; (timestamp with timezone when the prepared transaction was created)
4. Owner; (user who created the prepared transaction)
5. Database; (name of the database)

This view gives sufficient information about who owns the transaction and the database as well as when it was created. This is rather useful as we’ll see later on this blog.

Creating a Prepared Transaction

So now that we know what a prepared transaction is, let’s see how we can create one. Creating a prepared transaction is a four step process:

1. BEGIN (or START TRANSACTION)
2. Perform required operations
3. PREPARE TRANSACTION
4. COMMIT or ROLLBACK PREPARED

PREPARE TRANSACTION and COMMIT PREPARED or ROLLBACK PREPARED is followed by a GID that uniquely identifies the prepared transactions.

The following code block shows SQL statements are their outputs along with a view of pg_prepared_xacts view as we the transaction is prepared and then committed.

postgres=# BEGIN;
BEGIN
postgres=# CREATE TABLE foo(firstCol INT);
CREATE TABLE
postgres=# INSERT INTO foo VALUES(27);
INSERT 0 1
postgres=# PREPARE TRANSACTION 'foo_insert';
PREPARE TRANSACTION

postgres=# SELECT * FROM pg_prepared_xacts;
 transaction |    gid     |           prepared            | owner  | database 
-------------+------------+-------------------------------+--------+----------
       48219 | foo_insert | 2020-01-27 11:36:18.522511+00 | highgo | postgres
(1 row)

postgres=# COMMIT PREPARED 'foo_insert';
COMMIT PREPARED

postgres=# SELECT * FROM pg_prepared_xacts;
 transaction | gid | prepared | owner | database 
-------------+-----+----------+-------+----------
(0 rows)

When a server is stopped (or it crashes) with one or more active prepared transaction(s), it creates one file each for every active prepared transaction under pg_twophase folder within the data directory. Earlier in the blog, we created the following prepared transaction.

postgres=# SELECT * FROM pg_prepared_xacts;
transaction |    gid     |           prepared            | owner  | database 
-------------+------------+-------------------------------+--------+----------
       48219 | foo_insert | 2020-01-27 12:03:59.271897+00 | highgo | postgres
(1 row)

So I stopped the server without committing the transaction. The server created a file with the filename 0000BC5B which corresponds to the transaction ID of the prepared transaction.

[highgo@e7dc5351c7c7 bin]$ ls -l /tmp/data/pg_twophase/
total 4
-rw------- 1 highgo users 220 Jan 27 12:11 0000BC5B

0000BC5B equates to a value of 48219. When the server was restarted, it emitted the following messages in the startup log:

2020-01-27 12:16:56.025 UTC [370] LOG:  redo starts at 0/40000028
2020-01-27 12:16:56.026 UTC [370] LOG:  consistent recovery state reached at 0/40000100
2020-01-27 12:16:56.026 UTC [370] LOG:  redo done at 0/40000100
2020-01-27 12:16:56.065 UTC [370] LOG:  recovering prepared transaction 48219 from shared memory

If you don’t wish to recover a prepared transaction, you may simply delete the corresponding file under the pg_twophase folder.

That’s rather simple, no? So why shouldn’t we be using this more regularly? After all, it gives a higher probability of a commit operation being successful. If only things were this simple!

What Can Go Wrong with a Prepared Transaction?

A prepared transaction can be left unfinished (neither committed nor rollbacked) if the client has disappeared. It can happen for various reasons including a client crash, or a server crash leading to client’s connection getting terminated and never reconnects. You are really relying on a transaction manager to ensure that there are no orphaned prepared transactions.

Besides the crashes, there is another way a prepared transaction can be left unfinished. If a backup is restored that carried the preparation steps, but not the steps closing the transaction, you may still end up with an orphaned prepared transaction.

Or perhaps a DBA created a prepared transaction and forgot about closing it.

So what’s the big deal if a prepared transaction is left unfinished.

The Real Issue

The real issue is that an orphaned prepared transaction continues to hold key system resources which may include locks, or keeping alive a transaction ID that may hold back vacuum from cleaning up dead tuples that are no longer visible to any other transaction except for this orphaned prepared transaction.

Consider the transaction we created earlier in this blog. When the transaction is prepared, and prior to committing that transaction, if another transaction attempts to alter the table, it fails to acquire the required locks and hangs, until the prepared transaction is resolved, either committed or rolled back. Otherwise, the alter command hangs indefinitely, eventually, I had to issue a CTRL + C to stop the command.

postgres=# ALTER TABLE foo ADD COLUMN b INT;
^C
2020-01-27 12:52:15.809 UTC [395] ERROR:  canceling statement due to user request
2020-01-27 12:52:15.809 UTC [395] STATEMENT:  ALTER TABLE foo ADD COLUMN b INT;
Cancel request sent
ERROR:  canceling statement due to user request


postgres=# SELECT c.oid, c.relname, l.locktype, l.relation, l.mode 
FROM pg_class c 
INNER JOIN pg_locks l ON c.oid = l.relation
WHERE c.relname = 'foo';
  oid   | relname | locktype | relation |       mode       
--------+---------+----------+----------+------------------
 111197 | foo     | relation |   111197 | RowExclusiveLock
(1 row)

The challenge with vacuum can be a little more serious as in extreme cases, this may lead to database shutdown as the transaction will prevent transaction ID wrap around.

Discovering and Notifying

Although the expectation in general is that a prepared transaction is completed within a few seconds, however, that is not always the case. A prepared transaction may persist for a few minutes, hours, or even days. 

Maintaining metadata for these transactions may be a challenge on its own. However, I would suggest setting up a nomenclature that defines the max age a prepared transaction can have. For example, consider the following prepared transaction:

postgres=# BEGIN;
BEGIN
postgres=# INSERT INTO foo VALUES(85);
INSERT 0 1
postgres=# PREPARE TRANSACTION 'foo_insert 1m';
PREPARE TRANSACTION

Or consider the following transaction:

postgres=# BEGIN;
BEGIN
postgres=# INSERT INTO foo VALUES(2385);
INSERT 0 1
postgres=# PREPARE TRANSACTION 'foo_insert 1d';
PREPARE TRANSACTION

In these transaction names, the last part defines the age of a transaction. Any transactions that are past their age can easily be identified through the following simple SQL query:

postgres=# SELECT gid, prepared, REGEXP_REPLACE(gid, '.* ', '') AS age
FROM pg_prepared_xacts
WHERE prepared + CAST(regexp_replace(gid, '.* ', '') AS INTERVAL) < NOW();

      gid      |           prepared            | age 
---------------+-------------------------------+-----
 foo_insert 1m | 2020-01-27 13:17:33.549918+00 | 1m
(1 row)

This clearly shows a transaction that should no longer be valid. So an external agent or a cron job can easily lookup these transactions and either notify an administrator, or rollback back these transactions. This in my opinion in a simple and easy way of ensuring that orphan transactions can managed in your environment even if transaction manager has failed or a DBA has accidentally left one behind.

Conclusion

Prepared transactions are obviously a really important feature, but requires careful set up of the environment with a fall back notifier or cleanser that can easily ensure that these transactions are not taking up key resources unnecessarily, and the system remains in good shape. The handling of orphaned prepared transactions is still being discussed in the PostgreSQL community. Whether it becomes part of the server core is yet to be seen. In the meantime, we need to use external tools for management of these transactions or hack our way out of this problem. There’s always a better solution around the corner!

Hamid Akhtar

Hamid has more than 19 years of professional software development experience and has been involved with PostgreSQL for more than 8 years now. He was part of EnterpriseDB and managed PostgreSQL official installers for Windows, Linux and MacOS. Hamid has recently joined HighGo Software Inc. as Senior DevOps and Database Architect.

It is important (understatement) that you take regularly scheduled backups of your PostgreSQL system as well as manage how many backups you have, which is known as "backup retention." These best practices ensure that you always have a recent backup of your database system to recover from in the event of a disaster (or use to clone a new copy of your database) and that you don't run out of storage on your backup device or blow up your object storage bill (true story from a previous life, I had a year's worth of nightly backups on S3 when it should have been 21 days...though it was not as expensive as I thought. I also wasn't using pgBackRest which would have caught the error).

When I've given various talks on PostgreSQL and ask the question "do you take regular backups of your production systems," I don't see as many hands raised as I would like (and I'll also use this as an opportunity to say that having a replica is not a backup). However, if you are running PostgreSQL on Kubernetes using the PostgreSQL Operator, with a few commands, the answer to this question is "Yes!"

The PostgreSQL Operator uses the aforementioned open source pgBackRest backup and restore utility to help you manage backups, and includes the ability to schedule regular backups and set retention policies. Even better, as of the release of PostgreSQL Operator 4.2, pgBackRest is enabled by default, which the PostgreSQL Operator leverages to automate a few convenient behaviors, including scheduling full, differential, and incremental backups and setting retention policies!

Let's learn how we can use these features and the PostgreSQL Operator to schedule regular backups and set an appropriate retention policy with the PostgreSQL Operator.

Scheduling Backups

Migrating to PostgreSQL from Oracle is a topic that often comes up in discussions around PostgreSQL.  At Crunchy Data, we are of course not surprised that there is broad interest in moving to the world's most advanced database.

There are a lot of great reasons to migrate to PostgreSQL, and if you haven't looked at PostgreSQL lately, it would be a good time to do so again given the various improvements in PostgreSQL 12.

That said, migrating from one database to another is inherently new technology and can raise a multitude of questions. To help ease the transition, we are providing a few frequently asked questions and answers from users transitioning from Oracle to PostgreSQL, based on real life scenarios, to serve as a helpful resource.

Introduction

I recently did some work with a customer who had some strange behavior happening in his database. When I asked for his logs, I found that each line had a message, and just one timestamp prefixed to it. In other words, he had log_line_prefix = '%t '. This made it hard for me to figure out who did what, especially as his database was serving many clients and applications. It got me thinking, and I scoured through our other clients’ postgresql.conf files that had been shared with me over the years, and in ~130 conf files, I found the following:

• 5% of users don’t change log_line_prefix
• 7% of users don’t log a timestamp (but they might be using syslog, which would include its own timestamp)
• 38% only log a timestamp (and nothing else)
• The average number of parameters included in log_line_prefix is 0.93

A Bit of history

Wait a minute. On average less than one parameter in log_line_prefix for any postgresql.conf? How could that be? Bear in mind that prior to v. 10, the default for log_line_prefix was simply ''. That’s right–nothing. It was up to the DBA to set a value. Seeing that this wasn’t very useful, Christoph Berg submitted a patch to set the default to '%m [%p] '. While it’s not the best setting, it’s a significant improvement to nothing at all. What this does tell me though, is that many users out there using v. 9.x have not bothered to change log_line_prefix at all, making this one of the most neglected important features PostgreSQL has to offer.

More important than some may think

Adequate logging opens up the door to many possibilities. With log_connections and log_disconnections, you can see when a session began and ended. With log_min_duration_statement (along with auto_explain), you can identify any poorly-running queries. With log_autovacuum_min_duration, you can see what an autovacuum job did, how much space it freed up, and perhaps tip you off to any stray/idle transactions preventing you from vacuuming more. Same goes with log_temp_files, which can tip you off to any work_mem adjustments you may need. However, in order of any of this to be possible, log_line_prefix needs to be adequately set.

log_line_prefix can log many important facets of a session or query. There are 17 parameters that can be logged and while not all of them need to be included in your postgresql.conf, here are some of my favorites:

• %a - Application Name - Allows quick reference and filtering
• %u - User Name - Allows filter by user name
• %d - Database Name - Allows filter by database name
• %r - Remote Host IP/Name - Helps identify suspicious activity from a host
• %p - Process ID - Helps identify specific problematic sessions
• %l - Session/Process Log Line - Helps identify what a session has done
• %v/%x - Transaction IDs - Helps identify what queries a transaction ran

These, along with a timestamp (%m) make it possible for a DBA or developer to quickly filter on specific paramters to identify issues and collect historical data. Moreover, log analytics tools like pgbadger work best with a more comprehensive log_line_prefix, so something like log_line_prefix = '%m [%p:%l] (%v): host=%r,db=%d,user=%u,app=%a,client=%h ' is what I like to use. With this, I can do the following:

• grep on [<pid>] to find lines pertaining to a specific active backend to see what it’s done so far (cross referencing with SELECT * FROM pg_stat_acitivity)
• grep on a PID to see what it did before the database crashed
• Filter out apps with a specific application name (like autovacuum– because I know for a fact that autovacuum didn’t cause the problem I’m currently trying to investigate)
• Filter out a specific database name because in my multi-tenant setup, I don’t need to worry about that database
• Focus on a specific transaction ID to help my developer know at which step in his code a query is failing

Without setting log_line_prefix, all you have is a bunch of timestamps and a bunch of queries or error messages, not knowing how it relates with the set of users and applications that might be accessing the database. Setting log_line_prefix will lead to quicker diagnosis, faster incident resolution, happier users, and rested DBAs.

That’s not all!

While setting log_line_prefix is very important, from experience I also think the following are important for DBAs to maintain their sanity:

• log_min_duration_statement– helpful in identifying slow queries
• log_statement– good for auditing purposes
• log_[dis]connections– good for auditing purposes
• log_rotation_age/log_rotation_size– good for organization, and for keeping your logfiles small(ish)
• log_autovacuum_min_duration– gives insight into autovacuum behavior
• log_checkpoints– know what queries happened between checkpoints
• log_temp_files– helps identify work_mem shortages, I/O spikes
• auto_explain– not a parameter, but a useful extension

These parameters will help with diagnosis and in some cases, when coupled with pgbadger or pganalyze, can assist with capacity planning.

How much is too much?

Some may complain that logging too many things will lead to I/O overhead and actually increase time for diagnosis. DBAs don’t want to be drinking from a firehose at 3AM! While this may be true, some steps can be taken to mitigate these effects:

• Rotate your logfiles every hour (or more frequently on super-busy systems)
• Compress your logfiles after they’ve been rotated
• Put your log_directory on a separate partition

Getting familiar with commandline tools like grep, sed, and awk are also very important, so that you can quickly filter and zoom in on the suspected users, transactions, and processes.

Conclusion

There’s no excuse for ignoring logging. PostgreSQL’s logging is very powerful and informative, even without third-party log processing tools like Datadog and Splunk. It’s the first and most powerful resource (along with looking at pg_stat_acitivity) that DBAs have when it comes to figuring out what caused problems in your application stack, and the tell-all snitch when it comes to investigating a database crash. Don’t neglect it!

Recently, I was asked for advice on how to reasonably implement a common task of table change tracking – meaning a so-called “audit trail” for all row changes over time was required. The keyword “compliance” might also ring a bell in this context, here for those who work in finance or for government institutions. But as always with technology, there are a couple of different approaches with different benefits / drawbacks to choose from; let’s lay it out for the Internet Citizens! There’s a summary of pros and cons down below if you’re in a hurry.

TL;DR: sadly, there are no real standard built-in tools for these types of tasks, but in real life it mostly boils down to creating some “shadow” tables and writing some triggers.

The log_statement configuration parameter

This server parameter named log_statement has 4 possible values – none, ddl, mod, all. By default, it’s configured conservatively (in this case, with ‘none’), as all PostgreSQL parameters generally are, in order to avoid accidentally wasting resources. That way, it doesn’t log normal SELECT or DML statements to the server log – only warnings and errors. But when set to ‘mod’ (or ‘all’ which is a superset of ‘mod’), all top level data-changing SQL statements (INSERT, UPDATE, DELETE, TRUNCATE) will be stored in the server log!

Make sure to note the “top level” part though – this means that if you have some business logic exercising triggers on your tables and a chain of changes is triggered, you’ll not see any traces of these subsequent changes in the server log! Also, log_statement doesn’t work well if you rely on stored procedures to change data – since they’re usually called via normal SELECT-s, which don’t get flagged as data-changing operations with the ‘mod’ log_statement level. Worse, if you need to do some dynamic SQL within your stored procedures – even the ‘all’ level won’t catch them!

In short – the use cases for the whole approach are somewhat limited to basic CRUD patterns, and log_statement is not necessarily suitable for enterprise requirements.

PS – also note that superusers can change this log_statement setting on the fly; thus bypassing the auditing, and do stuff secretly– without any traces left! To remind you – “by design” with PostgreSQL, it’s never really possible to guard the system against a malicious superuser. Some methods just need some more work from the user, in short better be careful to whom you hand out superuser rights. Preferably, do not even allow remote superuser access, but that’s another topic – see here for more info, if you’re interested in security.

Pros

• Absolutely the easiest setup ever- only one parameter to enable. This is even true during live operation, so that it can also be used for ad-hoc troubleshooting.

Cons

• Catches only the top level DML statement issued by the user.
• Only explicitly defined column values will be stored to the server log, thus nothing usable from a statement such as: UPDATE tbl SET x = x + 1.
• No row context for multi-row updates – i.e. you’ll have no indication how many rows were altered by some action.
• No table / schema based filtering – all or nothing will be logged.
• Information storage is purely text based – possibly need to deal with huge log files where information is all tangled up, and searching is difficult. Might need additional extraction / parsing / indexing for fast search.
• Queries from failed transactions are also included.

The pgaudit extension

In short, pgaudit is a 3rd-party PostgreSQL extension that tries to improve on the quite limited default PostgreSQL text-based auditing / logging capabilities. It has been around for ages, so it’s stable enough to use, and there are even packages provided by PGDG repos for Debian / RH-based distros.

Its main drawback is the same as with the previous method, though – everything goes to the same server log with normal status / error messages. There’s no clear separation of “concerns”– thus searching will be a bit of work, and for fast “trail” access, you probably need to parse the files and store them in some other system, properly indexed. It’s also the same story for the generated volume of logs. At default settings, (when just enabling all) it’s way more write heavy than the log_statement-based approach. In short, be sure to tune the plentiful parameters to your needs. To warn users about that, the project README also nicely says: … be sure to assess the performance impact while testing and allocate plenty of space on the log volume.

Pros

• Quite granular logging / auditing options. Configurable by change type or by some role’s access privileges to table, view, etc.
• Internally / dynamically generated SQL-s can also be logged.

Cons

• A 3rd party extension.
• Possibly heavy disk / IO footprint, same as for the previous method.
• No row context for multi-row updates.
• Information storage is purely text-based – need to possibly deal with huge log files where information is all tangled up, and searching is difficult. Might need additional extraction / parsing / indexing for fast search.
• Queries from failed transactions are also included.

Custom audit tables and triggers for all tables

Custom audit tables and triggers must be the most classic / common approach to auditing, and all those working with RDBMS systems for a longer period have surely seen or implemented something like these features. The main idea – create a “shadow” history tracking / audit table for all relevant tables, where all changes to the main table will be logged, via triggers. However, since triggers are considered black magic by a lot of developers these days, I’ve also seen implementations via application code…but this can’t be recommended, as only in a non-existent perfect world are there no ad-hoc manual data changes.

The setup process here would look something like what you see below for every target table X, where we want to track who changed what rows / columns and when:

1. Create the “shadow” table for X, typically X_LOG with some typical audit columns like “change time”, “user doing the change”, “type of change” and then all or only important data columns from X.
2. Create a trigger function FX which inserts the new or old values (it’s a matter of taste) into X_LOG with above declared audit and data columns filled.
3. Declare a trigger on table X to call our tracking function FX for each altered row. The trigger would be typically an AFTER trigger as we don’t want to alter anything and just protocol, but when doing heavy multi-row transactions (thousands of rows per TX) it would make sense to test BEFORE triggers as well, as they should be more resource-friendly (give rollbacks / exceptions are rare).

Pros

• Explicit audit targeting, track only exactly what is important
• Isolation of concerns – changes are in a nice table, ready for fast inspection
• Fast search – normal tables can easily be indexed per need

Cons

• Need to write stored procedures and manage triggers definitions. FYI – triggers can also be written in other supported PL languages like Python, if you happen to dislike the de-facto trigger language of PL/pgSQL.
• Some schema duplication.
• Database growth. Previously, changes were written into server logs that are usually recycled and federated in any case, so it was not a big concern. At present, however, the audit tables may need explicit “care”.

One generic audit trigger and table for all audited tables

On a high level, this method is very similar to the previous one; the only change being that instead of having custom audit tables / trigger code for all “business” tables, we create a generic trigger function that can be used on all target tables, and that also logs all changes into a single table! By doing that, we’ve minimized the amount of table / code duplication – which could be of real value for big and complex systems – remember, DRY!

And how, you may wonder, would be the best way to implement it? Well, the best way to achieve such generic behaviour is to utilize the superb JSON functions of PostgreSQL, preferably the JSONB data type (available since v9.4), due to some space saving and faster search capabilities. BTW, if you happen to be running some earlier version, you should really think of upgrading, as versions 9.3 and lesser are not officially supported any more…and soon (February 13, 2020) PostgreSQL 9.4 will stop receiving fixes.

Since this approach is relatively unknown to the wider public, a piece of sample code probably wouldn’t hurt; check below for a sample. Note, however, that fiddling with JSONB along with the fact that this is basically a NoSQL type of storage, is not exactly as effective as normal tables / columns. You’ll have to pay a small performance and storage premium for this “generic” convenience.

CREATE TABLE generic_log (
  mtime timestamptz not null default now(),
  action char not null check (action in ('I', 'U', 'D')),
  username text not null,
  table_name text not null,
  row_data jsonb not null
);

CREATE INDEX ON generic_log USING brin (mtime);
CREATE INDEX ON generic_log ((row_data->>'my_pk’)) WHERE row_data->>'my_pk' IS NOT NULL;  // note the cast to text as JSONB can’t be indexed with B-tree
CREATE EXTENSION IF NOT EXISTS btree_gin;
CREATE INDEX ON generic_log USING gin (table_name);  // GiN is better for lots of repeating values


CREATE OR REPLACE FUNCTION public.generic_log_trigger()
 RETURNS trigger LANGUAGE plpgsql
AS $function$
BEGIN
  IF TG_OP = 'DELETE' THEN
    INSERT INTO generic_log VALUES (now(), 'D', session_user, TG_TABLE_NAME, to_json(OLD));
  ELSE
    INSERT INTO generic_log VALUES (now(), TG_OP::char , session_user, TG_TABLE_NAME, to_json(NEW));
  END IF;
  RETURN NULL;
END;
$function$;

CREATE TRIGGER log_generic AFTER INSERT OR UPDATE OR DELETE ON some_table FOR EACH ROW EXECUTE FUNCTION generic_log_trigger();

Pros

• Less code to manage.
• Automatic attachment of audit trail triggers can be easily configured for new tables, e.g. via event triggers.

Cons

• A bit more resources burned compared to custom per-table triggers
• Some more exotic indexing (GiN) may be needed
• SQL for searching may become a bit more complex

PS – note again, that with both of these trigger-based methods, superusers (or table owners) can temporarily disable the triggers and thus bypass our audit trail.

Logical replication

Logical replication, also known as “pub-sub” replication, is a relatively new thing in PostgreSQL (introduced in v10), and originally, not really an auditing feature but rather a near-zero-downtime upgrade mechanism (read this blog post with more details on that).

It can also be “abused” for auditing or CDC (Change Data Capture) purposes…and actually quite well! The biggest benefit – storage of any extra auditing data can be “outsourced” to an external system, and so-called “write amplification” can be avoided – meaning generally better performance, since extra writing to the disk happens somewhere else.

You need to choose between 2 implementation options though – PostgreSQL native or the custom application way

Logical replication – PostgreSQL native

PostgreSQL native logical replication means that you build up a master server similarly structured to the original server, re-adjust the schema a bit – dropping PK/UQ-s, create some triggers that tweak or throw away uninteresting data or store it in “shadow” tables (just like with the normal trigger-based approaches) and then configure data streaming with CREATE PUBLICATION / CREATE SUBSCRIPTION commands.

As usual, some constraints still apply – you might need to alter the schema slightly to get going. Large objects (up to 4TB blobs) are not supported, and with default settings, you’d only be getting the primary key and changed column data, i.e. not the latest “row image”. Also, it’s generally more hassle to set up and run – an extra node and monitoring is needed, since the publisher and subscriber will be sort of “physically coupled”, and there will be operational risks for the publisher (source server) – if the subscriber goes on a ”vacation” for too long the publisher might run out of disk space as all data changes will be reserved and stored as transaction logs (WAL) until they’re fetched (or the slot deleted). The latter actually applies for the “custom application” approach. So you should definitely spend a minute in careful consideration before jumping into some implementations.

On a positive note from the security side – “PostgreSQL native” can actually be configured in such a way that it’s not even possible for superusers on the source system to disable / bypass the auditing process and change something secretly! (i.e. temporarily leaving out some tables from the replication stream so that the subscriber doesn’t notice!) However, this only works with the standard (for upgrades at least) FOR ALL TABLES setup.

Logical replication – with custom applications

The “application way” means using some programming language (C, Python, Java, etc) where the PostgreSQL driver supports logical decoding. You’ll always be streaming the changes as they happen, and then inspect or stash away the data in your favourite format, or propagate into some other database system altogether. See here for a sample PostgreSQL implementation that also can easily be tested on the command line. To simplify it a bit – you can live-stream JSON change-sets out of PostgreSQL and do whatever you like with the data.

Pros

• Minimal extra resource usage penalty on the source system.
• Can be well-configured on the table level – one could leave out some tables, or only stream certain changes like INSERTS for some tables.
• Can be the safest option with regard to data integrity.
• Subscribers can also be purely virtual, i.e. applications doing “something” with the changed data

Cons

• Somewhat complex to set up and run.
• Postgres-native way requires careful syncing of future schema changes.
• Means coupling of 2 servers via replication slots, so monitoring is needed.

Summary of top pros and cons

Hope you got some ideas for your next auditing project with PostgreSQL!

The post Row change auditing options for PostgreSQL appeared first on Cybertec.

I just discovered a new utility for checking the health of a cluster.

Checking catalogues for corruption with pg_catcheck

Today I discovered a nice tool from EnterpriseDB named pg_catcheck that aims at checking the health of the PostgreSQL catalogs.
As you know, if the catalogs are damaged, the database can quickly get confused and not allow you to use as you wish. Luckily, this is something does not happen very often, or rather I should say I think I’ve seen this happening only once during my career (and I don’t remember the cause).
While I’m not sure I would be able to fix any problem in the catalogues by myself, having a tool that helps me understanding if everything is fine is a relief!

Installing pg_catcheck on FreeBSD

You need to get it from the project repository. There is at the moment one official release, but let’s use the HEAD (after all, releases are for feeble people!).

% git clone https://github.com/EnterpriseDB/pg_catcheck.git % cd pg_catcheck % gmake ... % sudo gmake install ... 

If everything works fine, you will end up with a program named pg_catcheck under /usr/local/bin.

Using pg_catcheck

As you can imagine, you need a database administrator to perform the check. The application supports pretty much the same options than psql to connect, and there’s an extra option --postgresql to indicate you are running against a vanilla PostgreSQL (on the other hand, with --enterprisedb the program will assume you are running an EnterpriseDB instance).

% pg_catcheck --postgresql-U postgres template1 progress: done(0 inconsistencies, 0 warnings, 0 errors