A few days ago a CVE was announced for Postgres. To say this CVE is a bit overblown is an understatement. The first thing to know is you’re likely completely safe. If you run on a managed service provider you are not going to be affected by this, and if you’re managing your own Postgres database all chances are you are equally as safe. This CVE received a note from Tom Lane on the pgsql-announce mailing list in response to it getting a broad amount of awareness and attention.

But, we thought this might be a good time to talk about a few principles and concepts that underly how Postgres works.

For quick context the CVE states that the copy program within Postgres when run as super user will allow arbitrary code execution. The copy program within Postgres is a great tool for bulk loading of data. The utility essentially loads data to/from disk.

Shifting to the Postgres super user… Postgres is an extremely powerful database, well really it is more of a data platform. As super user you can perform all types of powerful operations. Those can include things that touch the underlying system itself, or enabling extensions which allow for low level system access. These types of operations should always be treated with care an caution.

In Tom’s email to the mailing list he states:

We encourage all users of PostgreSQL to follow the best practice that is to never grant superuser access to remote or otherwise untrusted users. This is a standard security operating procedure that is followed in system administration and extends to database administration as well.

If you’re handing out superuser access to your database or connecting to your application with a superuser role, consider changing that immediately.

Superuser essentially has the full power of the system user it is running as. Postgres does not allow running itself as the root user for this very reason. We actually commonly receive the request to grant super user access for customers running on Citus Cloud. Despite it being commonly requested it is not something we support due to the security risks and implications that come with it. Instead we grant you a user with an explicit set of powerful permissions to accomplish what you need but not more. This is standard across most managed service providers as a practice to keep you safe.

When you need to upgrade PostgreSQL, there are a variety of approaches that you can use. To avoid application downtime, then not all of the options to upgrade postgres are suitable. When avoiding downtime is essential, then you might consider using replication as a means of upgrade, and depending on your scenario, you can choose to approach this task using either logical replication or physical (streaming) replication. Here, we take a look at the difference between logical and physical replication in PostgreSQL. Then we explore how to accomplish an upgrade using logical replication in more detail, and by doing so, avoid application downtime. In a subsequent article, we’ll investigate physical replication.

We have already discussed about a few methods available to perform PostgreSQL upgrades in our previous blog posts – PostgreSQL Upgrade Using pg_dumpall and PostgreSQL Upgrade Using pg_dump/pg_restore– as part of our Upgrading or Migrating Your Legacy PostgreSQL to Newer PostgreSQL Versions series. However, both of these methods involve downtime to application.

Types of logical replication

Here we’ll cover two types of replication you could implement:

1. Replication between PostgreSQL 10 and 11 versions using built-in logical replication.
2. Replication between PostgreSQL 9.4 or (<  PG 11) to PostgreSQL 11 using an extension named pglogical .

We might opt to implement replication as a method of upgrade to minimize downtime for the application. Once all the data to date has been replicated to another PostgreSQL server on the latest version, you can simply switch your application to the new server with a minimal downtime… though of course this does depends on the complexity of your application stack.

Logical replication in PostgreSQL allows users to perform a selective replication of a few tables and open a standby for writes. Whereas physical replication in PostgreSQL is a block level replication. In this case, each database in the master is replicated to a standby, and the standby is not open for writes. Going forward, we’ll refer to physical replication as streaming replication.

With logical replication, a standby can have replication enabled from multiple masters. This could be helpful in situations where you need to replicate data from several PostgreSQL databases (OLTP) to a single PostgreSQL server for reporting and data warehousing.

One of the biggest advantages of logical over streaming replication is that logical replication allows us to replicate changes from an older version PostgreSQL to a later version. Streaming replication works only when both the master and standby are of the same major version. We’d recommend they have same minor version too for best practice.

Replication between PostgreSQL 10 and 11 versions

Starting from PostgreSQL 10, logical replication is available with the PostgreSQL source by default. So, you could easily replicate a PostgreSQL 10 database to PostgreSQL 11. Logical Replication uses a publish and subscribe model. The node that sends the changes becomes a publisher. And the node that subscribes to those changes becomes a subscriber. You may have one or more subscriptions to a publication.

Publication

Publication is a set of changes generated from a group of tables. It is referred to as a change set or replication set. Publications can only contains tables and cannot contain any other objects. DMLs on these tables can be replicated but not DDLs.

In a publication, you can choose what type of DML to replicate: INSERT or DELETE or UPDATE or ALL. By default, it is ALL. You must have a replica identity set on the table being published to replicate UPDATES and DELETES to a subscriber. A replica identity set helps in identifying the rows to be updated or deleted.

The primary key of a table is its default replica identity. You can also make a unique index with NO NULL values as a replica identity. If there is no primary key or a unique index with NO NULLs, then you can set the replica_identity to FULL. When a replica identity is set to FULL, postgres uses the entire row as a key. Of course, this may be inefficient.

You might see ERRORS if a table with no primary key and a non-default replica identity has been added to a publication after an UPDATE or a DELETE operation.

Subscription

A subscriber can subscribe to one or more publications. Before adding the subscription, you must ensure that the tables being replicated have been created in the subscriber node. In order to achieve that, you can perform a schema-only dump from publisher to subscriber node.

An example of logical replication

The following example steps work for logical replication between PostgreSQL 10 and 11 versions only.

On the publishing node, create a publication. You can either add all tables or choose to add selected tables to the publication.

-- For adding ALL Tables in Database
CREATE PUBLICATION percpub FOR ALL TABLES;
-- For adding Selected Tables in Database
CREATE PUBLICATION percpub FOR TABLE scott.employee scott.departments;

On the subscriber node, create a subscription that refers to the publication on the publisher node. Perform a DDL dump of the tables to the subscriber node before creating the subscription, as mentioned above,

$ pg_dump -h publisher_server_ip -p 5432 -d percona -Fc -s -U postgres | pg_restore -d percona -h subscriber_node_ip -p 5432 -U postgres
CREATE SUBSCRIPTION percsub CONNECTION 'host=publisher_server_ip dbname=percona user=postgres password=secret port=5432' PUBLICATION percpub;

The above command also copies the pre-existing data from the tables. If you want to disable the copy of the pre-existing data, you can use the following syntax. It will then only start copying the changes to the publisher after you run this command.

CREATE SUBSCRIPTION percsub CONNECTION 'host=publisher_server_ip dbname=percona user=postgres password=oracle port=5432' PUBLICATION percpub WITH (copy_data = false);

Monitor the replication using the following command on the publishing node.

$ psql
\x
select * from pg_stat_replication;

Replication between PostgreSQL 9.4 and PostgreSQL 11

Now, what about the versions that are older than PostgreSQL 10? For this purpose, there is an extension named

pglogical

The following sequence of steps demonstrates a high-level procedure to setup replication between PG 9.4 and PG 11 using pglogical extension.

Step 1 : Consider pgserver_94 to be the source server with a database : percona_94 running on PostgreSQL 9.4. Create the following extensions.

[pgserver_94:] $psql -d percona_94 -c "CREATE EXTENSION pglogical_origin"
CREATE EXTENSION
[pgserver_94:] $psql -d percona_94 -c "CREATE EXTENSION pglogical"
CREATE EXTENSION

Step 2 : Now, you can go ahead and add either selected tables or all the tables in a schema or multiple schemas for replication. In the following example, you can see an error when there is no primary key on one of the tables.

[pgserver_94:] $psql -d percona_94
psql (9.4.21)
Type "help" for help.
percona_94=# SELECT pglogical.create_node(node_name := 'provider1',dsn := 'host=192.168.0.24 port=5432 dbname=percona_94');
create_node
-------------
2976894835
(1 row)
percona_94=# SELECT pglogical.replication_set_add_all_tables('default', ARRAY['public']);
ERROR: table pgbench_history cannot be added to replication set default
DETAIL: table does not have PRIMARY KEY and given replication set is configured to replicate UPDATEs and/or DELETEs
HINT: Add a PRIMARY KEY to the table
percona_94=# ALTER TABLE pgbench_history ADD PRIMARY KEY (tid,aid,delta);
ALTER TABLE
percona_94=# SELECT pglogical.replication_set_add_all_tables('default', ARRAY['public']);
replication_set_add_all_tables
--------------------------------
t
(1 row)

Step 3

On the subscriber node, which is our PostgreSQL 11 database, you can run similar commands as follows.

[pgserver_11:] $psql -d percona_11
psql (11.2)
Type "help" for help.
percona_11=# SELECT pglogical.create_node(node_name := 'subscriber1',dsn := 'host=127.0.0.1 port=5432 dbname=percona_11 password=secret');
 create_node
-------------
   330520249
(1 row)
percona_11=# SELECT pglogical.create_subscription(subscription_name := 'subscription1',provider_dsn := 'host=192.168.0.24 port=5432 dbname=percona_94 password=secret');
 create_subscription
---------------------
          1763399739
(1 row)

Step 4 You can then validate the replication status by querying a few tables pglogical always updates:

percona_11=# select * from pglogical.local_sync_status;
 sync_kind | sync_subid | sync_nspname |   sync_relname   | sync_status | sync_statuslsn
-----------+------------+--------------+------------------+-------------+----------------
 f         | 1763399739 | public       | pgbench_accounts | r           | 0/2EB7D48
 f         | 1763399739 | public       | pgbench_history  | r           | 0/2EB7D48
 f         | 1763399739 | public       | pgbench_tellers  | r           | 0/2EB7D48
 f         | 1763399739 | public       | pgbench_branches | r           | 0/2EB7D48
 d         | 1763399739 |              |                  | r           | 0/0
(5 rows)
percona_11=# select * from pglogical.subscription;
   sub_id   |   sub_name    | sub_origin | sub_target | sub_origin_if | sub_target_if | sub_enabled |             sub_slot_name              |         sub_rep
lication_sets          | sub_forward_origins | sub_apply_delay
------------+---------------+------------+------------+---------------+---------------+-------------+----------------------------------------+----------------
-----------------------+---------------------+-----------------
 1763399739 | subscription1 | 2976894835 |  330520249 |    2402836775 |    2049915666 | t           | pgl_percona_11_provider1_subscription1 | {default,defaul
t_insert_only,ddl_sql} | {all}               | 00:00:00
(1 row)

Primary key selection

At step 2, you saw how all the tables of schema : public got added to a replication set by creating a primary key on the table that doesn’t currently have one. The primary key I chose may not be the right one for that table as it is just for demonstration. However, when you choose a primary key, make sure that you are selecting the right one. It needs to be always unique and use column(s) that don’t normally contain NULLs. If you don’t research primary key selection thoroughly, you could cause downtime to your application. Here’s an example error that you could encounter:

[pgserver_94:] $pgbench -c 10 -T 300 -n percona_94
Client 7 aborted in state 12: ERROR: duplicate key value violates unique constraint "pgbench_history_pkey"
DETAIL: Key (tid, aid, delta)=(7, 63268, 2491) already exists.

So far we have seen how you can use pglogical to create replication between an older version to a newer version PostgreSQL. After you have set up replication, you can easily switch your applications to the latest version with a lower downtime.

To see more in action, why not subscribe to our webinar? And don’t forget at Percona Live in Austin, May 28-30 2019, we have two days of PostgreSQL content in a postgres dedicated track.

Announcing Release 10 of the PostgreSQL Buildfarm client

Principal feature: support for non-standard repositories:

• support multi-element branch names, such as “dev/featurename” or “bug/ticket_number/branchname”
• provide a get_branches() method in SCM module
• support regular expression branches of interest. This is matched against the list of available branches
• prune branches when doing git fetch.

This feature and some server side changes will be explored in detail in my presentation at pgCon in Ottawa next month. The feature doesn’t affect owners of animals in our normal public Build Farm. However, the items below are of use to them.

Other features/ behaviour changes:

• support for testing cross version upgrade extended back to 9.2
• support for core Postgres changes:
  • extended support for USE_MODULE_DB
  • new extra_float_digits regime
  • removal of user table oid support
  • removal of abstime and friends
  • changed log file locations
• don’t search for valgrind messages unless valgrind is configured
• make detection of when NO_TEMP_INSTALL is allowed more bulletproof

There are also various minor bug fixes and code improvements.

The release can be downloaded from https://github.com/PGBuildFarm/client-code/archive/REL_10.tar.gz or https://buildfarm.postgresql.org/downloads/latest-client.tgz

Needless to say, security is a topic that nobody in the wider IT industry can ignore nowadays, with a constant flow of reports on data breaches of various scales. Most of such cases don’t result from direct attacks against databases though, but more from targeting Application / API / Webserver problems as database servers are usually just not directly exposed to the Internet. And even if they were, PostgreSQL installations at least have a good starting point there, as the default settings are sane enough to only listen to local connections by default, preventing from most embarrassing kind of headlines. But of course PostgreSQL setups can also be compromised in many ways, so here’s a short listing of possible threats and some simple suggestions to mitigate those threats if possible.

Unnecessary PostgreSQL superuser access

This my friends, is the #1 point of danger in my opinion, so an appropriate time to repeat the classics – ”with great power comes great responsibility”. Superuser accounts are of course needed for maintenance so cannot be really disabled…but mostly they are vastly overused, even for simple things like normal schema evolution, which can be (mostly) perfectly managed with standard “owner privileges”. But Ok, about the threats – there’s the obvious: pumping out / dropping all table data. But the most important thing to remember – superusers can execute random commands on the OS level, under PostgreSQL process owner privileges. The easiest way to do that is to use the relatively unknown PROGRAM variation of the COPY command, which is also nicely documented as such. Some other methods to achieve the same that are quite hidden and sneaky:

• Achieving physical access to the machine with COPY PROGRAM

This attack assumes that the server has been setup for passwordless communication / data transfer over SSH using default settings (which is very common). The trick itself is disturbingly easy – just create a table with a single text field and execute something like that:

krl@postgres=# create table ssh(data text);
CREATE TABLE
krl@postgres=# \copy ssh from program 'cat ~/.ssh/id_rsa.pub'
COPY 1

The rest should be obvious – copy the key to a local RSA key file…and “ssh” to the machine. Given here you know the operating system user under which Postgres is running – most often it will be the default “postgres” or the key itself could include it. From there a malicious use can easily find some security holes on old unpatched systems and possibly advance to “root”, doing additional harm outside of PostgreSQL, meaning possible “game over” situation for the whole business. FYI – “COPY PROGRAM” works with Postgres 9.3 and newer, and sadly there’s no configuration flag to disable it. Functionality is only available to superusers though.

• “archive_command” abuse

When archiving is enabled, this seemingly innocent parameter can be abused to do nasty things. An example:

ALTER SYSTEM SET archive_command = 'rm -rf /home/postgres';
SELECT pg_reload_conf();
SELECT pg_switch_wal();    -- oopsie

• “file_fdw” arbitrary program “reads”

From version 10 it’s possible to define “virtual” tables that gather data from an external program’s output, with “legal” use cases encompassing reading compressed files and maybe some Webservice usage…but the command could be abused the same way as with previous example, Postgres just executes the command “as is”.

CREATE EXTENSION file_fdw;
CREATE SERVER csv FOREIGN DATA WRAPPER file_fdw;
CREATE FOREIGN TABLE test(data text) SERVER csv OPTIONS (program 'rm -rf /home/postgres');
SELECT * FROM test; -- #$@&%*!

• Untrusted PL-languages

When for example PL/Pythonu is installed it also childs play to execute random commands:

CREATE FUNCTION evil_stuff()
 RETURNS void
 LANGUAGE plpythonu
AS $SQL$
import subprocess
subprocess.call(["rm", "-rf", "/var/lib/postgresql/"])
$SQL$;
SELECT evil_stuff();

Mitigation: as limiting superuser actions is very difficult (the provided threat examples, excluding COPY PROGRAM, can be mitigated with careful setup though) so the only real strategy is – applications should be designed so that superuser access is not needed. And this should be very doable for 99% of users as for example fully managed PostgreSQL services, where you never get superuser access, have become quite popular among developers. Basic principle – hand out superuser access only to those who have physical server access also, as one can basically draw an equality sign between the two (see the above section about COPY PROGRAM).

Brute force password guessing

Second place on my personal threat list goes to weak passwords. What makes up for a weak password is probably debatable…but for me they are some shortish (<10 chars) words or names taken from English speaking domain, combined maybe with some number, i.e. anything that is too short and not randomly generated. The problem is that for such password there are quite some “top” lists available for download after some short Googling, like the “Top 1 Million most common passwords” here.

So what can you do with such lists in PostgreSQL context? Well, quite a bit…given that by default Postgres weirdly enough has no protection against brute-force password guessing! And brute force is extremely easy to perform, for example with something like that (disclaimer: you might get into trouble if trying to run that on a database that you don’t control anyways):

sudo apt install -y hydra
curl -s https://raw.githubusercontent.com/danielmiessler/SecLists/master/Passwords/Common-Credentials/10-million-password-list-top-1000000.txt > top-1m-pw.txt
hydra -l postgres -P top-1m-pw.txt postgres://mydb:5432
...
[STATUS] 17389.00 tries/min, 17389 tries in 00:01h, 982643 to do in 00:57h, 16 active
...
# … and maybe after some time, voila!

Couldn’t be easier. On my laptop, connecting to ‘localhost’, the tool managed ~290 connections attempts per second…which is not too much though, being a generic framework. So with some simple (single-threaded only, leaving lots to improve) custom Go code I managed to double it…so in short if you have a really weak password it could only take less than an hour before someones eyes will start glowing with excitement:)

Mitigation: 1) Use long, random passwords. 2) Install and configure the relatively unknown “auth_delay” contrib extension, that will halt the communication for a moment (configurable) in case a login attempt fails, making it slower to guess the passwords, even when opening many connections in parallel. Attention RDS Postgres users – funnily enough the extension is not whitelisted, so better make sure your passwords are good. 3) Actively monitor the server logs – doing full power brute-forcing generates many GB-s of logs per hour with FATAL entries, which should be picked up by a good monitoring setup.

Man in the middle attacks

Out of the box PostgreSQL is not immune to this quite a known security risk, where the main threat is that with some work someone can intercept / proxy all communication and also reverse engineer the password hashes (given default md5 hashes used) when they have access to the network for longer periods.

Mitigation: 1) As a minimum force SSL connections. Just remember that enabling “ssl” in the postgresql.conf doesn’t cut it, it needs to be enforced additionally also in the pg_hba.conf. 2) Phase out “md5” passwords in favor of the newer and more secure “scram-sha-256” – not all drivers sadly support it still as it’s relatively new :/ 3) When highest confidence is needed – employ certificates i.e. use “verify-ca“ or “verify-full” SSL modes when connecting.

Loose backups or logs

That’s a classic mistake. I’ve seen plenty of setups with very tight “online” access…but where at the end of the day Postgres backups or server logs (server logs can possibly also contain plain text passwords!) are placed onto a backup server together with some application server logs where more people have access to them, to for example analyze some other aspects of application behaviour. In encrypted form this would be of course somewhat OK, but just using a different folder and hoping application is a disaster waiting to happen.

Mitigation: 1) Clearly separate database backups / logs from anything else so that only DBAs would have access. 2) Use “at rest” encryption.

Disk space attack

This one might be also a minor one as it assumes a malicious inside job or a an accident by non-aware Postgres user. The fact is that at default settings, all users with login rights, even the normal unprivileged ones, could create both persistent and temporary objects (mostly tables) in the default “public” schema. That is of course not bad “per se” and adds convenience, but can again be abused – to create a table and insert so many rows into it (think generate_series) that server IO is depleted or the disk space runs eventually out and Postgres crashes.

Mitigation: 1) Don’t use “public” schema at all for important applications and create app specific schemas. 2) When already using “public” schema, then for those systems where temporary / auxiliary tables should actually not be needed for normal users, one should strip the CREATE rights from everyone besides maybe the schema owner. In SQL this would be something like:

REVOKE CREATE ON SCHEMA public FROM PUBLIC;
-- then hand out grants for those who really need them
GRANT CREATE ON SCHEMA public TO app_owner_role;

3) For more or less similar reasons also set the “temp_file_limit” parameter to some reasonably low value. The parameter controls how much disk space all temporary files of a query can maximally occupy. This is relevant only for transparent temp files caused by heavy queries though.

Distributed Denial of Service attacks for publicly exposed DBs

As a last threat, as with any services accessible over the Internet, also databases could fall under a DDoS attack if you have managed to annoy some competing companies or interest groups enough. And here actually there’s nothing that can be done on the Postgres side – even when you’ll configure the Host-based access config (pg_hba.conf) to “reject” all connection attempts from all unwanted IP ranges, it would be too much for the server to handle. This kind of filtering needs to happen one level lower in the stack, but luckily there are lot’s of software products / services acting as gateways to your DB (just Google “cloud based ddos protection”) filtering packets coming from know botnets and such. Some IaaS providers on who’s hardware your VMs are possibly running also have some basic protection often built-in. With DDoS the only consolidation is that you’re not losing any data but “just” revenue / customers.

Mitigation: 1) Try to avoid direct public access to your database. 2) Invest in some anti-DDoS service.

So that was my security checklist. Hope it made you think and maybe also go and check / change some server settings Thanks for reading and please let me know in the comments section if I missed something important.

The post Secure PostgreSQL – a reminder on various attack surfaces appeared first on Cybertec.

{
  "key1": "aaaa", /* 4 ^ 1 */
  "key2": "aaaaaaaaaaaaaaaa", /* 4 ^ 2 = 16 */
  ...
  "key10": "aaa ... aaa" /* 4 ^ 10 = 1M */
}

create table t(jb jsonb);
insert into t select (
  select jsonb_object_agg('key' || i, repeat('a', pow(4, i)::int)) from generate_series(1,10) i
) from generate_series(1, 1000);

select jb->'key1' from t;

key1-key5   key7    key8     key9     key10
  10 ms     48 ms   152 ms   548 ms   2037 ms

PostgreSQL is a powerful open source relational database management system. It extends the SQL language with additional features. A DBMS is not only defined by its performance and out of the box features, but also its ability to support bespoke/additional user-specific functionality. Some of these functionalities may be in the form of database constructs or modules, like stored procedures or functions, but their scope is generally limited to the functionality being exposed by the DBMS. For instance, how will you write a custom query-analyzing application that resides within your DBMS?

To support such options, PostgreSQL provides a pluggable architecture that allows you to install extensions. Extensions may consist of a configuration (control) file, a combination of SQL files, and dynamically loadable libraries.

This means you can write your own code as per the defined guidelines of an extension and plug it in a PostgreSQL instance without changing the actual PostgreSQL code tree. An extension by very definition extends what PostgreSQL can do, but more than that, it gives you the ability to interact with external entities. These external entities can be other database management systems like ClickHouse, Mongo or HDFs (normally these are called foreign data wrappers), or other interpreters or compilers (thus allowing us to write database functions in another language like Java, Python, Perl or TCL, etc.). Another potential use case of an extension can be for code obfuscation which allows you to protect your super secret code from prying eyes.

Build your own

To build your own extension, you don’t need a complete PostgreSQL code base. You can build and install an extension using installed PostgreSQL (it may require you to install a devel RPM or Debian package). Details about extensions can be found in PostgreSQL’s official documentation[1]. There many extensions available for different features in the contrib directory of PostgreSQL source. Other than the contrib directory, people are also writing extensions readily available on the internet but currently not part of the PostgreSQL source tree. The pg_stat_statements, PL/pgSQL, and PostGIS are examples of the best known or most widely used extensions.

Generally available PostgreSQL extensions may be classified into four main categories:

• Add support of a new language extension (PL/pgSQL, PL/Python, and PL/Java)
• Data type extensions where you can introduce new ((Hstore, cube, and hstore)
• Miscellaneous extensions (contrib folder has many miscellaneous extensions)
• Foreign Data Wrapper extension (postgres_fdw, mysqldb_fdw, clickhousedb_fdw)

There are four basic file types that are required for building an extension:

• Makefile: Which uses PGXS PostgreSQL’s build infrastructure for extensions.
• Control File: Carries information about the extension.
• SQL File(s): If the extension has any SQL code, it may reside in form SQL files (optional)
• C Code: The shared object that we want to build (optional).

Extension Makefile

To compile the C code, we need a makefile. It’s a very simple makefile with the exception of “PGXS”, which is PostgreSQL’s infrastructural makefile for creating extensions. The inclusion of “PGXS” is done by invoking pg_config binary with “–pgxs” flag. The detail of that file can be found at GitHub[2].

This is a sample makefile, which can be used to compile the C code.

EXTENSION = log
MODULE_big = log
DATA = log--0.0.1.sql
OBJS = log.o
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

Extension Control File

This file must be named as [EXTENSION NAME]. control file. The control file can contain many options which can be found at official documentation [3]. But in this example, I have used some basic options.

comments: Comments about the extension.

default_version: This is the extension SQL version. The name of the SQL file contains this information in the file name.

relocatable:  Tells PostgreSQL if it is possible to move contained objects into a different schema.

module_pathname:  This information is replaced with the actual lib file path.

comment = 'PostgreSQL Utility Command Logger
'default_version = '0.0.1'
relocatable = true
module_pathname = '$libdir/log'

The contents of the file can be seen using the psql command \dx psql.

postgres=# \dx log      
List of installed extensions Name   | Version | Schema |       Description
------------------------------------+---------+--------+----------------------------------
log                                 | 0.0.1   | public | PostgreSQL Utility Command Logger

Extension SQL File

This is a mapping file, which I used to map the PostgreSQL function with the corresponding C function. Whenever you call the SQL function then the corresponding C function is called. The name of the file must be [EXTENSION NAME]–[default-version].sql. This is the same default_version as defined in the control file.

CREATE FUNCTION pg_all_queries(OUT query TEXT, pid OUT TEXT)
RETURNS SETOF RECORDAS 'MODULE_PATHNAME',
'pg_all_queries'
LANGUAGE C STRICT VOLATILE;

Extension C Code

There are three kinds of functions you can write in c code.

The first is where you call your c code function using SQL function written in SQL file of the extension.

The second type of function is callbacks. You register that callback by assigning the pointer of the function. There is no need for an SQL function here. You call this function automatically when a specific event occurs.

The third type of function is called automatically, even without registering. These functions are called on events such as extension load/unload time etc.

This is the C file containing the definition of the C code. There is no restriction for the name, or of the number of C files.

#include "postgres.h"
/* OS Includes */
/* PostgreSQL Includes */
PG_MODULE_MAGIC;
void _PG_init(void);
void _PG_fini(void);
Datum pg_all_queries(PG_FUNCTION_ARGS);
PG_FUNCTION_INFO_V1(pg_all_queries);
static void process_utility(PlannedStmt *pstmt, const char *queryString,ProcessUtilityContext context,ParamListInfo params,QueryEnvironment *queryEnv,DestReceiver *dest,       char *completionTag);

You need to include postgres.h for the extension. You can include other PostgreSQL as needed. PG_MODULE_MAGIC is macro which you need to include in the C file for the extension. _PG_init and _PG_fini are functions that are called when the extension is loaded or unloaded, respectively.

Here is an example of the load and unload functions of an extension.

void _PG_init(void)
{
    /* ... C code here at time of extension loading ... */
    ProcessUtility_hook = process_utility;
}
Void _PG_fini(void)
{
    /* ... C code here at time of extension unloading ... */
}

Here is an example of a callback function that you can invoke whenever you call a utility statement e.g. any DDL statement. The “queryString” variable contains the actual query text.

static void process_utility(PlannedStmt *pstmt,
                           const char *queryString,
                           ProcessUtilityContext context,
                           ParamListInfo params,
                           QueryEnvironment *queryEnv,DestReceiver *dest,
                           char *completionTag)
{
    /* ... C code here ... */
    standard_ProcessUtility(pstmt,   
                            queryString,   
                            context,   
                            params,   
                            queryEnv,   
                            dest,
                            completionTag);
    /* ... C code here ... */
}

Finally, here’s an example of a C function that is invoked through a user-defined SQL function. This internally calls the C function contained in our shared object.

Datum pg_all_queries(PG_FUNCTION_ARGS)
{
    /* ... C code here ... */
    tupstore = tuplestore_begin_heap(true, false, work_mem);
    /* ... C code here ... */     
    values[0] = CStringGetTextDatum(query);
    values[1] = CStringGetTextDatum(pid);
    /* ... C code here ... */
    tuplestore_donestoring(tupstore);
    return (Datum) 0;
}

Compile and Install

Before compilation, you need to set the PATH for PostgreSQL’s bin directory if there is no pg_config available in the system paths.

export PATH=/usr/local/pgsql/bin:$PATH

make USE_PGXS=1

make USE_PGXS=1 install

Output

We can now use our extension through a simple SQL query. Following is the output that is coming straight out of extension written in C programming language.

postgres=# select * from pg_all_queries();          
query                      | pid
--------------------------+|------
create table foo(a int);  +| 8196
create table bar(a int);  +| 8196
drop table foo;           +| 8196
(3 rows)

I hope this example can serve as a starting point for you to create more useful extensions that will not only help you and your company, but will also give you an opportunity to share and help the PostgreSQL community grow.

The complete example can be found at Github[4].

[1]: https://www.postgresql.org/docs/current/external-extensions.html

[2]: https://github.com/postgres/postgres/blob/master/src/makefiles/pgxs.mk

[3]: https://www.postgresql.org/docs/9.1/extend-extensions.html

[4]: https://github.com/ibrarahmad/Blog-Examples/tree/master/log

—
Photo from Pexels

A while back, 2ndQuadrant notified a few of us that we should get more involved in Postgres Development in some capacity. Being as I’ve essentially fallen off the map in corresponding with the mailing lists in general, it would be a good way to get back into the habit.

But wait! Don’t we want more community involvement in general? Of course we do! So now is also a great opportunity to share my journey in the hopes others follow suit. Let’s go!

In the Beginning

So how does one start contributing? Do I have to know C? Must I poke around in the internals for weeks to form an understanding, however so tenuous, of the Black Magic that animates Postgres? Perhaps I should chant incantations to summon some dark entity to grant otherworldly powers necessary to comprehend the mind-bleedingly high-level conversations regularly churning within the deeply foreboding confines of the Hackers mailing list.

No.

God no.

If those were the prerequisites for getting involved, there would be approximately one person pushing Postgres forward, and his name is Tom Lane. Everyone else would be too terrified to even approach the process. I certainly count myself among those more timid individuals.

Instead, let’s start somewhere simple, and with something small. Sometimes all it takes to make a difference is to piggyback on the coattails of someone else who knows what they’re doing. If we’re too inexperienced to submit a patch, maybe we can review one instead.

Getting the Party Started (In Here)

Postgres development marches forward steadily in a form of punctuated equilibrium. Every few months, we throw a bunch of patches against the wall, and see what sticks. That cadence is currently marshaled by a master list of past and present commit fests going back to late 2014. It’s hardly an exhaustive resource dating back to antiquity, but it doesn’t need to be.

All we really need is the most recent iteration that’s in progress. At the time of this writing, that’s 2019-03.

Decisions, Decisions

Now for what might just be the most difficult portion of the whole endeavor: choosing a patch to review. Patches are grouped by category, and range from the relatively mundane documentation revision, to the more brain numbing voodoo of planner bugs.

Some of these, by their very nature, may seem storied and impenetrable. The Logical decoding of two-phase transactions patch for example, has been in commitfests since early 2017! It’s probably best to focus on patches that offer high utility, or seem personally interesting. There’s certainly no shortage of selection!

Just browse through a few, click through the patches until one seems interesting, and read the emails. As a personal hint, use this link after opening the "Emails" link in another tab:

It will let you view the entire conversation on the patch until now, and can be invaluable for understanding some of the background before moving forward.

I personally elected to review the patch on a separate table level option to control compression for a few reasons:

1. It’s really just an extension on the Postgres Grand Unified Config based on a parameter that already exists: toast_tuple_target.
2. The patch included documentation for the new parameter, so I’d have some basis for how it should work.
3. The patch included tests, so I could compare how Postgres behaved with and without it.
4. The existing conversation was relatively short, so I could provide some meaningful input that wouldn’t be immediately overshadowed by more experienced participants. Gotta maximize that value!
5. More selfishly, I worked with the author, so I could clarify things off list if necessary.

With that out of the way, it was time press "Become Reviewer" to get to "work".

Bits and Pieces

Each patch summary page will, or should, list all of the patches in the "Emails" pane. Simply grab the latest of these and download it into your local copy of the Postgres source.

Wait, you don’t have one of those? Do you have git installed? If not, that’s a whole different conversation. So let’s just assume git is available and get some Postgres code:

git clone git://git.postgresql.org/git/postgresql.git

Then the easiest way to proceed is to work on your own local branch. In most cases, we want to apply our patch to HEAD, so we just need to create a new branch as a snapshot of that before we start reviewing. This is what I did:

git checkout -b cf_table_compress

Now when HEAD advances, it won’t mess with our testing until we rebase or pull HEAD into our branch. But what about obtaining and applying the patch? Remember those email links? Know how to use wget? Great! Here’s the patch I retrieved:

wget https://www.postgresql.org/message-id/attachment/99931/0001-Add-a-table-level-option-to-control-compression.patch

Then we need to apply the patch itself. The patch utility applies patches to code referenced in the patch file. Using it can sometimes be confusing, but there’s really only two things relevant to us: the -p parameter, and the patch itself.

Just drop the patch in the base folder of your Postgres repository clone, and do this:

patch -p1 < 0001-Add-a-table-level-option-to-control-compression.patch

Thus a level-1 prefix application of the supplied patch is now applied to the Postgres code, assuming there were no errors. That in fact, can be our first criteria.

1. Did the patch apply cleanly?

Compile, compile, compile

So what now? Well, really we just follow the usual process of building most code with one exception. If we want to test a running Postgres instance, it's best to put the binaries somewhere they won't interfere with the rest of the system, a container, or otherwise. The easiest way is to just set the --prefix before building.

So do this:

./configure --prefix=/opt/cf_table_compress
make
make install

And it's also good to do this:

make check
make installcheck

That will invoke the test routines both before and after installing the build, to prove everything works. It's a good idea to do this even if the patch didn't include its own tests, because the patch itself may have broken existing functionality. So now we have some additional criteria to share:

1. Did the code compile cleanly?
2. Did all tests complete successfully?
3. Did the build install properly?
4. Does the installed version work?
5. Did tests succeed against the installed version?

Blazing a New Trail

At this point we're free to apply any further criteria we deem necessary. Some starting points that could prove useful:

• Does the patch do what it claims?
• Is the functionality accurately described?
• Can you think of any edge cases it should handle? Test those.
• Is there anything in the documentation that could be clarified?
• If applicable, were there any performance regressions?

Really, the sky is the limit. Just remember that sunlight is the best disinfectant. You're not just proving contributing isn't as hard as it looks, your efforts can directly make Postgres better by proving a patch is worthy of inclusion. You, yes you, have the power to reveal bad behavior before it becomes a bug we have to fix later.

Take as much time as you need, and be as meticulous as you feel necessary. Just share your observations before too long, or existing reviewers or constraints of the commitfest itself may beat you to the punch.

Sharing the Wisdom

Remember that original email thread that the patch summary page referenced? Once a patch has been put through its paces, it's time to tell everyone what you did, how it went, and how many thumbs (in which directions) the patch deserves.

Did it completely destroy everything, corrupt all data you inserted, and also make vague threats against your pets? Make a note about all of that. Did it perform as expected, and also clean your gutters before the spring rainfalls? We want to know that too. You were the one performing the tests, and did all the hard work to decide what you wanted to verify or augment. Don't keep it to yourself!

With that said, there really is no formal review procedure. There appears to be a loosely advocated format, but really all that's necessary is to convey your experiences without omitting anything critical. My own initial effort was probably unduly ostentatious, but hey, I'm still learning. It's OK to be a bit awkward until experience digs a more comfortable rut.

We want to hear from you! We crave your input! Be a part of the Postgres team.

You know you want to.

PostgreSQL has been my livelihood since 2004 – so I am naturally biased in its favor. I think it is the coolest piece of software on the planet, and I am not alone.

DB-Engines

See those 2 badges up there? That’s 2 years in a row. DB-Engines monitors a total of 343 databases and their trends. For each of 2017 and 2018, DB-Engines states that PostgreSQL gained more popularity in their rankings than any other database – hence the award.

This is no small feat. DB-Engines has a complex methodology of calculating popularity including search engine queries, technical discussions on popular forums, related job offers, and social media trends. Coming out on top 2 years in a row shows how much technologists love Postgres.

Stack Overflow

You Google for the solution of a problem you are facing. Chances are the first hit you are going to get will be a link from Stack Overflow. Millions of registered users interact hundreds of thousands of times every day answering questions, sharing knowledge, and making technology so much more easy to use for fellow enthusiasts.

Stack Overflow runs a survey each year asking the developer community about their favorite technologies. Want to guess what I am driving at?

That’s the result of over 100,000 technology enthusiasts from across the world coming together and loving PostgreSQL.

Hacker News

Hacker news – run by Y Combinator, one of the top USA accelerators since 2005 – is another forum that brings together technology enthusiasts from all around the world.

Do I really need to label the blue line? ;-)

The graph above posts trends over the past 8 years and compares popularity between MySQL, SQL Server, MongoDB, and our rising star – PostgreSQL. And just look at how it is rising!

Postgres is definitely the coolest database out there, and this is only reason #1 :-)

Stay tuned for more!

If you’ve visited almost any web page on the PostgreSQL Build Farm server in the last few days you might have noticed that it is sporting a new RSS feed, of changes in status. This is similar to the information on the buildfarm-status-green-chgs mailing list, except that it has all status changes, not just to and from green. This new feature fulfills a long outstanding request.

pg_checksums is a renaming of the tool called pg_verify_checksums which has been introduced in Postgres 11. Version 12 is introducing new options and possibilities which explain the renaming, as the tool has become much more multi-purpose.

First, it is now possible to enable and disable checksums for an offline cluster:

commit: ed308d78379008b2cebca30a986f97f992ee6122
author: Michael Paquier <michael@paquier.xyz>
date: Sat, 23 Mar 2019 08:12:55 +0900
Add options to enable and disable checksums in pg_checksums

An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.

When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged.  If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).

Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de

Here is how it works. The tool is able to do three modes now in total:

• –check, the default if nothing is specified and what pg_verify_checksums was already able to do. This mode scans all the relation file blocks, reporting any mismatch.
• –enable, which enables data checksums. This rewrites all the relation file blocks, and finishes the operation by updating the control file. Note that this can be take time depending on the size of the instance, and that the tool has no parallel mode.
• –disables which disables data checksums by only updating the control file.

Hence, taking a cluster which has data checksums disabled, here is how to enable them. First the instance to switch needs to be cleanly shut down:

$ pg_controldata -D /my/data/folder/ | grep state
Database cluster state:               shut down

And then enabling data checksums is a matter of running this command, where the change gets reflected to the control file:

$ pg_checksums --enable -D /my/data/folder/
Checksum operation completed
Files scanned:  1144
Blocks scanned: 3487
pg_checksums: syncing data directory
pg_checksums: updating control file
Checksums enabled in cluster
$ pg_controldata -D /my/data/folder/ | grep checksum
Data page checksum version:           1

Repeating the same operation results in a failure (disabling data checksums where they are already disabled share the same fate):

$ pg_checksums --enable -D /my/data/folder/
pg_checksums: error: data checksums are already enabled in cluster

Then disabling checksums can be done like that:

$ pg_checksums --disable -D /my/data/folder/
pg_checksums: syncing data directory
pg_checksums: updating control file
Checksums disabled in cluster
$ pg_checksums --disable -D /my/data/folder/
pg_checksums: error: data checksums are already disabled in cluster
$ pg_controldata | grep checksum
Data page checksum version:           0

Finally, note that the tool is able to handle failures or interruptions in-between gracefully. For example, if the host in the process enabling data checksums is plugged off, then the data folder will remain in a state where they are disabled as the update of the control file happens last. Hence, the operation can be retried from scratch.

pg_verify_checksums is already a rather powerful tool when it comes to backup validation, but enabling checksums after an upgrade was still a barrier. Using Postgres 10, it is possible to use logical replication with a new instance initialized to have data checksums enabled when using initdb, still this takes time and resources as the initial data copy could take long. Note that if you have a cluster which relies on backup tools doing physical copy of relation blocks, pg_rewind being such an tool, it is possible to finish with a cluster which has checksums enabled still some pages could be broken if these pages come from a cluster having checksums disabled. Hence if switching checksums in a set of Postgres nodes, you should be careful to enable checksums consistently on all nodes at the same time.

Now, as data checksums are only compiled when a session flushes a page to disk or at shared buffer eviction, and because WAL do not need to compile checksums even if a full-page write is taken, enabling checksums with minimum downtime becomes much easier by relying on physical replication (WAL streaming). For example, assuming that no physical copy of relation blocks are done across multiple nodes, one could do the following with a set of two nodes, a primary and a standby:

• Both primary and standby have data checksums disabled, and the goal is to enable data checksums.
• First, stop cleanly the standby, and enable checksums on it with –enable.
• Start the standby, and make it catch up with the primary.
• Stop cleanly the primary.
• Promote the standby and do a failover to it.
• Enable checksums on the previous primary.
• Plug it back to the promoted standby, both instances have now checksums enabled.

On top of that, an option to output the progress of any operation run has been added:

commit: 280e5f14056bf34a0f52320f659fb93acfda0876
author: Michael Paquier <michael@paquier.xyz>
date: Tue, 2 Apr 2019 10:58:07 +0900
Add progress reporting to pg_checksums

This adds a new option to pg_checksums called -P/--progress, showing
every second some information about the computation state of an
operation for --check and --enable (--disable only updates the control
file and is quick).  This requires a pre-scan of the data folder so as
the total size of checksummable items can be calculated, and then it
gets compared to the amount processed.

Similarly to what is done for pg_rewind and pg_basebackup, the
information printed in the progress report consists of the current
amount of data computed and the total amount of data to compute.  This
could be extended later on.

Author: Michael Banck, Bernd Helmle
Reviewed-by: Fabien Coelho, Michael Paquier
Discussion: https://postgr.es/m/1535719851.1286.17.camel@credativ.de

Note that this is valid only for –check and –enable. Reports are every second, as follows (this is a fresh instance):

$ pg_checksums --enable --progress
27/27 MB (100%) computed

This requires an extra scan of the data folder so as it is possible to know the total size of all elements having checksums beforehand, costing some extra resources, but it can be useful for reporting when the operation takes a long time on a large cluster.

Native streaming replication in PostgreSQL works only between servers running the same major version. We discussed about logical replication in our previous blog post. In that post, we saw how logical replication could help us set up migration between two different PostgreSQL versions. However, logical replication works only for the currently supported versions of PostgreSQL, for example between PostgreSQL 9.4 and PostgreSQL 11. So what about the legacy versions that are older than 9.4? Slony-I could help us meet this replication requirement.

Replication between different PostgreSQL versions with Slony-I is useful for migration from legacy database installations to the latest available version. So what is Slony and how does it work?

This post is the fourth of our Upgrading or Migrating Your Legacy PostgreSQL to Newer PostgreSQL Versions series where we’ll be exploring different methods available to upgrade your PostgreSQL databases.

Slony

Slony is an application-level logical replication implementation for PostgreSQL. Rather, we could say that it is an external replication tool that requires a separate installation and configuration. Slony has been around for a long time. The latest version supports PostgreSQL versions from 8.4 and up to 11.

The main goal for replication is to ship changes from one database server to another. To better understand the architecture, you should know the terms such as Slon, Events and Slonik in Slony-I.

An aside: Slony means elephants in Russian, and elephants are indeed reputed to have a great memory. A slightly angry, but nevertheless pretty elephant, “Slonik”, looks at you from the PostgreSQL logo image.

Slon

Slon is a daemon that runs on each PostgreSQL node in Slony-I replication. These daemons are used for processing configuration and replication events for each PostgreSQL server. Each PostgreSQL server is called a “node”. All nodes together form a Slony “cluster”.

The “publisher node” is a source for replicated changes. While “subscriber” nodes receive and apply changes from the publisher node.

In order to setup replication, we should specify all replicated tables or “set”. Actual subscription works within a specific set. Changes to the tables being replicated are grouped together into SYNCs. These group of transactions are applied together to the subscriber nodes.

Events

Changes are transferred from the publisher in the form of “events”. When an event is processed by the slon daemon on a remote node, it generates a “confirmation”. Events are also used to notify nodes about configuration changes like adding/removing new nodes, new subscriptions or DDL changes.

Each event has a unique origin identifier, sequence number, transaction id for the snapshot on provider node for this event, multiple arguments, and timestamp with timezone.

Triggers written in PL/pgSQL register all changes in replicated tables. Unfortunately, there is no reliable way yet to handle changes to large objects (BLOBS), DDLs, or changes to users and roles.

slonik

Slonik means a little elephant. It is a command line utility with parser and interpreter and it accepts “slonik scripts” – a simple declarative scripting language. It is designed to overcome the limitations of procedural language. You use slonik commands to set up or modify slony replication and they can be embedded in shell scripts. It can accept commands from standard input or from files. The following example shows how a slonik script being fed to the slonik utility, and then embedded in shell scripts.

The script to create the initial configuration for the simple master-slave setup of our pgbench database looks like this:

#!/bin/sh
slonik <<_EOF_
 cluster name = percona_pg;
 node 1 admin conninfo = 'dbname=pg93 host=pg93_host user=percona_pg93_user';
 node 2 admin conninfo = 'dbname=pg11 host=pg11_host user=percona_pg11_user';
 #--
 # Creates a _$(clustername), this example, _percona_pg schema
 #--
 init cluster ( id=1, comment = 'Legacy PG Node');
 #--
 # Add a list of tables being replicated to a set.
 #--
create set (id=1, origin=1, comment='pgbench');
 set add table (set id=1, origin=1, id=1, fully qualified name = 'public.pgbench_accounts', comment='accounts');
 set add table (set id=1, origin=1, id=2, fully qualified name = 'public.pgbench_branches', comment='branches');
 set add table (set id=1, origin=1, id=3, fully qualified name = 'public.pgbench_tellers', comment='tellers');
 set add table (set id=1, origin=1, id=4, fully qualified name = 'public.pgbench_history', comment='history');
 #--
 # Create the second node (the slave) tell the 2 nodes how to connect to
 # each other and how they should listen for events.
 #--
 store node (id=2, comment = 'Target node', event node=1);
 store path (server = 1, client = 2, conninfo='dbname=pg93 host=pg93_host user=percona_pg93_user');
 store path (server = 2, client = 1, conninfo='dbname=pg11 host=pg11_host user=percona_pg11_user');
_EOF_

Why Slony is useful for migrations?

Despite the benefits of internal logical replication, this external solution is better for migrations between different versions that are older than PostgreSQL 9.4. The trigger-based approach depends on the database API – both older and newer versions should be compatible for PL/pgSQL and SQL syntax.

How to adapt your database for usage with Slony?

• Your tables have to have primary keys. Add a serial field to all tables without primary key
• OID blobs will not have their changes replicated. If you have columns with small-length values, you could convert these to BYTEA. For a really large objects like images, it’s better to store data externally e.g. use S3 in Amazon cloud. If it’s too hard to change your application, you could apply blob changes at the last stage of migration.
• ALTER TABLE and other DDL operations. Slony can’t detect table structure changes. Instead you should use an EXECUTE SCRIPT slonik command to apply a SQL file with DDL or SQL strings to the whole replication cluster.

Online migration from legacy PostgreSQL

1. Create replication user with superuser privileges. It’s possible to use fine-grained privileges, but they are significantly harder to setup.
2. Create a database on the destination, setup access by TCP/IP
3. Copy table definitions from master to slaves
4. Install Slony-I. On servers with an old OS distribution you might find it simpler to install Slony-I from the source code.
5. Define cluster, set of tables, and connection information to nodes as a list of slonik commands
6. Start the slon daemon on each postgresql server. Check standard output or log files for any potential communication errors.
7. Execute subscription slonik commands to start sync
8. Test your read-only queries with a newer version of postgres
9. Once all the data has been replicated and is in sync, stop your applications and repoint them to the new postgres server.
10. Use “uninstall node” on the newer version of PostgreSQL to remove all traces of Slony replication

Downgrade steps

To downgrade, follow the same procedure as upgrade. Slony allows you to replicate from and to any versions of PosgreSQL supported by your slony version. The minimum supported version is 8.4.

Summary

So far, we have seen a high level overview of how Slony can be helpful to perform an upgrade with the least possible downtime. Come and see more of this in action during our Webinar.  And don’t forget at Percona Live in Austin, May 28-30 2019, we’ll have two days of PostgreSQL content in a postgres dedicated track.

—
Image derived from photo by Chen Hu on Unsplash

The problem

If your database has a large number of small tables, you likely have a lot of wasted space. To demonstrate this, let’s create a table with a single record:

create table foo (str text);
insert into foo values ('a');
VACUUM foo;

Now let’s find out the path of the file containing our data, relative to the data directory:

select pg_relation_filepath('foo');
 pg_relation_filepath
----------------------
 base/16384/16405
(1 row)

(For more information on PostgreSQL files and directories, see Craig Ringer’s article on the topic.)

Notice what happens when we drop to the system command line within the data directory and list the files, adding a wild card to the path above:

$ ls base/16384/16405*

base/16384/16405
base/16384/16405_fsm
base/16384/16405_vm

The file named 16405 is the heap of our table, but what about the others? These are auxiliary files, called “relation forks” that contain additional information to help PostgreSQL access and maintain the table.

• 16405_fsm is the free space map. Its job is to know which pages in the table have space available for inserting records.
• 16405_vm is the visibility map. Its job is to know which heap pages may need VACUUM-ing or freezing, and also which heap pages must be visited during index-only scans.

There are other files associated with this table, but to find them we have to use a query:

select
pg_relation_filepath(c.reltoastrelid) as toast_table_path,
pg_relation_filepath(i.indexrelid) as toast_index_path
from pg_class c
left outer join pg_index i on c.reltoastrelid=i.indrelid
where c.relname = 'foo';

 toast_table_path | toast_index_path
------------------+------------------
 base/16384/16408 | base/16384/16410
(1 row)

This gives us the path of the toast table for our table, as well as that of the toast table’s index. If we insert any records with large enough strings, they will be compressed and stored here.

How much disk space do we need to store our one record?

\x
select
pg_relation_size(c.oid, 'main') as heap_size,
pg_relation_size(c.oid, 'fsm') as fsm_size,
pg_relation_size(c.oid, 'vm') as vm_size,
pg_relation_size(c.reltoastrelid) as toast_table_size,
pg_relation_size(i.indexrelid) as toast_index_size
from pg_class c
left outer join pg_index i on c.reltoastrelid=i.indrelid
where c.relname = 'foo';

-[ RECORD 1 ]----+------
heap_size        | 8192
fsm_size         | 24576
vm_size          | 8192
toast_table_size | 0
toast_index_size | 8192

To store a record with one string value, we need five files totalling 48kB of space! You may be thinking “So what? My tables are many gigabytes in size — why should I worry about a few kB?”. Well, consider the situation where an application uses schemas for multi-tenancy. Such a case could have thousands of schemas, each with hundreds of small tables, and the amount of space taken up by auxiliary files quickly becomes very high.

Is there anything we can do to reduce the amount of space consumed?

For starters, we can get rid of the toast table and its toast index by using a constraint on the length of the column:

create table foo_no_toast (str varchar(500));
insert into foo_no_toast values ('a');
VACUUM foo_no_toast;

select
pg_relation_size(c.oid, 'main') as heap_size,
pg_relation_size(c.oid, 'fsm') as fsm_size,
pg_relation_size(c.oid, 'vm') as vm_size,
pg_relation_size(c.reltoastrelid) as toast_table_size,
pg_relation_size(i.indexrelid) as toast_index_size
from pg_class c
left outer join pg_index i on c.reltoastrelid=i.indrelid
where c.relname = 'foo_no_toast';

-[ RECORD 1 ]----+------
heap_size        | 8192
fsm_size         | 24576
vm_size          | 8192
toast_table_size |
toast_index_size |

So now we have 3 files totalling 40 kB. This works because internally Postgres has a threshold tuple size beyond which it has to store the tuple in the toast table. In our new table here, Postgres knows this table can’t have any tuples exceeding this threshold, so it doesn’t bother creating the toast table.

This isn’t a big improvement, however. For one, we only saved a little bit of space. For another, it’s difficult to ensure the constraint on the column type is large enough to allow any string your application might want to insert, yet small enough to prevent the creation of the toast table. I obtained 500 in my example with a little experimentation, but the result will vary depending on the number of columns, the database encoding, and possibly on the major version of PostgreSQL.

Wouldn’t it be nice if we could get rid of the free space map? After all, it’s 3x bigger than the heap it’s keeping track of! It turns out there will soon be an easy way to do that.

Upgrade!

Starting in PostgreSQL version 12, tables only get a free space map if they exceed four pages, or 32kB with the default page size of 8kB.

Let’s try this on a recent git checkout:

create table foo_no_fsm (str text);
insert into foo_no_fsm values ('a');
VACUUM foo_no_fsm;

select
pg_relation_size(c.oid, 'main') as heap_size,
pg_relation_size(c.oid, 'fsm') as fsm_size,
pg_relation_size(c.oid, 'vm') as vm_size,
pg_relation_size(c.reltoastrelid) as toast_table_size,
pg_relation_size(i.indexrelid) as toast_index_size
from pg_class c
left outer join pg_index i on c.reltoastrelid=i.indrelid
where c.relname = 'foo_no_fsm';

-[ RECORD 1 ]----+-----
heap_size        | 8192
fsm_size         | 0
vm_size          | 8192
toast_table_size | 0
toast_index_size | 8192

At 24kB, it’s half the size of the equivalent table on PG11, without the disadvantages we had with constraints on the length of the datatype. Plus, the process of upgrading to version 12 will automatically discard unnecessary free space maps for you, so databases with a large number of small tables will see an immediate reduction in space used.

When you need to upgrade your PostgreSQL databases, there are a number of options available to you. In this post we’ll take a look at how you can upgrade PostgreSQL versions using pg_upgrade, a built-in tool that allows in-place upgrade of your software. Using pg_upgrade allows you, potentially, to minimize your downtime, an essential consideration for many organizations. It also allows you to perform a postgres upgrade with very minimal effort.

In our previous posts, we discussed various methods and tools that can help us perform a PostgreSQL upgrade – (1) pg_dumpall, (2) pg_dump and pg_restore with pg_dumpall, (3) logical replication and pglogical, and (4) slony. Methods 1 and 2 can involve additional downtime compared to the approaches taken in 3 and 4. Whilst performing an upgrade using logical replication or slony may be time consuming and require a lot of monitoring, it can be worth it if you can minimize downtime. If you have large databases that are busy with a lot of transactions, you may be better served using logical replication or slony.

This post is the fifth of our Upgrading or Migrating Your Legacy PostgreSQL to Newer PostgreSQL Versions series. These posts lead up to a live webinar, where we’ll be exploring different methods available to upgrade your PostgreSQL databases. If it’s beyond the live webinar date when you read this, you’ll find the recording at that same link.

pg_upgrade

pg_upgrade (formerly pg_migrator– until PostgreSQL 8.4) is a built-in tool that helps in upgrading a legacy PostgreSQL server to a newer version without the need of a dump and restore. The oldest version from when you can upgrade your PostgreSQL using pg_upgrade is 8.4.x. It is capable of performing faster upgrades by taking into consideration that system tables are the ones that undergo the most change between two major versions. The internal data storage format is less often affected.

In fact, in one of our tests we were able to perform an upgrade of a 2 TB database server from PostgreSQL 9.6.5 to 11.1 in less than 10 seconds. Now that is fast!

Overview of the process

To understand how it works, consider a PostgreSQL server running on 9.3.3 that needs to upgrade to PostgreSQL 11. You should install the latest binaries for your new PostgreSQL version on the server – let’s say PostgreSQL 11.2 – before you begin the upgrade process.

Preparation and consistency checks

Once you have installed the new PostgreSQL version, initialize a new data directory using the new binaries and start it on another port i.e. a different port to the one used by PostgreSQL 9.3.3, in our example. Use pg_upgrade to perform consistency checks between the two servers – PG 9.3.3 and PG 11.2 – running on two different ports. If you get any errors, such as a missing extension, you need to to fix these before you proceeding to the upgrade. Once the consistency check has been passed, you can proceed.

Here is how the log looks if you should get an error while performing consistency checks.

$ /usr/pgsql-11/bin/pg_upgrade -b /usr/pgsql-9.3/bin -B /usr/pgsql-11/bin -d /var/lib/pgsql/9.3/data -D /var/lib/pgsql/11/data_new -c
Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for invalid "unknown" user columns                 ok
Checking for hash indexes                                   ok
Checking for roles starting with "pg_"                      ok
Checking for incompatible "line" data type                  ok
Checking for presence of required libraries                 fatal
Your installation references loadable libraries that are missing from the
new installation.  You can add these libraries to the new installation,
or remove the functions using them from the old installation.  A list of
problem libraries is in the file:
    loadable_libraries.txt
Failure, exiting
$ cat loadable_libraries.txt
could not load library "$libdir/pg_repack": ERROR:  could not access file "$libdir/pg_repack": No such file or directory

To proceed beyond the error, in this example you’d need to install this missing extension pg_repack for the new PostgreSQL version, and rerun the check to make sure that you receive no errors and all the checks are passed.

Carrying out the upgrade

Once passed, you can proceed in one of two ways. One option is to let pg_upgrade copy the datafiles of the old data directory to the new data directory initialized by the new PostgreSQL version. The second option is to let pg_upgrade use hard links instead of copying data files. Copying a database of several terabytes may be time consuming. Using the hard links method makes the process really quick as it does not involve copying files.

To use hard links with pg_upgrade, you pass an additional argument -k as you can see in the following command.

$ /usr/pgsql-11/bin/pg_upgrade -b /usr/pgsql-9.3/bin -B /usr/pgsql-11/bin -d /var/lib/pgsql/9.3/data -D /var/lib/pgsql/11/data_new -k

In the Unix file system, a file or a directory is a link to an inode (index node) that stores metadata (disk block location, attributes, etc) of the data stored in them. Each inode is identified by an integer or an inode number. When you use pg_upgrade with hard links, it internally creates another file/directory in the new data directory that links to the same inode as it was in the older data directory for that file/directory. So, it skips the physical copy of the objects, but creates each object and links them to the same inode.

This reduces the disk IO and avoids the need for additional space in the server. An important point to note is that this option works only when you are upgrading your PostgreSQL on the same file system. This means, for example, if you want to upgrade to a new or a faster disk during the database upgrade, the hard link option does not work. In that case, you would need to use the file copy method.

So far, we have seen a high level overview of how pg_upgrade with hard links help you to perform an upgrade with lowest possible downtime. Come see more in action during our Webinar.  And don’t forget at Percona Live in Austin, May 28-30 2019, we’ll have two days of PostgreSQL content in a postgres dedicated track.

—
Elephant image based on photo from Pexels

Have you ever heard about cursors in PostgreSQL or in SQL in general? If not you should definitely read this article in depth and learn how to reduce memory consumption in PostgreSQL easily. Cursors have been around for many years and are in my judgement one of the most underappreciated feature of all times. Therefore it makes sense to take a closer look at cursors and see what they can be used for.

The purpose of a cursor in PostgreSQL

Consider the following example:

test=# CREATE TABLE t_large (id int);
CREATE TABLE
test=# INSERT INTO t_large 
	SELECT * FROM generate_series(1, 10000000);
INSERT 0 10000000

I have created a table containing 10 million rows so that we can play with the data. Let us run a simple query now:

test=# SELECT * FROM t_large;
    id    
----------
        1
        2
        3
…

The first thing you will notice is that the query does not return immediately. There is a reason for that: PostgreSQL will send the data to the client and the client will return as soon as ALL the data has been received. If you happen to select a couple thousand rows, life is good and everything will be just fine. However, what happens if you do a “SELECT * …” on a table containing 10 billion rows? Usually the client will die with an “out of memory” error and your applications will simply die. There is no way to keep such a large table in memory. Throwing ever more RAM at the problem is not feasible either (and pretty stupid too).

Using DECLARE CURSOR and FETCH

DECLARE CURSOR and FETCH can come to the rescue. What is the core idea? We can fetch data in small chunks and only prepare the data at the time it is fetched – not earlier. Here is how it works:

test=# BEGIN;
BEGIN
test=# DECLARE mycur CURSOR FOR 
	SELECT * FROM t_large WHERE id > 0;
DECLARE CURSOR
test=# FETCH NEXT FROM mycur;
 id 
----
  1
(1 row)

test=# FETCH 4 FROM mycur;
 id 
----
  2
  3
  4
  5
(4 rows)

test=# COMMIT;
COMMIT

The first important thing to notice is that a cursor can only be declared inside a transaction. However, there is more: The second important this is that DECLARE CURSOR itself is lightning fast. It does not calculate the data yet but only prepares the query so that your data can be created when you call FETCH. To gather all the data from the server you can simply run FETCH until the resultset is empty. At the you can simply commit the transaction.

Note that a cursor is closed on commit as you can see in the next listing:

test=# FETCH 4 FROM mycur;
ERROR:  cursor "mycur" does not exist
test

The FETCH command is ways more powerful than most people think. It allows you to navigate in your resultset and fetch rows as desired:

test=# \h FETCH
Command:     FETCH
Description: retrieve rows from a query using a cursor
Syntax:
FETCH [ direction [ FROM | IN ] ] cursor_name

where direction can be empty or one of:

    NEXT
    PRIOR
    FIRST
    LAST
    ABSOLUTE count
    RELATIVE count
    count
    ALL
    FORWARD
    FORWARD count
    FORWARD ALL
    BACKWARD
    BACKWARD count
    BACKWARD ALL

Cursors are an easy and efficient way to retrieve data from the server. However, you have to keep one thing in mind: Latency. Asking the network for one row at a time will add considerable network overhead (latency). It therefore makes sense to fetch data in reasonably large chunks. I found it useful to fetch 10.000 rows at a time. 10.000 can still reside in memory easily while still ensuring reasonably low networking overhead. Of course I highly encourage you to do your own experience to see, what is best in your specific cases.

Cursors and the PostgreSQL optimizer

Cursors are treated by the optimizer in a special way. If you are running a “normal” statement PostgreSQL will optimize for total runtime. It will assume that you really want all the data and optimize accordingly. However, in case of a cursor it assumes that only a fraction of the data will actually be consumed by the client. The following example shows, how this works:

test=# CREATE TABLE t_random AS 
	SELECT random() AS r 
	FROM generate_series(1, 1000000);
SELECT 1000000
test=# CREATE INDEX idx_random ON t_random (r);
CREATE INDEX
test=# ANALYZE ;
ANALYZE

I have created a table, which contains 1 million random rows. Finally I have created a simple index. To make sure that the example works I have told the optimizer that indexes are super expensive (random_page_cost):

test=# SET random_page_cost TO 100;
SET

Let us take a look at an example now: If the query is executed as cursor you will notice that PostgreSQL goes for an index scan to speed up the creation of the first 10% of the data. If the entire resultset is fetched, PostgreSQL will go for a sequential scan and sort the data because the index scan is considered to be too expensive:

test=# BEGIN;
BEGIN
test=# explain DECLARE cur CURSOR FOR SELECT * FROM t_random ORDER BY r;
                                        QUERY PLAN                                         
-------------------------------------------------------------------------------------------
 Index Only Scan using idx_random on t_random  (cost=0.42..732000.04 rows=1000000 width=8)
(1 row)

test=# explain SELECT * FROM t_random ORDER BY r;
                                     QUERY PLAN                                      
-------------------------------------------------------------------------------------
 Gather Merge  (cost=132326.50..229555.59 rows=833334 width=8)
   Workers Planned: 2
   ->  Sort  (cost=131326.48..132368.15 rows=416667 width=8)
         Sort Key: r
         ->  Parallel Seq Scan on t_random  (cost=0.00..8591.67 rows=416667 width=8)
(5 rows)
test=# COMMIT;
COMMIT

The main question arising now is: How does the optimizer know that the first 10% should be fast and that we are not looking for the entire resultset? A runtime setting is going to control this kind of behavior: cursor_tuple_fraction will configure this kind of behavior:

test=# SHOW cursor_tuple_fraction;
 cursor_tuple_fraction 
-----------------------
 0.1
(1 row)

The default value is 0.1, which means that PostgreSQL optimizes for the first 10%. The parameter can be changed easily in postgresql.conf just for your current session.

Using cursors across transactions

So far you have seen that a cursor can only be used inside a transaction. COMMIT or ROLLBACK will destroy the cursor. However, in some (usually rare) cases it can be necessary to have cursors, which actually are able to survive a transaction. Fortunately PostgreSQL has a solution to the problem: WITH HOLD cursors.

Here is how it works:

test=# BEGIN;
BEGIN
test=# DECLARE cur CURSOR WITH HOLD FOR 
	SELECT * FROM t_random ORDER BY r;
DECLARE CURSOR
test=# \timing
Timing is on.
test=# COMMIT;
COMMIT
Time: 651.211 ms

As you can see the WITH HOLD cursor has been declared just like a normal cursor. The interesting part is the COMMIT: To make sure that the data can survive the transaction PostgreSQL has to materialize the result. Therefore the COMMIT takes quite some time. However, the FETCH can now happen after the COMMIT:

test=# FETCH 4 FROM cur;
          r           
----------------------
 2.76602804660797e-07
 6.17466866970062e-07
 3.60095873475075e-06
 4.77954745292664e-06
(4 rows)

If you are making use of WITH HOLD cursors you have to keep in mind that the cursor has to be closed as well. Otherwise your connection will keep accumulating new cursors and store the result.

test=# \h CLOSE
Command:     CLOSE
Description: close a cursor
Syntax:
CLOSE { name | ALL }

Do you want to learn more about PostgreSQL and the optimizer in general consider? Check out one of our older posts right now.

The post DECLARE CURSOR in PostgreSQL or how to reduce memory consumption appeared first on Cybertec.

PostgreSQL has a very unique way to review the code submitted by developers. Most open source software uses Github pull requests to accommodate users’ code. PostgreSQL has a Github page but doesn’t  manage pull requests using Github. Many years ago, PostgreSQL introduced CommitFest to manage its patches, where postgres opens for a three to four month “festival” to accept patches. The CommitFest is set up to keep track of patches’ statuses. In practice, CommitFest is mainly an admin page used to manage patches. Volunteers and community committers review the submitted code, after which committers are able commit those patches that have been approved. There have been almost 22 CommitFest events since 2014.

The process

When a patch is submitted, a volunteer can choose to review it. A review can be done by more than one person. At any point, a patch will be in one of the following statuses:

• Committed
• Moved to next CF (CommitFest)
• Needs Review
• Ready for Committer
• Rejected
• Returned with Feedback
• Waiting on Author
• Withdrawn

Needs review indicates that the patch is waiting for a reviewer to analyze it. Once a reviewer has confirmed that the patch in question is good to be committed, a community committer will pick up the patch, review it and commit it if all is well, otherwise, they reject it or return it with feedback.

Patch categories

Patches are categorized in various ways including:

• Bug Fixes
• Clients
• Code Comments
• Documentation
• Miscellaneous
• Monitoring & Control
• Performance
• Procedural Languages
• Refactoring
• Replication & Recovery
• SQL Commands
• Server Features
• System Administration

A complete history of a patch, along with a discussion on email, is maintained with each revision which makes it very simple to track the complete process from submission of a patch to its eventual conclusion.

Here are some graphs that show various stats for the past CommitFests.

Now it’s time to end the 22nd CommitFest, the first such festival for 2019. This event saw 207 patches, almost 52% of which were committed to the master branch, which is a good ratio. This master branch becomes the development branch for the next major PostgreSQL release.

Summary of outcomes from CommitFest 22:

• Patches committed: 100.
• Moved to next CommitFest: 84.
• Withdrawn: 5.
• Rejected: 3.
• Returned with Feedback: 15

As a reviewer, I worked on some of these patches this time around, and I’ll devote some more time in the next CommitFest to more reviews. If you like the idea, then I’d encourage you to come and participate in the review process – we can always use more people. Even if you are not a developer, you can help to test a patch and verify the fix on a different environment that the developer may not have.

You can read more about how a CommitFest is run at this PostgreSQL wiki page.

A few months ago I worked to improve the great pgenv tool by theory. Today, I try to spread the word in the hope this tool can grow a little more!

An article about pgenv

tl;dr

I proposed a talk about pgenv, a Bash tool to manage several PostgreSQL instances on the same local machine. My talk has been rejected, and I hate to waste what I have already prepared, so I decided to transform my talk in an article, that has been quickly accepted on Haikin9 Devops Issue!

I should have written about this a couple of months ago, but I did not had time to.
My hope is that pgenv gets more and more users, so that it can grow and become someday a widely used tool. Quite frankly, I don’t see this happening while being in Bash, for both portability and flexibility, and I suspect Perl is much more the language for a more flexible implementation. However, who knows? Gathering users is also a way to gather contributors and bring therefore new ideas to this small but very useful project.

In the meantime, if you have time and will, try testing the build from git patch, that allows you to build and manage a development version of our beloved database.

As of 15 April 2019, there is only one repository RPM per distro, and it includes repository information for all available PostgreSQL releases.

This change, announced by Devrim on the pgsql-pkg-yum mailing list, has some impacts.

Announce

The announce from Devrim may be found here.

• Instead of having separate repo RPMs per PostgreSQL major version, we now have one single repo RPM that supports all supported PostgreSQL releases. The new packages obsolete the current ones.

• The repo RPM version has been bumped to 42. Hopefully that will be the end of the “The repo RPM is 10-4, how can I find 10-7 repo rpm, so that I can install PostgreSQL 10.7?” type questions.

• The “latest” suffix has been added to all repo RPMs.

Installation

Let’s see some impacts of those changes on CentOS 7.

As usual, go to https://www.postgresql.org/download/linux/redhat/ and chose the version (11), the platform (CentOS 7) and the architecture (x86_64) you want to install.

Today, you still get the link to the pgdg-centos11-11-2 rpm.

Let’s install it:

# yum install https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-7-x86_64/pgdg-centos11-11-2.noarch.rpm
Loaded plugins: fastestmirror
pgdg-centos11-11-2.noarch.rpm
Examining /var/tmp/yum-root-5eSWGp/pgdg-centos11-11-2.noarch.rpm: pgdg-redhat-repo-42.0-4.noarch
Marking /var/tmp/yum-root-5eSWGp/pgdg-centos11-11-2.noarch.rpm to be installed

Resolving Dependencies
--> Running transaction check
---> Package pgdg-redhat-repo.noarch 0:42.0-4 will be installed
--> Finished Dependency Resolution

Dependencies Resolved
========================================================================================================
 Package                   Arch            Version            Repository                           Size
========================================================================================================
Installing:
 pgdg-redhat-repo          noarch          42.0-4             /pgdg-centos11-11-2.noarch          6.8 k

Transaction Summary
========================================================================================================
Install  1 Package

Total size: 6.8 k
Installed size: 6.8 k

In fact, the new pgdg-redhat-repo rpm will be installed…

The yum .repo file will now contains the urls of all supported PostgreSQL releases:

# cat /etc/yum.repos.d/pgdg-redhat-all.repo |grep "\["[pgdg12]
[pgdg11]
[pgdg10]
[pgdg96]
[pgdg95]
[pgdg94]
...

The repositories for version 9.4 to 11 are enabled by default.

The PostgreSQL packages are then easily reachable and you might even install two different releases at once:

# yum install postgresql11-server postgresql10-server
...

Dependencies Resolved
========================================================================================================
 Package                        Arch              Version                       Repository         Size
========================================================================================================
Installing:
 postgresql10-server            x86_64            10.7-2PGDG.rhel7              pgdg10            4.5 M
 postgresql11-server            x86_64            11.2-2PGDG.rhel7              pgdg11            4.7 M
Installing for dependencies:
 libicu                         x86_64            50.1.2-17.el7                 base              6.9 M
 postgresql10                   x86_64            10.7-2PGDG.rhel7              pgdg10            1.6 M
 postgresql10-libs              x86_64            10.7-2PGDG.rhel7              pgdg10            355 k
 postgresql11                   x86_64            11.2-2PGDG.rhel7              pgdg11            1.6 M
 postgresql11-libs              x86_64            11.2-2PGDG.rhel7              pgdg11            360 k

Transaction Summary
========================================================================================================
Install  2 Packages (+5 Dependent packages)

Total download size: 20 M
Installed size: 80 M

To simplify blog posts or automated scripts, you might easily use the simlink to the actual latest repo RPM:

https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm

Updates

This change also has an impact on existing installations.

On, for example, an existing v11 installation:

# cat /etc/yum.repos.d/pgdg-11-centos.repo |grep "\["[pgdg11]
...

The update operation will replace the old pgdg-centos11 rpm by the new one and create the new .repo file:

# yum update pgdg-centos11
...
Resolving Dependencies
--> Running transaction check
---> Package pgdg-centos11.noarch 0:11-2 will be obsoleted
---> Package pgdg-redhat-repo.noarch 0:42.0-4 will be obsoleting
--> Finished Dependency Resolution

Dependencies Resolved
========================================================================================================
 Package                        Arch                 Version                 Repository            Size
========================================================================================================
Installing:
 pgdg-redhat-repo               noarch               42.0-4                  pgdg11               5.6 k
     replacing  pgdg-centos11.noarch 11-2

Transaction Summary
========================================================================================================
Install  1 Package

Total download size: 5.6 k

EOL’d releases

While the yum repositories for EOL’d releases still exist, the repo rpms usually found on https://yum.postgresql.org/repopackages.php#pg93 aren’t available anymore: 404 - Not Found.

In case you need to use the old repositories, you can add it manually:

# cat /etc/yum.repos.d/pgdg-93-centos.repo [pgdg93]
name=PostgreSQL 9.3 $releasever - $basearchbaseurl=https://download.postgresql.org/pub/repos/yum/9.3/redhat/rhel-$releasever-$basearchenabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-93

[pgdg93-source]
name=PostgreSQL 9.3 $releasever - $basearch - Source
failovermethod=priority
baseurl=https://download.postgresql.org/pub/repos/yum/srpms/9.3/redhat/rhel-$releasever-$basearchenabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-93

Of course, since those releases are not supported anymore, you shouldn’t have to use it anymore…

Conclusion

While this change is still new, it will make our lives easier in the future…

In this blog post, we will explore 

pg_dump

pg_restore

pg_dump

pg_restore

This post is the second of our Upgrading or Migrating Your Legacy PostgreSQL to Newer PostgreSQL Versions series where we’ll be exploring different methods available to upgrade your PostgreSQL databases.

About pg_dump

pg_dump

pg_dump

pg_dumpall

pg_dumpall

pg_dump formats

pg_dump

pg_dump

(-Fc)

pg_restore

If the dump is taken using a plain-text format, pg_dump generates a script file of multiple SQL commands. It can be restored using psql.

A custom format dump, however, is compressed and is not human-readable.

A dump taken in plain text format may be slightly larger in size when compared to a custom format dump.

At times, you may wish to perform schema changes in your target PostgreSQL database before restore, for example, table partitioning. Or you may wish to restore only a selected list of objects from a dump file.

In such cases, you cannot restore a selected list of tables from a plain format dump of a database. If you take the database dump in custom format,  you can use pg_restore, which will help you choose a specific set of tables for restoration.

Steps involved in upgrade

The most important point to remember is that both dump and restore should be performed using the latest binaries. For example, if we need to migrate from version 9.3 to version 11, we should be using the pg_dump binary of PostgreSQL 11 to connect to 9.3 .

When a server is equipped with two different versions of binaries, it is good practice to specify the full path of the pg_dump from the latest version as follows :

/usr/lib/postgresql/11/bin/pg_dump <connection_info_of_source_system> <options>

Getting the global dumps

In PostgreSQL, users/roles are global to the database cluster, and the same user can have privileges on objects in different databases. These are called “Globals” because they are applicable for all the databases within the instance. Creation of globals in the target system at the earliest opportunity is very important, because rest of the DDLs may contain GRANTs to these users/roles. It is good practice to dump the globals into a file, and to examine the file, before importing into destination system. This can be achieved using the following command :

/usr/lib/postgresql/11/bin/pg_dumpall -g -p 5432 > /tmp/globals_only.sql

Since this produces a plain SQL dump file, it can be fed to

psql

/usr/lib/postgresql/11/bin/pg_dumpall -g <source_connection_info> | psql -p <destination_connection_info>

The above command would work for an upgrade in a local server. You can add an additional argument

-h

hostname

<destination_connection_info>

Schema Only Dumps

The next stage of the migration involves the creation of schema objects. At this point, you might want to move different database objects to different tablespaces, and partition a few of the tables. If such schema modifications are part of the plan, then we should extract the schema definition to a plain text file. Here’s an example command that can be used to achieve this :

/usr/lib/postgresql/11/bin/pg_dump -s -d databasename -p 5432 > /tmp/schema_only.sql

In general, the majority of the database objects won’t need any modifications. In such cases, it is good practice to dump the schema objects as such into the destination database using a

PIPE

/usr/lib/postgresql/11/bin/pg_dump -s -d databasename <source_connection> | psql -d database <destination_connection>

Once all the schema objects are created, we should be able to drop only those objects which need modification. We can then recreate them with their modified definition.

Copying data

This is the stage when the majority of the data transfers between the database servers. If there is good bandwidth between source and destination, we should look to achieve maximum parallelism at this stage. In many situations, we could analyze the foreign key dependency hierarchy and import data in parallel batches for a group of tables. Data-only copying is possible using

-a

--data-only

pg_dump

Copying the data of individual tables

You might have to incorporate schema changes as part of an upgrade. In this case, you can copy the data of a few tables individually. We provide an example here:

/usr/lib/postgresql/11/bin/pg_dump <sourcedb_connection_info> -d <database> -a -t schema.tablename | psql <destinationdb_connection_info> <databasename>

There could be special situations where you need to append only a partial selection of the data. This happens especially on time-series data. In such cases, you can use copy commands with a WHERE clause cto extract and import specific data. You can see this in the following example :

/usr/lib/postgresql/11/bin/psql <sourcedb_connection_info> -c "COPY (select * from <table> where <filter condition>)” > /tmp/selected_table_data.sql

Summary

pg_dump/pg_restore may be useful if you need to perform a faster upgrade of PostgreSQL server with a modified schema and bloat-free relations. To see more about this method in action, please subscribe to our webinar here.

—
image based on photos by Skitterphoto and Magda Ehlers from Pexels

The full text search is based on transforming the initial text into a tsvector. For example:

test=> select to_tsvector('english', 'This text is being processed.');
     to_tsvector      
----------------------
 'process':5 'text':2

This result is a sorted list of lexems, with their relative positions in the initial text, obtained by this process:

Raw text => Parser => Dictionaries (configurable) => tsvector

When there is enough data, we tend to index these vectors with a GIN or GIST index to speed up text search queries.

In SQL we can inspect the intermediate results of this process with the ts_debug function:

test=> select * from ts_debug('english', 'This text is being processed.');
   alias   |   description   |   token   |  dictionaries  |  dictionary  |  lexemes  
-----------+-----------------+-----------+----------------+--------------+-----------
 asciiword | Word, all ASCII | This      | {english_stem} | english_stem | {}
 blank     | Space symbols   |           | {}             |              | 
 asciiword | Word, all ASCII | text      | {english_stem} | english_stem | {text}
 blank     | Space symbols   |           | {}             |              | 
 asciiword | Word, all ASCII | is        | {english_stem} | english_stem | {}
 blank     | Space symbols   |           | {}             |              | 
 asciiword | Word, all ASCII | being     | {english_stem} | english_stem | {}
 blank     | Space symbols   |           | {}             |              | 
 asciiword | Word, all ASCII | processed | {english_stem} | english_stem | {process}
 blank     | Space symbols   | .         | {}             |              | 

The parser breaks down the text into tokens (token column), each token being associated with a type (alias and description columns). Then depending on their types, these tokens are submitted as input to dictionaries mapped to these types, which may produce one lexem, or several, or zero to eliminate the term from the output vector.

In the above example, spaces and punctuation are eliminated because there are not mapped to any dictionary; common terms (“this”, “is”, “being”) are eliminated as stop words by the english_stem dictionary; “text” is kept untouched, and “processed” is stemmed as “process” by english_stem.

What about undesirables pieces of text?

Sometimes the initial text is not “clean” in the sense that it contains noise that we’d rather not index. For instance when indexing mail messages, badly formatted messages may have base64 contents that slip into text parts. When these contents correspond to attached files, they can be pretty big. Looking at how this kind of data gets transformed into lexems, here’s what we can see:

=# \x
=# select * from  ts_debug('english', 'Q29uc2lzdGVuY3kgYWxzbyBtZWFucyB0aGF0IHdoZW4gQWxpY2UgYW5kIEJvYiBhcmUgcnVubmlu');
-[ RECORD 1 ]+-------------------------------------------------------------------------------
alias        | numword
description  | Word, letters and digits
token        | Q29uc2lzdGVuY3kgYWxzbyBtZWFucyB0aGF0IHdoZW4gQWxpY2UgYW5kIEJvYiBhcmUgcnVubmlu
dictionaries | {simple}
dictionary   | simple
lexemes      | {q29uc2lzdgvuy3kgywxzbybtzwfucyb0agf0ihdozw4gqwxpy2ugyw5kiejvyibhcmugcnvubmlu}

So this text is parsed as a single numword token, from which a single long lexem is produced, since numword tokens are associated with the simple dictionary that just downcases the word. So that brings up a question: how to avoid that kind of term in the vector?

One idea is to consider that very long tokens are uninteresting for search, and as such they can just be eliminated. Even though there are really long words in some languages, such as the german Rindfleischetikettierungsüberwachungsaufgabenübertragungsgesetz with its 63 characters(!), we can imagine setting a length limit over which words are ignored, just like stop words are ignored.

Filtering by length

It’s not difficult to create a dictionary to filter out long words. A text search dictionary takes the form of two functions in C, along with a few SQL declarative statements to create it and assign it to a text configuration.

In PostgreSQL source code, there are several examples of dictionaries than can be used as models:

• dict_simple a built-in dictionary, part of core.
• dict_int, a contrib module to filter out number longer than a given number of digits, so very close to what we want in this post.
• dict_xsyn, a contrib module to add lexems to the output that are not in the original text but are synonyms of words in the text.
• unaccent, a contrib module providing first a callable SQL function to remove accents, but also exposing it as a filtering dictionary, which means that lexems produced are passed to the next dictionary in the chain.

A new dictionary project can be started by pretty much copy-pasting one of these examples, since much of the code is declarative stuff that you don’t want to write from scratch.

There are two C functions to produce:

• An INIT function that receives the configuration parameters of the dictionary. We can use this function to process our maximum length a parameter, instead of hard-coding it in the source.

• A LEXIZE function that takes a token’s text as input and needs to produce zero, one or several lexems corresponding to that piece of text. The function also indicates if these lexems are to be passed to the rest of the chain of dictionaries. In our case we want to eliminate the token if it’s longer than our limit, or pass it unchanged.

Let’s call this dictionary dictmaxlen and length its parameter. Following the model of contrib modules, we can package its code and declarations in a Postgres extension.

The declarations actually create a dictionary template rather than a dictionary, if we want to use the correct terminology. A dictionary is instantiated from a template with CREATE TEXT SEARCH DICTIONARY (TEMPLATE = ...) with the values for the parameters.

Here are the SQL declarations for the functions and the template:

CREATEFUNCTIONdictmaxlen_init(internal)RETURNSinternalAS'MODULE_PATHNAME'LANGUAGECSTRICT;CREATEFUNCTIONdictmaxlen_lexize(internal,internal,internal,internal)RETURNSinternalAS'MODULE_PATHNAME'LANGUAGECSTRICT;CREATETEXTSEARCHTEMPLATEdictmaxlen_template(LEXIZE=dictmaxlen_lexize,INIT=dictmaxlen_init);

The only specific bit here is the name dictmaxlen, otherwise any other dictionary would have the same declarations.

C functions

Dictionary initialization (called when instantiating)

Datumdictmaxlen_init(PG_FUNCTION_ARGS){List*options=(List*)PG_GETARG_POINTER(0);DictMaxLen*d;ListCell*l;d=(DictMaxLen*)palloc0(sizeof(DictMaxLen));d->maxlen=50;/* 50 chars by defaut */foreach(l,options){DefElem*defel=(DefElem*)lfirst(l);if(strcmp(defel->defname,"length")==0)d->maxlen=atoi(defGetString(defel));else{ereport(ERROR,(errcode(ERRCODE_INVALID_PARAMETER_VALUE),errmsg("unrecognized dictionary parameter: \"%s\"",defel->defname)));}}PG_RETURN_POINTER(d);}

Generation of lexems

Datumdictmaxlen_lexize(PG_FUNCTION_ARGS){DictMaxLen*d=(DictMaxLen*)PG_GETARG_POINTER(0);char*token=(char*)PG_GETARG_POINTER(1);intbyte_length=PG_GETARG_INT32(2);if(pg_mbstrlen_with_len(token,byte_length)>d->maxlen){/* 
     * If the word is longer than our limit, returns an array without
     * any lexem.
     */TSLexeme*res=palloc0(sizeof(TSLexeme));PG_RETURN_POINTER(res);}else{/* If the word is short, pass it unmodified */PG_RETURN_POINTER(NULL);}}

Encapsulating the code into an extension

As most for contrib modules, it’s convenient to package this code as an extension to distribute and deploy it easily.

Creating an extension is relatively easy with PGXS, that is part of the postgres toolbox (for Linux distributions, it’s often provided in a development package, such as postgresql-server-dev-11 for Debian).

An extension needs a control file. The following stanza will do the job (filename: dict_maxlen.control):

# dict_maxlen extension
comment = 'text search template for a dictionary filtering out long words'
default_version = '1.0'
module_pathname = '$libdir/dict_maxlen'
relocatable = true

Thanks to PGXS, we can use a simplied Makefile that will transparently include the declarations with the paths and libraries involved in the build. Here is a ready-to-use Makefile than can build our simple extension with make && make install:

EXTENSION = dict_maxlen
EXTVERSION = 1.0
PG_CONFIG = pg_config

MODULE_big = dict_maxlen
OBJS = dict_maxlen.o

DATA = $(wildcard *.sql)

PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

Usage

Once the extension is compiled is installed, we can instantiate it in a database and create the dictionary:

CREATEEXTENSIONdict_maxlen;CREATETEXTSEARCHDICTIONARYdictmaxlen(TEMPLATE=dictmaxlen_template,LENGTH=40-- for example);

Now this dictionary needs to be mapped to tokens (the output of the parser), with ALTER TEXT SEARCH CONFIGURATION ... ALTER MAPPING. In the example above we saw that the kind of token produced with the base64 content was numword, but we may also want to map our dictionary to tokens containing only letters: word and asciiword, or to any other of the 23 kinds of tokens that the parser can generate as of Postgres 11.

With psql this mapping can be visualized with \dF+. For english, the defaults are:

postgres=> \dF+ english
Text search configuration "pg_catalog.english"
Parser: "pg_catalog.default"
      Token      | Dictionaries 
-----------------+--------------
 asciihword      | english_stem
 asciiword       | english_stem
 email           | simple
 file            | simple
 float           | simple
 host            | simple
 hword           | english_stem
 hword_asciipart | english_stem
 hword_numpart   | simple
 hword_part      | english_stem
 int             | simple
 numhword        | simple
 numword         | simple
 sfloat          | simple
 uint            | simple
 url             | simple
 url_path        | simple
 version         | simple
 word            | english_stem

To avoid messing up with the built-in english configuration, let’s derive a new specific text search configuration with its own mappings:

CREATETEXTSEARCHCONFIGURATIONmytsconf(COPY=pg_catalog.english);ALTERTEXTSEARCHCONFIGURATIONmytsconfALTERMAPPINGFORasciiword,wordWITHdictmaxlen,english_stem;ALTERTEXTSEARCHCONFIGURATIONmytsconfALTERMAPPINGFORnumwordWITHdictmaxlen,simple;

Now let’s check the new configuration with psql:

=# \dF+ mytsconf
Text search configuration "public.mytsconf"
Parser: "pg_catalog.default"
      Token      |      Dictionaries      
-----------------+------------------------
 asciihword      | english_stem
 asciiword       | dictmaxlen,english_stem
 email           | simple
 file            | simple
 float           | simple
 host            | simple
 hword           | english_stem
 hword_asciipart | english_stem
 hword_numpart   | simple
 hword_part      | english_stem
 int             | simple
 numhword        | simple
 numword         | dictmaxlen,simple
 sfloat          | simple
 uint            | simple
 url             | simple
 url_path        | simple
 version         | simple
 word            | dictmaxlen,english_stem

And check what happens now to a numword over 40 characters:

=#selectto_tsvector('mytsconf','A long word: Q29uc2lzdGVuY3kgYWxzbyBtZWFucyB0aGF0IHdoZW4');to_tsvector-------------------'long':2'word':3

Et voilà! The undesirable token has been left out, as expected.

We may use this mytsconf configuration by passing it as an explicit first argument to text search functions, but it could also be set as a default.

For the duration of the session:

SET default_text_search_config TO 'mytsconf';

For the database (durably):

ALTER DATABASE dbname SET default_text_search_config TO 'mytsconf';

The source code for this example is available at github.

Over the past months, a great number of PostgreSQL User Groups and Meetups showed up all over Europe. It’s good to see that interest in PostgreSQL is growing!

Some of the user groups approached the PostgreSQL Europe board, and asked for support. Mostly for swag, but also for sending speakers, or other kind of support. We are happy to help!

In order to handle all of these requests, the PostgreSQL Europe board created a set of guidelines for user group meetings. The current version can be found on the PostgreSQL Europe website, under “Community”, and then “Community User Group Recognition Guidelines”. User groups which approach the PostgreSQL Europe board for support are expected to comply by these guidelines. Every user group is self-certified under these guidelines. If you have reason to believe that a self-certified status for a user group is not correct, please contact the PostgreSQL Europe board under “Contact”.