Continuing our series of PostgreSQL Data Types today we’re going to introduce the PostgreSQL ENUM type.

This data type has been added to PostgreSQL in order to make it easier to support migrations from MySQL. Proper relational design would use a reference table and a foreign key instead.

A bit of background

Interaction with indexes

Parallelism

Replication

SKIP LOCKED

Recently I did some PostgreSQL consulting in the Berlin area (Germany) when I stumbled over an interesting request: How can data be shared across function calls in PostgreSQL? I recalled some one of the other features of PostgreSQL (15+ years old or so) to solve the issue. Here is how it works.

Stored procedures in PostgreSQL

As many of you might know PostgreSQL allows you to write stored procedures in many different languages. Two of the more popular ones are Perl and Python, which have been around for quite some time. The cool thing is: Both languages offer a way to share variables across function calls. In Perl you can make use of the $_SHARED variable, which is always there.

Here is an example:

CREATE OR REPLACE FUNCTION set_var(int)
RETURNS int AS $$
   $_SHARED{'some_name'} = $_[0];
   return $_[0];
$$ LANGUAGE plperl;

What the code does is to assign a value to some_name and returns the assigned value. Some other function can then make use of this data, which is stored inside your database connection. Here is an example:

CREATE OR REPLACE FUNCTION increment_var()
RETURNS int AS $$
   $_SHARED{'some_name'} += 1;
   return $_SHARED{'some_name'};
$$ LANGUAGE plperl;

This function will simply increment the value and return it. As you can see the code is pretty simple and easy to write.

Assigning shared variables

The following listing shows, how the code can be used. The first call will assign a value to the function while the second one will simply increment that value:

test=# SELECT set_var(5);
 set_var
---------
 5
(1 row)

test=# SELECT increment_var(), increment_var();
 increment_var | increment_var
---------------+---------------
             6 | 7
(1 row)

It is especially noteworthy here that the second column will already see the changes made by the first column, which is exactly what we want here.

Shared variables and transactions

When working with shared variables in PL/Perl or PL/Python you have to keep in mind that those changes will not be transactional as all the rest in PostgreSQL is. Even if you rollback a transaction you can observe that those values will stay incremented:

test=# BEGIN;
BEGIN
test=# SELECT increment_var(), increment_var();
increment_var  | increment_var
---------------+---------------
             8 | 9
(1 row)
test=# ROLLBACK;
ROLLBACK
test=# SELECT increment_var(), increment_var();
 increment_var | increment_var
---------------+---------------
            10 | 11
(1 row)

This behavior makes shared values actually a nice thing to have if you want to preserve data across transactions.

The post PostgreSQL: Sharing data across function calls appeared first on Cybertec.

plperl is a great extension for PostgreSQL that allows the execution of Perl 5 code within the database.

plperl: which version of Perl?

When executing Perl 5 code within the database, PostgreSQL uses the /embedded Perl 5/ to create one (or more) instance of the interpreter. The version of the compiler and virtual machine that runs depends on how PostgreSQL has been compiled, or better, how libperl.so has been created. It is possible to use a specific version of Perl without having to change the system wide Perl 5, and in particular it is possible with some effort to use perlbrew to this aim.

Understanding which perl the database is executing

To know which perl executable the server will run it is possible to use the Config module for a little introspection:

#DOLANGUAGEplperlu$PERL$useConfig;elog(INFO,'Perl executable '.$Config{perlpath});elog(INFO,'Perl version '.$Config{version});elog(INFO,'Perl library '.$Config{libperl});$PERL$;

For example, the above piece of code produces the following output on my system:

INFO: Perl executable /usr/local/bin/perl INFO: Perl version 5.24.3 INFO: Perl library libperl.so.5.24.3 

that tells clearly the perl executable is at version 5.24.3. The same could have been checked from the plperl.so library file, that is linked to the above version of the library:

Hey everyone! Just a quick note to let you all know that both CJ and myself are manning the PostgreSQL community booth at this year’s PyCon next week in Cleveland, OH.

The Python and PostgreSQL communities have a long history together, and we’re super excited to be able to represent PostgreSQL at the largest annual gathering of Pythonistas. If you can make the drive, come on up to Cleveland, say hi, and learn some Python!

https://us.pycon.org/2018/#!

With the addition of logical replication in Postgres 10, it’s natural to ask "what’s next"? Though not directly supported yet, would it be possible to subscribe two Postgres 10 nodes to each other? What kind of future would that be, and what kind of scenarios would be ideal for such an arrangement?

As it turns out, we already have a kind of answer thanks to the latency inherent to the speed of light: locality. If we can provide a local database node for every physical app location, we also reduce latency by multiple orders of magnitude.

Let’s explore the niche BDR was designed to fill.

What is Postgres-BDR?

Postgres-BDR is simply short for Postgres Bi-Directional Replication. Believe it or not, that’s all it needs to be. The implications of the name itself are numerous once fully explored, and we’ll be doing plenty of that.

So what does it do?

• Logical replication
• Multi-Master
• Basic conflict resolution (last update wins)
• Distributed locking
• Global sequences
• High latency replay (imagine a node leaves and comes back later)

The key to having Multi-Master and associated functionality is logical replication. Once we can attach to the logical replication stream, we just need a piece of software to communicate between all participating nodes to consume those streams. This is necessary to prevent nodes from re-transmitting rows received from another system and resolve basic conflicts.

Why Geographical Distribution?

Specifically:

• Local database instances
• Inter-node communication is high latency
• Eventual consistency is the only consistency (for now)
• That’s a Good Thing!(tm)

We don’t want latency between our application and the database, so it’s better to operate locally. This moves the latency into the back-end between the database nodes, and that means the dreaded "eventual consistency" model. It’s a natural consequence, but we can use it to our advantage.

Consider This

We have an application stack that operates in four locations: Sydney, Dubai, Dallas, and Tokyo.

We tried to get a data center in Antarctica, but the penguins kept going on strike. Despite that, here’s a look at average latencies between those locations:

Based on the lag, Tokyo seems to be something of a centralized location. Still, those round-trip-times are awful.

Eww, Math

Imagine our initial database is in Dubai.

Say we start in Dubai and want to expand to the US at first. We may be tempted to simply spin up an AWS or Google Cloud instance or two running our application. But like all modern websites, we make liberal use of AJAX and each page contains multiple database-backed components. Even five simple queries could drastically bloat page load time and drive away potential customers.

That’s bad, m’kay? This is hardly a new problem, and there are several existing solutions:

1. Front-end caching
   • Problems with extremely dynamic content.
   • Writes still need to go to Dubai.
2. Back-end caching
   • Only works for small-medium size datasets
   • Writes still need to go to Dubai.
3. Database replica
   • Writes still need to go to Dubai.

We could use something like Varnish to cache the pages themselves, and that works for a few cases where there isn’t a lot of dynamic or user-generated content. Or if there is, we better have a lot of cache storage. Hopefully we don’t have any pages that need to write data, because that still has to go back to Dubai.

Alternatively, we could use memchached, Cassandra, or some other intermediate layer the application can populate and interact with instead. This is better, but can get stale depending on how we refresh these caches. Some of these cache layers can even capture writes locally and persist by committing to the primary store… in Dubai. This makes for great interactivity with potential implications regarding data consistency.

And then there’s outright Postgres streaming replicas. This is much closer to ideal for highly dynamic content, barring any unexpected network interruptions. And as usual, writes must be made in Dubai.

Yet all three of these solutions suffer from the same downfall.

Broken Record

All those darn writes being rerouted to Dubai. Writes always need to end up in the primary database, and this is where BDR comes in.

If our primary database is in Dubai, all our writes must eventually make it there, one way or another. Then it must propagate through all of our caching layers and replicas before it’s considered committed by the end user.

Or we can remove the Middle Man with BDR. In order to do that, we want a situation where all writes are local to their region. Something like this:

With BDR, we still have a "master" copy in every case, but it’s not integral to basic operation.

How do we Get There?

1. Identify problem areas
   • Find all sequence usage
   • Isolate conflict scenarios
2. Install BDR
3. Create and subscribe

Now we get to the nitty-gritty. We need to start by making sure our sequences don’t clobber each other between the regions. Then we should consider all the ways interactions between regions might conflict. This may require a careful and time-consuming audit of the application stack, but it’s better than being surprised later.

Then we can work on deploying everything. Believe it or not, that’s probably the easiest part.

Seeking Sequences

We start our adventure by finding any serial columns that increment by using the Postgres nextval() function. BDR has a mechanism for ensuring sequence values never collide across the cluster, but we must first identify all sequences that are associated with a column that isn’t already a BIGINT. A query like this might work:

SELECT table_schema, table_name, column_name
  FROM information_schema.columns
 WHERE column_default LIKE 'nextval\(%'
   AND data_type != 'bigint';

BDR’s built-in global sequence functionality needs BIGINT columns because it uses a lot of bit-shift math and packs it into a 64-bit integer. The sequence is no longer sequential, but it’s unique for up to 8-million values per second, per node.

Conflict Avoidance

The best way to prevent conflicts is to avoid them outright. Aggregate columns like "total" or "sum" are potential hot-spots for conflicting updates. If such a column must be used, consider constructing it from a ledger or event log instead. This way it doesn’t matter if Sydney adds 100 to a balance, and Texas adds 50, because each event is atomic.

Once applications can, they should interact with data specific to their region whenever possible. The less cross-talk between different regions, the less potential there is for conflict. If we scale down slightly, we can even consider this within a single datacenter. The goal is to separate write targets by using sticky sessions, natural hashes, assigned application zones, and so on.

Similarly, even if a session is operating outside of its region, conflicts can be avoided if the application doesn’t switch back and forth. If an application can’t "see" multiple copies of the database at alternating times, it’s much less likely to generate a conflicting row due to latency mismatches.

Off to the Races

Why are stored aggregates bad? Race conditions. Consider this scenario on a simple account table with a balance column.

1. Sydney adds 50 to 100 balance
2. Tokyo adds 50 to 100 balance
3. Result should be 200, but is 150 instead due to latency
4. Ledger of credit / debit would always be correct
5. Think of a cash register receipt
6. Use materialized aggregates for reports or summaries

Going into more detail helps illustrate what kinds of conflicts we want to prevent. A ledger has no concept of sum, just plus or minus. It’s possible to refund specific line-items so the full history is maintained. We can also generate the summary for all rows in the grouping. If a historical row is temporarily missing on a remote node, transient summaries may temporarily vary, but the incorrect balance will never exist as actual data.

These kinds of modifications are the safest approach, but are not always necessary in isolated cases. This is why it’s important to evaluate first.

Shop S-Mart

Local database; local writes. This is important and comes in two flavors:

1. Tokyo operates on Tokyo data; Texas on Texas
   • Update conflicts less likely
   • Fewer / no race conditions
2. Dubai operates in Dubai; Sydney in Sydney
   • Prevents cross-data center conflicts
   • Local writes are faster anyway

Since every BDR node has a copy of the full database, we could theoretically operate outside of our region. Of course, it’s better if we don’t. If Tokyo operates on a row from Texas, there’s a good chance Texas will overwrite that change or cause a conflict. We can avoid a non-deterministic result by keeping regional operations together.

Alternatively, the application may be configured in Sydney and try to operate on Sydney rows in the Dubai data center. That’s fine until other Sydney nodes are simultaneously connected to the database in Sydney. Unless there’s a very good reason for this, it’s probably a bad idea. It’s slower, defeats the purpose of geographical database distribution in general, and is almost guaranteed to generate conflicts.

Of course, if one region must be offline for maintenance or other purposes, using another geographical database is encouraged. Though even then, since we’re talking about continental divides, it’s probably better to have multiple local alternatives instead.

Journey of 1000 Miles

Assuming we have an existing cluster, how do we distribute to new zones? Let’s start with a basic arbitrary table from a corporate ordering system.

CREATE TABLE system_order (
  order_id      SERIAL PRIMARY KEY,
  product_id    INT NOT NULL,
  customer_id   INT NOT NULL,
  unit_count    INT NOT NULL,
  status        VARCHAR NOT NULL DEFAULT 'pending',
  reading_date  TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_system_order_product_id ON system_order (unit_count);
CREATE INDEX idx_system_order_reading_date ON system_order (reading_date);

Take note of the SERIAL primary key, which corresponds to the underlying INT type. It’s an easy mistake if values aren’t expected to exceed 2-billion, but such isn’t compatible with BDR global sequences. A script like this might work for converting the existing tables before adding BDR to the mix:

COPY (
  SELECT 'ALTER TABLE ' || table_schema || '.' || table_name ||
         ' ALTER COLUMN ' || column_name || ' TYPE BIGINT;'
    FROM information_schema.columns
   WHERE column_default LIKE 'nextval\(%'
     AND data_type != 'bigint'
) TO '/tmp/alter_columns.sql';

\i /tmp/alter_columns.sql

Be wary that changing an underlying column type in Postgres results in the entire table being rebuilt. This may require extended downtime, or a more protracted migration using logical replication to another cluster where the types have already been converted.

We should also convert any columns that reference these as foreign keys, since the types should match. The query that does this is rather daunting however, so if requested, we’ll share it in the comments.

Seeding the World

Let’s assume Dubai is the origin node. This is currently how we’d initialize a cluster using the upcoming BDR 3 syntax:

CREATE EXTENSION pglogical;
CREATE EXTENSION bdr;

SELECT bdr.create_node(
  node_name := 'dubai',
  local_dsn := 'dbname=storefront host=dubai.host'
);

SELECT bdr.create_node_group(
  node_group_name := 'megacorp'
);

SELECT bdr.wait_for_join_completion();

With BDR, we "seed" a cluster from a single instance. In this case, the company started in Dubai.

Non Sequential

Now let’s fix those sequences. BDR 3 has a function that makes a metadata change so it treats a sequence as global automatically, using the 64-bit packing technique discussed earlier. We can bulk-convert all of them this way:

SELECT bdr.alter_sequence_set_kind(c.oid, 'timeshard')
  FROM pg_class c
  JOIN pg_namespace n ON (n.oid = c.relnamespace)
 WHERE c.relkind = 'S'
   AND n.nspname NOT IN (
         'pg_catalog', 'information_schema', 'pg_toast',
         'pglogical', 'bdr'
       );

From this point on, values generated by nextval() for these sequences will be essentially arbitrary numeric results with up to 19 digits. For platforms that already use something like UUIDs, this kind of conversion isn’t necessary. And be wary of applications that may malfunction when interacting with such large numbers; numeric overflow is nobody’s friend.

Johnny Appleseed

Tokyo has the fastest RTT, so obviously it was the next step as the company grew. Let’s create a node there now on an empty shell database:

CREATE EXTENSION pglogical;
CREATE EXTENSION bdr;

SELECT bdr.create_node(
  node_name := 'tokyo',
  local_dsn := 'dbname=storefront host=tokyo.host'
);

SELECT bdr.join_node_group(
  join_target_dsn := 'dbname=storefront host=dubai.host'
);

SELECT bdr.wait_for_join_completion();

Looks similar to creating the seed node, right? The only difference is that we need to join an existing BDR cluster instead of starting a new one. And then we just keep repeating the process for Texas, Sydney, or wherever we want until we’re satisfied. Nodes everywhere, until latency is finally at an acceptable level.

Next Steps

Once the cluster exists:

1. Point applications to local copies
2. Create any necessary read replicas/caches
3. Be smug (optional)

For every node we roll out, we can move the local application stack to use it instead of the initial copy in Dubai. That will make both the local users and the Dubai database much happier.

This also means each local BDR node can have any number of streaming replicas for read scaling or standby targets. This alone warrants another entire blog post, but we’ll save that for another time.

And of course if you’re so inclined, you can be happy for drastically improving the company’s application infrastructure. Don’t worry, we won’t judge.

Finishing UP

The remaining caveats aren’t too onerous.

Since BDR uses replication slots, if a node is unreachable, Postgres will begin to retain WAL files for consumption when it returns. To plan for this, we can just take the amount of WAL files that elapse over a certain time period. Each one is 16MB, which tells us how much space to set aside. If a node goes over this limit, we remove it.

Conflicts can be logged to a table, and we recommend enabling that functionality. Then a monitoring system should always watch the contents of that table on each individual node. Conflicts resolve locally, so if they happen, only the node that experienced the conflict will have rows there. Keep an eye on this to make sure BDR’s resolution was the right one.

A transaction on a local BDR node gets committed locally before it enters the Postgres logical replication stream. This asynchronous nature is a strength of why BDR works so well locally. It’s also the reason conflicts are possible. Keep this in mind and it will always be easier to troubleshoot conflicts.

In the end, we’re left with something that contributes to a global cumulative data resource, with the same responsiveness of local access. All thanks to harnessing some of the deeper capabilities of logical replication.

As long as we understand the inherent design limitations of multi-master architectures, we can safely deploy our application stack and dedicated databases anywhere in the world.

For more information on use cases, deployment strategies, and other concepts related to muli-master and BDR concepts, we have a Postgres-BDR whitepaper that goes into more depth about these topics.

PostgreSQL query tuning is our daily bread at Cybertec, and once you have done some of that, you’ll start bristling whenever you see an OR in a query, because they are usually the cause for bad query performance.

Of course there is a reason why there is an OR in SQL, and if you cannot avoid it, you have to use it. But you should be aware of the performance implications.

In this article I’ll explore “good” and “bad” ORs and what you can do to avoid the latter.

A little sample schema

We’ll use this simple setup for demonstration:

CREATE TABLE a(id integer NOT NULL, a_val text NOT NULL);

INSERT INTO a
   SELECT i, md5(i::text)
   FROM generate_series(1, 100000) i;

CREATE TABLE b(id integer NOT NULL, b_val text NOT NULL);

INSERT INTO b
   SELECT i, md5(i::text)
   FROM generate_series(1, 100000) i;

ALTER TABLE a ADD PRIMARY KEY (id);
ALTER TABLE b ADD PRIMARY KEY (id);
ALTER TABLE b ADD FOREIGN KEY (id) REFERENCES a;

VACUUM (ANALYZE) a;
VACUUM (ANALYZE) b;

Suppose that we want to run queries with equality and LIKE conditions on the text columns, so we need some indexes:

CREATE INDEX a_val_idx ON a(a_val text_pattern_ops);
CREATE INDEX b_val_idx ON b(b_val text_pattern_ops);

Have a look at the documentation if you don’t understand text_pattern_ops.

The “good” OR

An OR is fine in most parts of an SQL query: if it is not used to filter out rows from your query result, it will have no negative effect on query performance.

So if your OR appears in a CASE expression in the SELECT list, don’t worry.

Unfortunately you usually find the OR where it hurts: in the WHERE clause.

The “bad” OR

Now for an example of an OR in a WHERE clause that is still pretty nice:

EXPLAIN (COSTS off)
SELECT id FROM a
WHERE id = 42
   OR a_val = 'value 42';

                        QUERY PLAN                         
-----------------------------------------------------------
 Bitmap Heap Scan on a
   Recheck Cond: ((id = 42) OR (a_val = 'value 42'::text))
   ->  BitmapOr
         ->  Bitmap Index Scan on a_pkey
               Index Cond: (id = 42)
         ->  Bitmap Index Scan on a_val_idx
               Index Cond: (a_val = 'value 42'::text)
(7 rows)

PostgreSQL can actually use an index scan for the query, because it can combine the bitmaps for both indexes with a “bitmap OR”.
Note, however, that a bitmap index scan is much more expensive than a normal index scan — it has to scan the complete index. Moreover, it uses much more RAM; each of these bitmaps can use up to work_mem memory.

A multi-column index on (id, a_val) won’t help at all with this query, so there is no really cheap way to execute this query.
If you need better performance, see the trick from the “ugly” section below.

IN is better than OR

Now for a more stupid variant of the above query:

EXPLAIN (COSTS off)
SELECT id FROM a
WHERE id = 42
   OR id = 4711;

                 QUERY PLAN                 
--------------------------------------------
 Bitmap Heap Scan on a
   Recheck Cond: ((id = 42) OR (id = 4711))
   ->  BitmapOr
         ->  Bitmap Index Scan on a_pkey
               Index Cond: (id = 42)
         ->  Bitmap Index Scan on a_pkey
               Index Cond: (id = 4711)
(7 rows)

Again, a bitmap index scan is used. But there is a simple method to rewrite that query without the pesky OR:

EXPLAIN (COSTS off)
SELECT id FROM a
WHERE id IN (42, 4711);

                    QUERY PLAN                     
---------------------------------------------------
 Index Only Scan using a_pkey on a
   Index Cond: (id = ANY ('{42,4711}'::integer[]))
(2 rows)

You see? As soon as you get rid of the OR, an efficient index scan can be used!

You might say that this is good for equality conditions, but what about the following query:

SELECT id FROM a
WHERE a_val LIKE 'something%'
   OR a_val LIKE 'other%';

To improve that query, observe that the PostgreSQL optimizer rewrote the IN in the previous query to = ANY.

This is a case of the standard SQL “quantified comparison predicate”: <comparison operator> ANY is true if the comparison is TRUE for any of the values on the right-hand side (the standard only defines this for subqueries on the right-hand side, but PostgreSQL extends the syntax to arrays).

Now LIKE is a comparison operator as well, so we can write:

EXPLAIN (COSTS off)
SELECT id FROM a
WHERE a_val LIKE ANY (ARRAY['something%', 'other%']);

                        QUERY PLAN                        
----------------------------------------------------------
 Seq Scan on a
   Filter: (a_val ~~ ANY ('{something%,other%}'::text[]))
(2 rows)

Unfortunately, the index cannot be used here.

pg_trgm to the rescue

But we are not at the end of our wits yet! There is such a wealth of indexes in PostgreSQL; let’s try a different one. For this, we need the pg_trgm extension:

CREATE EXTENSION pg_trgm;

Then we can create a GIN trigram index on the column:

CREATE INDEX a_val_trgm_idx ON a USING gin (a_val gin_trgm_ops);

Now things are looking better:

EXPLAIN (COSTS off)
SELECT id FROM a
WHERE a_val LIKE ANY (ARRAY['something%', 'other%']);

                             QUERY PLAN                             
--------------------------------------------------------------------
 Bitmap Heap Scan on a
   Recheck Cond: (a_val ~~ ANY ('{something%,other%}'::text[]))
   ->  Bitmap Index Scan on a_val_trgm_idx
         Index Cond: (a_val ~~ ANY ('{something%,other%}'::text[]))
(4 rows)

Feel the power of trigram indexes!

Note 1: This index can also be used if the search pattern starts with %

Note 2: The GIN index can become quite large. To avoid that, you can also use a GiST index, which is much smaller, but less efficient to search.

The “ugly” OR

Things become really bad if OR combines conditions from different tables:

EXPLAIN (COSTS off)
SELECT id, a.a_val, b.b_val
FROM a JOIN b USING (id)
WHERE a.id = 42
   OR b.id = 42;

                 QUERY PLAN                  
---------------------------------------------
 Merge Join
   Merge Cond: (a.id = b.id)
   Join Filter: ((a.id = 42) OR (b.id = 42))
   ->  Index Scan using a_pkey on a
   ->  Index Scan using b_pkey on b
(5 rows)

Here we have to compute the complete join between the two tables and afterwards filter out all rows matching the condition. In our example, that would mean computing 100000 rows only to throw away the 99999 that do not natch the condition.

Avoiding the ugly OR

Fortunately, there is an equivalent query that is longer to write, but much cheaper to execute:

EXPLAIN (COSTS off)
   SELECT id, a.a_val, b.b_val
   FROM a JOIN b USING (id)
   WHERE a.id = 42
UNION
   SELECT id, a.a_val, b.b_val
   FROM a JOIN b USING (id)
   WHERE b.id = 42;

                        QUERY PLAN                        
----------------------------------------------------------
 Unique
   ->  Sort
         Sort Key: a.id, a.a_val, b.b_val
         ->  Append
               ->  Nested Loop
                     ->  Index Scan using a_pkey on a
                           Index Cond: (id = 42)
                     ->  Index Scan using b_pkey on b
                           Index Cond: (id = 42)
               ->  Nested Loop
                     ->  Index Scan using a_pkey on a a_1
                           Index Cond: (id = 42)
                     ->  Index Scan using b_pkey on b b_1
                           Index Cond: (id = 42)
(14 rows)

Both parts of the query can make use of efficient index scans and return one row, and since the rows happen to be identical, UNION will reduce them to one row.

If you can be certain that both branches of the query will return distinct sets, it is better to use UNION ALL instead of UNION, because that doesn’t have to do the extra processing to remove duplicates.

When using this trick, you should be aware that rewriting a query in that fashion does not always result in an equivalent query: if the original query can return identical rows, these would be removed by the UNION. In our case, we don’t have to worry, because the primary keys were included in the query result. I find that this is hardly ever a problem in practice.

The post Avoiding “OR” for better query performance appeared first on Cybertec.

Continuing our series of PostgreSQL Data Types today we’re going to introduce the PostgreSQL Point type.

In order to put the Point datatype in a context where it makes sense, we’re going to download a complete geolocation data set and normalize it, thus making good use of both the normalization good practice and those other PostgreSQL data types we’ve been learning about in the previous articles of this series.

Buckle-up, this is a long article with a lot of SQL inside.

plperl does not allow direct sub invocation, so the only way is to execute a query.

plperl: invoking other subroutines

The official plperl documentation shows you a way to use a subref to invoke code shared across different plperl functions via the special global hash %_SHARED. While this is a good approach, it only works for code attached to the hash, that is a kind of closure (e.g., a dispatch table), and requires each time an initialization of the %_SHARED hash since plperl interpreters does not share nothing across sections.

The other way, always working, is to execute a query to perform the SELECT that will invoke the function. As an example:

CREATEORREPLACEFUNCTIONplperl_trampoline(fun_nametext)RETURNSTEXTAS$PERL$my($fun_name)=@_;returnundefif(!$fun_name);elog(DEBUG,"Calling [$fun_name]");my$result_set=spi_exec_query("SELECT $fun_name() AS result;");return$result_set->{rows}[0]->{result};$PERL$LANGUAGEplperl;

so that you can simply do:

>selectplperl_trampoline('now');plperl_trampoline------------------------------2018-05-0413:09:17.11772+02

The problem of this solution should be clear: it can work only for a...

• Philadelphia 
• Ohio (in combination with Ohio Linux Fest) 
• South Africa 
• Seattle 
• Austin 
• Jersey City (PostgresConf US 2018) 
• Nepal 

PostgresConf US 2018

Logistics

• Days: 5, 2 for training, 3 for Breakout sessions and summits
• Official Attendance #: 601
• Content: Over 207 sessions submitted
• Sessions: Over 108 sessions provided 

Partner Support (Sponsors): 28

Diversity

Professional Growth

We held a Career and Talent Fair. A dozen companies were present to connect with potential employees.

We also held a surprisingly well attended speed mentoring session for potential employees (Especially helpful for many of the WWC) on resumes and interview practices.

Leadership

This year saw the continued elevation of our primary leadership: Viral Shah, Lloyd Albin, Amanda Nystrom, and Debra Cerda. They continued to increase their presence and responsibility within the conference and dedicated hundreds of hours voluntarily to the growth of people. Our international members have also increased their leadership roles with our on-the-ground teams in South Africa and China.

Summits

We had our standard Regulated Industry Summit but also a Greenplum Summit. As I am sure you are aware Greenplum is an Open Source, Postgres based MPP database. They were by far the most popular booth in the entire conference and their summit was very well attended. The relationship with Pivotal and the success of the Greenplum Summit allowed us to learn new ways to bring together the entire Postgres Ecosystem. We expect to run a minimum of 3 more summits at PostgresConf US 2019.

Contribution

We were able to have several excellent (and long) meetings with leaders of Pivotal, Microsoft, Google, and Amazon on how they can begin contributing more back to Postgresql.org. All of them expressed a deep drive to contribute and a desire to learn more about the core community. Of particular note is Google, who would like to contribute the following back to the community:

https://github.com/google/pg_page_verification

We discussed with them the process and various changes they would need to make (license and code style, etc.). We also educated them on the PostgreSQL.Org rigorous review process.

Microsoft is reviewing how they can contribute but they showed an interest in build farm nodes, professional technical writers to help with docs, and potentially code contribution to our Windows port.

International Collaboration

The Chinese Open Source Promotion Union launched the Chinese Postgres Association. We invited them to PostgresConf US and introduced them to the United States Community. We expect great things from the Chinese community in the future.

Future

As we continue to build up our on-the-ground teams, we will likely hold less events in the U.S. this year. We will instead be focused on a smaller number of events in the U.S. and adding events in China and Europe. We have had an amazing amount of support from the Chinese community and the current goal is 1000 attendees for that conference.

Our current plan of events for the U.S. are

• San Jose (October 2018) 
• Philadelphia 
• PostgresConf US (Manhattan) 

Future International Events

• October 2018. 
• Spring of 2019. 
• Spring of 2019. 

This may change as we are actively recruiting on-the-ground teams to help us grow the community.

Collaboration

Our goal is collaboration and growth with other PostgreSQL community and Ecosystem efforts. We want to allow each potential community member to find a home. A place that they feel positive about contributing to the community as a whole. As we continue to grow as a community, it is vital to recognize that each member has their own needs, desires, and return on investment requirement (professional or personal) that they are seeking.

Tidbits of note

• Pivotal went public during PostgresConf US 2018. 
• We (PostgreSQL and PostgresConf) were picked up by ZDNet during the conference
• Another flattering article about PostgreSQL

On DBEngines PostgreSQL is the 4th most popular database but the significance is that of the other 3, we are the only ones that are growing in popularity. 

Hi. I work as a data architect in San Francisco and I’m auditing Dr. Jones class to stay up to date on the latest technologies and she mentioned you might be able to help me before I get too deep into the design of a new system.

I would be happy to help. Can you give me an overview of where you’re at?

Well my default was just to use Postgres. I had a few questions on what schema designs might make most sense.

Well I’m working with more interesting data architectures. Really getting excited about what’s possible with neomodern data architectures, they make it so my app devs can build any feature their hearts desire.

I thought your expertise used to be relational databases?

It was, but neomodern data architectures are better. Neomodern data architectures allow it so app devs can build any feature they like without having to think about data models. Really, it’s the future of databases.

Hmmm, my app is a pretty straightforward web app that allows salespeople to send campaigns and track metrics on the campaigns. My app should be fine in a Postgres database right?

That might work for you, but really, if you have to define a data model up front, then you’re limiting yourself.

How do you mean limiting? Can’t I just add new tables as I need them? Or add columns to existing tables?

Well you could, but meta document stores can take care of that for you

Meta document stores?

Yeah, you have a custom document for each data model, but then have a document store that auto parses each document and can tell you the structure without having to read the document itself

Oh nice, so I can have an index of all my data.

Exactly, and because you have a bunch of smaller document stores you can distribute them around the world and read the data from the nearest one

So reading from the nearest copy of data would be faster?

Well it’s supposed to be, but to make sure the data is correct you have to read from at least 3 nodes to get consensus

Consensus?

Yeah concensus. Since the underlying storage is all in blockchain, there has to be an agreement about which of the data is a source of truth

How long does concensus take?

Right now, most queries take around 400 milliseconds, but I’m planning to rewrite the consensus algorithm over the next two months

In the past my Postgres database would perform reads in usually a millisecond or less. 400 ms doesn’t sound any better?

Well the key is scale, at larger scale. We’re planning ahead for really large growth so we needed to have something distributed in place

And what about writes? Is write performance just as bad?

Writes take about 3 seconds, but we don’t write to the database directly

You don’t write to the database?! What do you write to then?

Well we tried to write directly to the database, but for some reason there was clock skew which gave inconsistencies. So now we’re planning to upgrade to TrueTime atomic clocks in July.

What the hell! Where do you get atomic clocks from?

We sent our infrastructure guy to China to get our atomic clocks custom designed

This is running on your own hardware? Not on a cloud vendor like AWS, Microsoft, or Google?

We budget our hardware on a 5 year cycle, and accounting wouldn’t let us move the classification. Over the next 3 years, we’ll save about 50% on hardware cost, and almost break even when you include the people hours

Ouch! That sounds like a lot of work. Wait, can we back up for a minute. You said you don’t write to the database. If you don’t write directly to the database, what do you write to then?

Right now we dual write to Redis and Memcached

You’re running two things that you write to?! Why would you even need to do that?

They’re not normally used for transactions, but since they’re so fast, we can write to both. And since we write to both, we have a better chance of transactions

This is sounding like more work than I was hoping for. Is that all I need to know?

Oh we haven’t even gotten to the big wins yet

Okay, color me curious. What’s the payoff to all this investment?

I’ve seen the biggest value in how we’ve been able to give each analyst their own data lake.

Data lake?

Yeah, it’s like a partial copy of the database that has just what that person wants

You lost me

Because it’s schemaless, we have to use the meta document to parse it into a structured format so people can read it

How do you parse it?

We have event streams that feed out from Kafka

Where do you stream it?

Into Apache Spark

What’s Spark then do?

It computes pre-pre-aggregates

And those then get queried?

Oh, no. We push the pre-pre-aggregates to Druid, which computes our pre-aggregates

Okay, then you query the pre-aggregates?

No, that go into our data warehouse. Our data warehouse is great for analysts. It doesn’t help for user facing things, but our analysts can create some really powerful reports.

Oh, so you can make business decisions in real time?

Well because the pipeline data is usually about 12 hours behind, our business decisions are 12 hours behind. But we’re confident we can get it down to 8 hrs.

Okay, I think you lost me somewhere, but you’re saying the application developers are way more productive at least?

Oh the application developers haven’t gotten to try it yet, we’re just wrapping up our staging environment, then we have to start ordering the hardware for production. We plan to go live by 2022.

I think I may just stick with Postgres.

• Credit for inspiration to Paul Biggar

In the previous post we introduced the PostgreSQL’s dependency system.

At first sight the implementation can look like a maze where the succession of relationships are not clear.

This post will try to give a practical example to show how pg_depend can act like an Ariadne’s thread in order to resolve the dependencies.

The scenario presented is very simple but can be used as a starting point for more complex requirements.

The following commit adds a new feature which is part of Postgres 11, and matters a lot for a couple of tools:

commit: e79350fef2917522571add750e3e21af293b50fe
author: Stephen Frost <sfrost@snowman.net>
date: Fri, 6 Apr 2018 14:47:10 -0400
Remove explicit superuser checks in favor of ACLs

This removes the explicit superuser checks in the various file-access
functions in the backend, specifically pg_ls_dir(), pg_read_file(),
pg_read_binary_file(), and pg_stat_file().  Instead, EXECUTE is REVOKE'd
from public for these, meaning that only a superuser is able to run them
by default, but access to them can be GRANT'd to other roles.

Reviewed-By: Michael Paquier
Discussion: https://postgr.es/m/20171231191939.GR2416%40tamriel.snowman.net

This is rather a simple thing: a set of in-core functions like using a hardcoded superuser check to make sure that they do not run with unprivileged user rights. For the last couple of releases, an effort has been made to remove those hardcoded checks so as one can GRANT execution access to a couple or more functions so as actions which would need a full superuser (a user who theoritically can do anything on the cluster and administers it), are delegated to extra users with rights limited to those actions.

This commit, while making lookups to the data directory easier, is actually very useful for pg_rewind as it removes the need of having a database superuser in order to perform the rewind operation when the source server is up and running.

In order to get to this state, one can create a dedicated user and then grant execution to a subset of functions, which can be done as follows:

CREATE USER rewind_user LOGIN;
GRANT EXECUTE ON function pg_catalog.pg_ls_dir(text, boolean, boolean) TO rewind_user;
GRANT EXECUTE ON function pg_catalog.pg_stat_file(text, boolean) TO rewind_user;
GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text) TO rewind_user;
GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, boolean) TO rewind_user;

Once run, then this new database user “rewind_user” will be able to run pg_rewind without superuser rights, which matters for a lot of deployments as restricting superuser access to a cluster as much as possible is a common security policy. Note that pg_dump is able to dump ACLs on system functions since 9.6, so once put in place those policies remain in logical backups.

When: 6-8pm Thursday May 17, 2018
Where: iovation
Who: Dylan Hornstein
What: Learning SQL

During this PDXPUG meetup, we will talk about one person’s journey to learn SQL. From joining iovation’s Reporting Team without any experience with SQL or relational databases, to using SQL every day for adhoc data analysis and bulk data dumps, Dylan Hornstein will talk about his experience getting familiar with SQL and learning to navigate a relational database, as well as some challenges and tips he’s found along the way. The presentation is geared towards those starting out in data roles and we will likely expand into a wider conversation around using SQL and understanding data.

Dylan has been a Data Analyst at iovation for three years now. Having spent six months as a Client Manager before moving to his Data Analyst role, Dylan has experience working directly with iovation’s clients as well as working behind the scenes with the data. In his current role, he is responsible for providing reports, adhoc analysis and bulk data dumps for clients and internal teams. While Dylan had prior work experience as a Data Analyst, his move to iovation’s Reporting Team came with a steep learning curve, as he was new to working with SQL and relational databases.

—
If you have a job posting or event you would like me to announce at the meeting, please send it along. The deadline for inclusion is 5pm the day before the meeting.
—

Our meeting will be held at iovation, on the 3rd floor of the US Bancorp Tower at 111 SW 5th (5th & Oak). It’s right on the Green & Yellow Max lines. Underground bike parking is available in the parking garage; outdoors all around the block in the usual spots. No bikes in the office, sorry! For access to the 3rd floor of the plaza, please either take the lobby stairs to the third floor or take the plaza elevator (near Subway and Rabbit’s Cafe) to the third floor. There will be signs directing you to the meeting room. All attendess must check in at the iovation front desk.

See you there!

The slides from my talk on PostgreSQL Replication at PerconaLive 2018 are available.