As United States PostgreSQL continues its support for PostgreSQL User Groups it is my pleasure to announce that the Dallas/Forth Worth PostgreSQL User Group has decided to become part of the PgUS family. The resolution passed with 100% consent and it is great to see them on board. You may visit the resolution here:

https://postgresql.us/node/144

read more

In an effort to utilize infrastructure that is far more capable than what we can as volunteers provide, PgUS has moved to Google Apps. As a 501c3 we are offered a host of benefits that will help us grow the organization as well as increase collaboration and communication among the board and corporation members. We expect to be performing technical and advocacy hangouts often, as well as configuring a PgUS YouTube channel to put forth video resources for those wanting to know and learn more about PostgreSQL.

Look for more news in the future. It is exciting times!

PostGIS raster has so so many functions and probably at least 10 ways of doing something some much much slower than others. Suppose you have a raster, or you have a raster area of interest — say elevation raster for example, and you want to know the distinct pixel values in the area. The temptation is to reach for ST_Value function in raster, but there is a much much more efficient function to use, and that is the ST_ValueCount function.

ST_ValueCount function is one of many statistical raster functions available with PostGIS 2.0+. It is a set returning function that returns 2 values for each row: a pixel value (val), and a count of pixels in the raster that have that value. It also has variants that allow you to filter for certain pixel values.

This tip was prompted by the question on stackexchange How can I extract all distinct values from a PostGIS Raster?

pg_xlog with a dummy file
(image by Andrew Malone)

Running out of disk space in the pg_xlog directory is a fairly common Postgres problem. This important directory holds the WAL (Write Ahead Log) files. (WAL files contain a record of all changes made to the database - see the link for more details). Because of the near write‑only nature of this directory, it is often put on a separate disk. Fixing the out of space error is fairly easy: I will discuss a few remedies below.

When the pg_xlog directory fills up and new files cannot be written to it, Postgres will stop running, try to automatically restart, fail to do so, and give up. The pg_xlog directory is so important that Postgres cannot function until there is enough space cleared out to start writing files again. When this problem occurs, the Postgres logs will give you a pretty clear indication of the problem. They will look similar to this:

PANIC:  could not write to file "pg_xlog/xlogtemp.559": No space left on device
STATEMENT:  insert into abc(a) select 123 from generate_series(1,12345)
LOG:  server process (PID 559) was terminated by signal 6: Aborted
DETAIL:  Failed process was running: insert into abc(a) select 123 from generate_series(1,12345)
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally an
d possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted; last known up at 2014-09-16 10:36:47 EDT
LOG:  database system was not properly shut down; automatic recovery in progress
FATAL:  the database system is in recovery mode
LOG:  redo starts at 0/162FE44
LOG:  redo done at 0/1FFFF78
LOG:  last completed transaction was at log time 2014-09-16 10:38:50.010177-04
PANIC:  could not write to file "pg_xlog/xlogtemp.563": No space left on device
LOG:  startup process (PID 563) was terminated by signal 6: Aborted
LOG:  aborting startup due to startup process failure

The "PANIC" seen above is the most severe log_level Postgres has, and it basically causes a "full stop right now!". You will note in the above snippet that a normal SQL command caused the problem, which then caused all other Postgres processes to terminate. Postgres then tried to restart itself, but immediately ran into the same problem (no disk space) and thus refused to start back up. (The "FATAL" line above was another client trying to connect while all of this was going on.)

Before we can look at how to fix things, a little background will help. When Postgres is running normally, there is a finite number of WAL files (roughly twice the value of checkpoint_segments) that exist in the pg_xlog directory. Postgres deletes older WAL files, so the total number of files never climbs too high. When something prevents Postgres from removing the older files, the number of WAL files can grow quite dramatically, culminating in the out of space condition seen above. Our solution is therefore two-fold: fix whatever is preventing the old files from being deleted, and clear out enough disk space to allow Postgres to start up again.

The first step is to determine why the WAL files are not being removed. The most common case is a failing archive_command. If this is the case, you will see archive-specific errors in your Postgres log. The usual causes are a failed network, downed remote server, or incorrect copying permissions. You might see some errors like this:

2013-05-06 23:51:35 EDT [19421]: [206-1] user=,db=,remote= LOG:  archive command failed with exit code 14
2013-05-06 23:51:35 EDT [19421]: [207-1] user=,db=,remote= DETAIL:  The failed archive command was: rsync --whole-file --ignore-existing --delete-after -a pg_xlog/000000010000006B00000016 backup:/archive/000000010000006B00000016
rsync: Failed to exec ssh: Permission denied (13)
# the above was from an actual bug report; the problem was SELinux

There are some other reasons why WAL would not be removed, such as failure to complete a checkpoint, but they are very rare so we will focus on archive_command. The quickest solution is to fix the underlying problem by bringing the remote server back up, fixing the permissions, etc. (To debug, try emulating the archive_command you are using with a small text file, as the postgres user. It is generally safe to ship non-WAL files to the same remote directory). If you cannot easily or quickly get your archive_command working, change it to a dummy command that always returns true:

# On Nix boxes:
archive_command = '/bin/true'
# On BSD boxes:
archive_command = '/usr/bin/true'
# On Windows boxes:
archive_command = 'REM'

This will allow the archive_command to complete successfully, and thus lets Postgres start removing older, unused WAL files. However, you cannot start the server yet, because the lack of disk space is still a problem. Here is what the logs would look like if you tried to start it up again:

LOG:  database system shutdown was interrupted; last known up at 2014-09-16 10:38:54 EDT
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  redo starts at 0/162FE44
LOG:  redo done at 0/1FFFF78
LOG:  last completed transaction was at log time 2014-09-16 10:38:50.010177-04
PANIC:  could not write to file "pg_xlog/xlogtemp.602": No space left on device
LOG:  startup process (PID 602) was terminated by signal 6: Aborted
LOG:  aborting startup due to startup process failure

At this point, you must provide Postgres a little bit of room in the partition/disk that the pg_xlog directory is in. There are three approaches to doing so: removing some non-WAL files, resizing the partition, and removing some of the WAL files yourself.

The easiest solution is to clear up space by removing any non-WAL files that are on the same partition. If you do not have pg_xlog on its own partition, just remove a few files (or move them to another partition) and then start Postgres. You don't need much space - a few hundred megabytes should be more than enough.

This problem occurs often enough that I have a best practice: create a dummy file on your pg_xlog partition whose sole purpose is to get deleted after this problem occurs, and thus free up enough space to allow Postgres to start! Disk space is cheap these days, so just create a 300MB file and put it in place like so (on Linux):

dd if=/bin/zero of=/pgdata/pg_xlog/DO_NOT_MOVE_THIS_FILE bs=1MB count=300

This is a nice trick, because you don't have to worry about finding a file to remove, or determine which WALs to delete - simply move or delete the file and you are done. Once things are back to normal, don't forget to put it back in place.

Another way to get more space in your pg_xlog partition is to resize it. Obviously this is only an option if your OS/filesystem has been setup to allow resizing, but if it is, this is a quick and easy way to give Postgres enough space to startup again. No example code on this one, as the way to resize disks varies so much.

The final way is to remove some older WAL files. You need to determine which files are safest to remove. One way to determine this is to use the pg_controldata program. Just run it with the location of you data directory as the only argument, and you should be rewarded with a screenfull of arcane information. The important lines will look like this:

Latest checkpoint's REDO location:    0/4000020
Latest checkpoint's REDO WAL file:    000000010000000000000005

This second line represents the last WAL file processed, and it should be safe to remove any files older than that one. (Unfortunately, older versions of PostgreSQL will not show that line, and only the REDO location. While the canonical way to translate the location to a filename is with the pg_xlogfile_name() function, it is of little use in this situation, as it requires a live database! Thus, you may need another solution.)

Once you know which WAL file to keep by looking at the pg_controldata output, you can simply delete all WAL files older than that one. As with all mass deletion actions, I recommend a three-part approach. First, back everything up. This could be as simple as copying all the files in the pg_xlog directory somewhere else. Second, do a trial run. This means seeing what the deletion would do without actually deleting the files. For some commands, this means using a --dry-run or similar option, but in our example below, we can simply leave out the "-delete" argument. Third, carefully perform the actual deletion. In our example above, we could clear the old WAL files by doing:

$ cp -r /pgdata/pg_xlog/* /home/greg/backups/
$ find -L /pgdata/pg_xlog -not -newer /pgdata/pg_xlog/000000010000000000000005 | sort | less
$ find -L /pgdata/pg_xlog -not -newer /pgdata/pg_xlog/000000010000000000000005 -delete

Once you have straightened out the archive_command and cleared out some disk space, you are ready to start Postgres up. You may want to adjust your pg_hba.conf to keep everyone else out until you verify all is working. When you start Postgres, the logs will look like this:

LOG:  database system was shut down at 2014-09-16 10:28:12 EDT
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

After a few minutes, check on the pg_xlog directory, and you should see that Postgres has deleted all the extra WAL files, and the number left should be roughly twice the checkpoint_segments setting. If you adjusted pg_hba.conf, adjust it again to let clients back in. If you changed your archive_command to always return truth, remember to change it back as well as generate a new base backup

Now that the problem is fixed, how do you prevent it from happening again? First, you should use the 'tail_n_mail' program to monitor your Postgres log files, so that the moment the archive_command starts failing, you will receive an email and can deal with it right away. Making sure your pg_xlog partition has plenty of space is a good strategy as well, as the longer it takes to fill up, the more time you have to correct the problem before you run out of disk space.

Another way to stay on top of the problem is to get alerted when the pg_xlog directory starts filling up. Regardless of whether it is on its own partition or not, you should be using a standard tool like Nagios to alert you when the disk space starts to run low. You can also use the check_postgres program to alert you when the number of WAL files in the pg_xlog directory goes above a specified number.

In summary, things you should do now to prevent, detect, and/or mitigate the problem of running out of disk space in pg_xlog:

1. Move pg_xlog to its own partition. This not only increases performance, but keeps things simple and makes thins like disk resizing easier.
2. Create a dummy file in the pg_xlog directory as described above. This is a placeholder file that will prevent the partition from being completely filled with WAL files when 100% disk space is reached.
3. Use tail_n_mail to instantly detect archive_command failures and deal with them before they lead to a disk space error (not to mention the stale standby server problem!)
4. Monitor the disk space and/or number of WAL files (via check_postgres) so that you are notified that the WALs are growing out of control. Otherwise your first notification may be when the database PANICs and shuts down!

In summary, don't panic if you run out of space. Do the steps above, and rest assured that no data corruption or data loss has occurred. It's not fun, but there are far worse Postgres problems to run into! :)

copy the_table(jsonfield) 
from '/path/to/jsondata' 
csv quote e'\x01' delimiter e'\x02';

The following simple commit has improved the verbose logs of pg_dump (the ones that can be invocated with option -v and that are useful to keep a log trace when using cron jobs kicking pg_dump), by making the schema names of the relations dumped show up as well:

commit: 2bde29739d1e28f58e901b7e53057b8ddc0ec286
author: Heikki Linnakangas <heikki.linnakangas@iki.fi>
date: Tue, 26 Aug 2014 11:50:48 +0300
Show schema names in pg_dump verbose output.

Fabrízio de Royes Mello, reviewed by Michael Paquier

Let's take the case of a simple schema, with the same table name used on two different schemas:

=# CREATE SCHEMA foo1;
CREATE SCHEMA
=# CREATE SCHEMA foo2;
CREATE SCHEMA
=# CREATE TABLE foo1.dumped_table (a int);
CREATE TABLE
=# CREATE TABLE foo2.dumped_table (a int);
CREATE TABLE

With pg_dump bundled with 9.4 and older versions, each relation cannot be really identified (think about the case of having multiple versions of an application schema stored in the same database, but with different schema names):

$ pg_dump -v 2>&1 >/dev/null | grep dumped_table | grep TABLE
pg_dump: creating TABLE dumped_table
pg_dump: creating TABLE dumped_table
pg_dump: setting owner and privileges for TABLE dumped_table
pg_dump: setting owner and privileges for TABLE dumped_table
pg_dump: setting owner and privileges for TABLE DATA dumped_table
pg_dump: setting owner and privileges for TABLE DATA dumped_table

Now with 9.5, the following logs are showed.

$ pg_dump -v 2>&1 >/dev/null | grep dumped_table | grep TABLE
pg_dump: creating TABLE "foo1"."dumped_table"
pg_dump: creating TABLE "foo2"."dumped_table"
pg_dump: setting owner and privileges for TABLE "foo1"."dumped_table"
pg_dump: setting owner and privileges for TABLE "foo2"."dumped_table"
pg_dump: setting owner and privileges for TABLE DATA "foo1"."dumped_table"
pg_dump: setting owner and privileges for TABLE DATA "foo2"."dumped_table"

Note as well the quotes put around the relation and schema names, making this output more consistent with the other utilities in PostgreSQL. Also, this is of course not only limited to relations, but to any object that can be defined on a schema.

The newest round of performance comparisons of PostgreSQL and MongoDB produced a near repeat of the results from the first tests that proved PostgreSQL can outperform MongoDB. The advances Postgres has made with JSON and JSONB have transformed Postgres’ ability to support a document database.

select count(*) from jr where jr @@ ' similar_product_ids && ["B000089778"] 
AND product_sales_rank( $ > 10000 AND $  < 20000)';

 
select count(*) from jr where jr @@  ' similar_product_ids && ["B000089778"]';

db.reviews.find(  {  $and :[ {similar_product_ids: { $in:["B000089778"]}},    {product_sales_rank:{$gt:10000, $lt:20000}}] } )
.explain()
{
	"n" : 45,
	 ….................
	"millis" : 7,
	"indexBounds" : {
		"similar_product_ids" : [                                   
			[
				"B000089778",
				"B000089778"
			]
		]
	},
}

Aggregate (actual time=0.479..0.479 rows=1 loops=1)
   ->  Bitmap Heap Scan on jr (actual time=0.079..0.472 rows=45 loops=1)
         Recheck Cond: (jr @@ '"similar_product_ids" && ["B000089778"]'::jsquery)
         Filter: ((((jr ->> 'product_sales_rank'::text))::integer > 10000) AND 
(((jr ->> 'product_sales_rank'::text))::integer < 20000))
         Rows Removed by Filter: 140
         Heap Blocks: exact=107
         ->  Bitmap Index Scan on jr_path_value_idx (actual time=0.041..0.041 rows=185 loops=1)
               Index Cond: (jr @@ '"similar_product_ids" && ["B000089778"]'::jsquery)
 Execution time: 0.506 ms

explain (analyze, costs off) select count(*) from jr where jr @@ 'similar_product_ids &&["B000089778"] 
AND product_sales_rank( $ > 10000 AND $  < 20000)'
------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate (actual time=0.422..0.422 rows=1 loops=1)
   ->  Bitmap Heap Scan on jr (actual time=0.099..0.416 rows=45 loops=1)
         Recheck Cond: (jr @@ '("similar_product_ids" && ["B000089778"] AND "product_sales_rank"($ > 10000 AND $ < 20000))'::jsquery)
         Rows Removed by Index Recheck: 140
         Heap Blocks: exact=107
         ->  Bitmap Index Scan on jr_path_value_idx (actual time=0.060..0.060 rows=185 loops=1)
               Index Cond: (jr @@ '("similar_product_ids" && ["B000089778"] AND "product_sales_rank"($ > 10000 AND $ < 20000))'::jsquery)
 Execution time: 0.480 ms

SELECT gin_debug_query_path_value('similar_product_ids && ["B000089778"] 
                       AND product_sales_rank( $ > 10000 AND $  < 20000)');
           gin_debug_query_path_value
-------------------------------------------------
 similar_product_ids.# = "B000089778" , entry 0 +

SELECT gin_debug_query_path_value('product_sales_rank > 10000');
      gin_debug_query_path_value
---------------------------------------
 product_sales_rank > 10000 , entry 0 +

SELECT gin_debug_query_path_value('product_sales_rank /*-- noindex */ > 10000');
 gin_debug_query_path_value
----------------------------
 NULL                      +

This is the post I promised last week, explaining a few common issues with memory contexts. The issues mostly lead to excessive overhead, i.e. excessive usage of resources - usually memory. And that's exactly what this post is about, so whenever I say "overhead" you can read "excessive memory usage." I will also try to give advices on how to avoid those issues or minimize the impact.

As I briefly mentioned when explaining allocation set internals, there are three main sources of overhead:

• chunk headers - for large chunks, this gets negligible
• unused space (because of 2^N chunks) - expected ~25% for randomly sized chunks, but can get much worse
• reuse not working efficiently - we'll see some examples how this can happen

If you haven't read that post, it's probably the right time to do that. Also, if you want to learn more about memory management and allocator implementations, there's a great post at IBM developerWorks explaining it quite well and also listing many interesting additional resources (e.g. various malloc implementations).

So let's see some usual (but somehow unexpected) examples of palloc overhead.

Allocating many tiny pieces of memory

Say we need to store a lot of small elements, a few bytes each - e.g. 64-bit integers, words in an ispell dictionary or something like that. You could do a separate palloc call for each element (and keep the pointer), but in that case you'll pay the 'header' price for each element. For example by doing this

int64*items[1024];for(i=0;i<1024;i++)items[i]=(int*)palloc(sizeof(int64));

you might think you allocated ~8kB of memory (1024 x 8B), but in fact this prepends each value with 16B header. So you end up with about 24kB (not counting the items array, which needs additional 8kB), which is ~3x the requested amount. If we asked for smaller values (e.g. 4B integers), the overhead would be even larger because 8B is the smallest chunk allocated by palloc (as mentioned in the previous post).

Let's assume all the elements have about the same life span / scope (i.e. can be freed at the same time), and don't need to be passed somewhere else (i.e. are used only locally). In such cases the best approach is often preallocating a large chunk of memory, and then slicing it into small pieces - essentially doing what the allocation set allocator does when slicing the block into chunks, but without the chunk headers.

Of course, that means you won't be able to call pfree on the elements, but that's OK because we assumed only local usage (if we pass one of the elements somewhere else, that code would be unaware of this). But if the elements have the same life span (as for example words in an ispell dictionary), we don't really need to call the pfree on the individual words as the dictionary gets freed at once.

This is especially trivial to do when the elements are of fixed size, and you know how many of them to expect - in that case you can just do

char*tmp=palloc(elementSize*elementCount)tmp->element#0(tmp+elementSize)->element#1...(tmp+k*elementSize)->element#k

and you're done. But what to do when the elements are variable-sized, or when you don't know how many of them to expect? Well, who says you have to allocate all the elements at once? You can allocate a large chunk of memory, use it for elements until you run our of space, then allocate another one and start over.

A very simple example of this approach is compact_palloc0 method in spell.c, which is used for efficient allocation of words and other dictionary-related data (for the purpose of fulltext-search) (the following code is somewhat simplified, to make it easier to understand):

# define COMPACT_ALLOC_CHUNK 8192/* current chunk and available space */char*current_ptr=NULL;Sizeavailable_space=0;staticvoid*compact_palloc0(IspellDict*Conf,size_tsize){void*result;/* Need more space? */if(size>available_space){current_ptr=palloc0(COMPACT_ALLOC_CHUNK);available_space=COMPACT_ALLOC_CHUNK;}result=(void*)current_ptr;current_ptr+=size;available_space-=size;returnresult;}

This effectively adds another (very simple) allocator on top of the AllocSet. The "large" chunks (allocated using palloc) are registered in the parent memory context and will be freed when that memory allocator gets destroyed.

Another example such "dense" allocation was recently committed into our hashjoin implementation - it's however slightly more complicated (to support some hashjoin-specific requirements). See commit 45f6240a for more details.

Allocating pieces that are not 2^N bytes

The other trap you may fall into is the 2^N sizing of chunks. There are plenty of ways how to "achieve" that - I'll show you a real-world example from my quantile extension. I already fixed it, so you have to look at commit c1e7bba9 to see it.

Let's say you're writing an aggregate function MEDIAN(float) - to achieve that, you need to collect the float values into the aggregate state (so that you can later sort them and choose the "middle" value, which is the median). The aggregate state might be represented by this structure

typedefstructmedian_state{intnelements;intnext;float*elements;}median_state;

The float elements are accumulated into 'elements' array, which is resized on the fly as needed. The size of the array (including unusued part) is tracked by 'nelements' and 'next' is index of the next available element. So when next == nelements happends, we need to enlarge the array, which is done like this:

#define SLICE_SIZE 5...if(state->nelement==0){state->nelements=SLICE_SIZE;state->elements=(float*)palloc(sizeof(float)*data->nelements);}elseif(state->next>state->nelements-1){state->nelements=state->nelements+SLICE_SIZE;state->elements=(float*)repalloc(state->elements,sizeof(float)*data->nelements);}

Can you spot the problem? It's pretty obvious :-/

Well, every time we call repalloc, the 'elements' array grows by 20 bytes (SLICE_SIZE * 4B). That's rather pointless, because it either fits into the current chunk (and then it's almost free), or the array has to be moved to a larger chunk. But it does not save any memory at all.

Moreover there's a small inefficiency, because 20 is not a divisor of chunk sizes (following the 2^N rule). This wastes a bit of memory, but for larger chunks this is negligible.

It however clearly shows that constant growth does not work, because it does not follow the 2^N chunk size pattern. A saner version of the resize would be about this:

#define SLICE_SIZE 8...if(state->nelement==0){state->nelements=SLICE_SIZE;state->elements=(float*)palloc(sizeof(float)*data->nelements);}elseif(state->next>state->nelements-1){state->nelements=state->nelements*2;state->elements=(float*)repalloc(state->elements,sizeof(float)*data->nelements);}

So, this is better - it starts with 8 elements (because 8 * 4B = 32B, which is 2^5 and thus follows the 2^N rule). I could have used smaller value (up to 2, because the smallest chunk is 8B). This would have about the same memory consumption as before, but it's more correct and the repalloc will be called much less frequently (exponentially less).

Now, let's see a more serious example, that I almost commited into count_distinct but luckily realized the error before doing that:

Let's say the state for the MEDIAN() aggregate was defined like this:

typedefstructmedian_state{intnelements;intnext;floatelements[1];}median_state;

Placing a single-element array at the end of struct is a well known trick to define variable length structures. The resize then might look like this:

#define SLICE_SIZE 8...if(state->nelement==0){state->nelements=SLICE_SIZE;state->elements=(float*)palloc(offsetof(median_state,element)+sizeof(float)*data->nelements);}elseif(state->next>state->nelements-1){state->nelements=state->nelements*2;state->elements=(float*)repalloc(state->elements,offsetof(median_state,element)+sizeof(float)*data->nelements);}

We're still keeping the array nicely sized (still 2^N bytes), so perfect, right? Well, no. What gets allocated is the whole structure, and sadly that's always (2^N + 8B), because the two integers are part of the chunk. So we'll always get the perfect size + 2B. Which pretty much says we'll get 2x the necessary chunk size (because of the overflowing 8B). But this time we're guaranteed to waste the upper half of the chunk (except the first 8B). That kinda sucks.

There are two ways to fix this - either by allocating the array separately (which is what I did in count_distinct) or keeping the total size 2^N (and tweaking the nelements appropriately).

Creating many small contexts

Sometimes "less is more" and it certainly holds for memory contexts. A nice example is array_agg - an aggregate function that accumulates all the values into an array. The heavylifting is done by accumArrayResult in arrayfuncs.c. This piece of code performs initialization when the first value is passed to the function:

if(astate==NULL){/* First time through --- initialize *//* Make a temporary context to hold all the junk */arr_context=AllocSetContextCreate(rcontext,"accumArrayResult",ALLOCSET_DEFAULT_MINSIZE,ALLOCSET_DEFAULT_INITSIZE,ALLOCSET_DEFAULT_MAXSIZE);oldcontext=MemoryContextSwitchTo(arr_context);astate=(ArrayBuildState*)palloc(sizeof(ArrayBuildState));astate->mcontext=arr_context;astate->alen=64;/* arbitrary starting array size */astate->dvalues=(Datum*)palloc(astate->alen*sizeof(Datum));astate->dnulls=(bool*)palloc(astate->alen*sizeof(bool));astate->nelems=0;astate->element_type=element_type;get_typlenbyvalalign(element_type,&astate->typlen,&astate->typbyval,&astate->typalign);}

It's slightly complicated because it sets a lot of values in the aggregate state, but apparently it preallocates space for 64 elements (astate->alen), which is 512B on 64-bit architectures. It then properly doubles the size (not shown here bor brevity). Perfect, right? What could go wrong?

Well, the first thing the code actually does is creating a dedicated memory context (for this group), and it uses ALLOCSET_DEFAULT_INITSIZE as the initial block size. And ALLOCSET_DEFAULT_INITSIZE is 8kB, so on the first palloc, this memory context allocates 8kB block. The fact that we wanted to preallocate space for only 64 elements is irrelevant - we'll get 16x that.

Now, imagine aggregation with many distinct groups. Each group will get 8kB, even though there may be a single item in the array. And we actually get bug reports related to this.

What makes this even worse is that keeping per-group memory contexts makes it impossible to reuse chunks across groups.

Gradually growing request sizes

The last issue is quite different from the previous ones, because it's about (in)efficient chunk reuse.

As I mentioned, chunks are not really freed - instead they're moved to a freelist for later reuse. So when you do palloc(50) you'll get 64B chunk, and when you do pfree(chunk) it'll be moved to a freelist and eventually used for other requests needing 64B chunks.

But what happens if the requested sizes only grow? Consider for example this example:

inti=0,j=0;size_tcurrent_size=8;intnchunks=100;intnloops=10;char*tmp[nchunks];for(i=0;i<nchunks;i++)tmp[i]=palloc(current_size);for(i=0;i<nloops;i++){current_size*=2;for(j=0;j<nchunks;j++)tmp[j]=repalloc(tmp[j],current_size);}

So, how much memory is allocated at the end? There are 100 chunks, and the final chunk size is 8kB. So it has to be 800kB, right?

Actually, it's about double that, because none of the smaller chunks will ever be reused. The fact that the request sizes only grow prevents chunk reuse - the chunks will get stuck in the freelists until the memory context is destroyed.

It's however true that this overhead is bounded (it can't be higher than 100%, because it's bounded by a sum of infinite series 1/(2^k)), and most workloads actually mix requests of various sizes (making the reuse possible).

Summary

• Don't try to be overly smart - follow the 2^N rule by allocating properly sized pieces and doubling the size when needed.
• Where applicable, consider using dense allocation (as for example compact_palloc in spell.c).

Most web applications will add/remove columns over time. This is extremely common early on and even mature applications will continue modifying their schemas with new columns. An all too common pitfall when adding new columns is setting a not null constraint in Postgres.

Not null constraints

What happens when you have a not null constraint on a table is it will re-write the entire table. Under the cover Postgres is really just an append only log. So when you update or delete data it’s really just writing new data. This means when you add a column with a new value it has to write a new record. If you do this requiring columns to not be null then you’re re-writing your entire table.

Where this becomes problematic for larger applications is it will hold a lock preventing you from writing new data during this time.

A better way

Of course you may want to not allow nulls and you may want to set a default value, the problem simply comes when you try to do this all at once. The safest approach at least in terms of uptime for your table –> data –> appliction is to break apart these steps.

1. Start by simply adding the column with allowing nulls but setting a default value
2. Run a background job that will go and retroactively update the new column to your default value
3. Add your not null constraint.

Yes it’s a few extra steps, but I can say from having walked through this with a number of developers and their apps it makes for a much smoother process for making changes to your apps.

When: 6-8pm Thu Oct 16, 2014
Where: Iovation
What: PgOpen Recap (gabrielle, Mark, John M);  New Relic Instrumentation of Pg Queries (Andrew)

Please note the new earlier meeting time! We’ll try this over the winter.

Two topics this month:  PgOpen attendees will discuss highlights of that conference, and Andrew will talk about some New Relic-y stuff.

Our meeting will be held at Iovation, on the 32nd floor of the US Bancorp Tower at 111 SW 5th (5th & Oak).  It’s right on the Green & Yellow Max lines.  Underground bike parking is available in the parking garage;  outdoors all around the block in the usual spots.  No bikes in the office, sorry!

Elevators open at 5:45 and building security closes access to the floor at 6:30.

Row-level security is a new feature of PostgreSQL 9.5 that has been introduced by this commit:

commit: 491c029dbc4206779cf659aa0ff986af7831d2ff
author: Stephen Frost <sfrost@snowman.net>
date: Fri, 19 Sep 2014 11:18:35 -0400
Row-Level Security Policies (RLS)

Building on the updatable security-barrier views work, add the
ability to define policies on tables to limit the set of rows
which are returned from a query and which are allowed to be added
to a table.  Expressions defined by the policy for filtering are
added to the security barrier quals of the query, while expressions
defined to check records being added to a table are added to the
with-check options of the query.

Behind this jargon is a feature that could be defined in short words as a complementary permission manager of GRANT and REVOKE that allows controlling at row level which tuples can be retrieved for a read query or manipulated using INSERT, UPDATE or DELETE. This row control mechanism is controlled using a new query called CREATE POLICY (of course its flavor ALTER POLICY to update an existing policy and DROP POLICY to remove a policy exist as well). By default, tables have no restrictions in terms of how rows can be added and manipulated. However they can be made able to accept level restriction policies using ALTER TABLE and ENABLE ROW LEVEL SECURITY. Now, let's imagine the following table where a list of employees and their respective salaries can be read (salary is an integer as this is entirely fictive situation and refers to no real situation, quoique...):

=# CREATE TABLE employee_data (id int,
       employee text,
       salary int,
       phone_number text);
CREATE TABLE
=# CREATE ROLE ceo;
CREATE ROLE
=# CREATE ROLE jeanne;
CREATE ROLE
=# CREATE ROLE bob;
CREATE ROLE
=# INSERT INTO employee_data VALUES (1, 'ceo', 300000, '080-7777-8888');
INSERT 0 1
=# INSERT INTO employee_data VALUES (2, 'jeanne', 1000, '090-1111-2222');
INSERT 0 1
=# INSERT INTO employee_data VALUES (3, 'bob', 30000, '090-2222-3333');
INSERT 0 1

Now let's set some global permissions on this relation using GRANT. Logically, the CEO has a complete control (?!) on the grid of salary of his employees.

=# GRANT SELECT, INSERT, UPDATE, DELETE ON employee_data TO ceo;
GRANT

A normal employee can have information access to all the information, and can update as well his/her phone number or even his/her name:

=# GRANT SELECT (id, employee, phone_number, salary)
   ON employee_data TO public;
GRANT
=# GRANT UPDATE (employee, phone_number) ON employee_data TO public;
GRANT

As things stand now though, everybody is able to manipulate other's private data and not only his own. For example Jeanne can update her CEO's name:

=# SET ROLE jeanne;
SET
=> UPDATE employee_data
   SET employee = 'Raise our salaries -- Signed: Jeanne'
   WHERE employee = 'CEO';
UPDATE 1

Row-level security can be used to control with more granularity what are the rows that can be manipulated for a set of circumstances that are defined with a policy. First RLS must be enabled on the given table:

=# ALTER TABLE employee_data ENABLE ROW LEVEL SECURITY;
ALTER TABLE

Note that if there are no policies defined and that RLS is enabled normal users that even have GRANT access to a certain set of operations can do nothing:

=> set role ceo;
SET
=> UPDATE employee_data SET employee = 'I am God' WHERE id = 1;
UPDATE 0

So it is absolutely mandatory to set policies to bring the level of control wanted for a relation if RLS is in the game. First a policy needs to be defined to let the CEO have a complete access on the table (the default being FOR ALL all the operations are authorized this way to the CEO), and luckily Jeanne just got a promotion:

=# CREATE POLICY ceo_policy ON employee_data TO ceo
   USING (true) WITH CHECK (true);
CREATE POLICY
=# SET ROLE ceo;
SET
=> UPDATE employee_data SET salary = 5000 WHERE employee = 'jeanne' ;
UPDATE 1
=> SELECT * FROM employee_data ORDER BY id;
 id | employee | salary | phone_number
----+----------+--------+---------------
  1 | ceo      | 300000 | 080-7777-8888
  2 | jeanne   |   5000 | 090-1111-2222
  3 | bob      |  30000 | 090-2222-3333
(3 rows)

Even with SELECT access allowed through GRANT, Bob and Jeanne cannot view any row so they cannot view even their own information. This can be solved with a new policy (note in this case the clause USING that can be be used to define a boolean expression on which the rows are filtered):

=# CREATE POLICY read_own_data ON employee_data
   FOR SELECT USING (current_user = employee);
CREATE POLICY
=# SET ROLE jeanne;
SET
=> SELECT * FROM employee_data;
 id | employee | salary | phone_number
----+----------+--------+---------------
  2 | jeanne   |   5000 | 090-1111-2222
(1 row)

A user should be able to modify his own information as well, and note now the WITH CHECK clause that can be used to check the validity of a row once it has been manipulated. In this case, the employee name cannot be updated to a value other than the role name (well it was better not to give UPDATE access with GRANT to this column but this would have made an example above invalid...), and the new phone number cannot be NULL (have you though that the ceo can actually set his phone number to NULL, something less flexible with CHECK at relation level):

=# CREATE POLICY modify_own_data ON employee_data
   FOR UPDATE USING (current_user = employee)
   WITH CHECK (employee = current_user AND phone_number IS NOT NULL);
CREATE POLICY
=# SET ROLE jeanne;
SET
=> UDATE employee_data SET id = 10; -- blocked by GRANT
ERROR:  42501: permission denied for relation employee_data
LOCATION:  aclcheck_error, aclchk.c:3371
=> UPDATE employee_data SET phone_number = NULL; -- blocked by policy 
ERROR:  44000: new row violates WITH CHECK OPTION for "employee_data"
DETAIL:  Failing row contains (2, jeanne, 5000, null).
LOCATION:  ExecWithCheckOptions, execMain.c:1684
=> UPDATE employee_data SET phone_number = '1-1000-2000'; -- OK
UPDATE 1

Using this new policy, Jeanne has updated her phone number, and the CEO can check that freely:

=> SET ROLE ceo;
SET
=> SELECT * FROM employee_data ORDER BY id;
 id | employee | salary | phone_number
----+----------+--------+---------------
  1 | ceo      | 300000 | 080-7777-8888
  2 | jeanne   |   5000 | 1-1000-2000
  3 | bob      |  30000 | 090-2222-3333
(3 rows)

So, while GRANT and REVOKE offer control of the actions that can be done on a relation for a set of users vertically (control of columns), RLS offers the possibility to control things horizontally for each record so when using this feature be sure to use both together and wisely.