At FOSS4G this year, I wanted to take a run at the decision process around open source with particular reference to the decision to adopt PostGIS: what do managers need to know before they can get comfortable with the idea of making the move.

The Manager's Guide to PostGIS — Paul Ramsey from FOSS4G on Vimeo.

Preface

The Problems

The Example Module and Repository

NULL handling

How PGObject Serializes

How and When PGObject Deserializes

Use Cases

Caveats

Today, I presented on “Postgres in Amazon RDS” topic at Postgres Open Conference in Chicago. Here is the slide deck:

GCE: 670.00
16 CPUS
60G Memory
2500GB HD space

GCE: 763.08
16 CPUS
104G Memory
2500GB HD space

Amazon: 911.88
16 CPUS
30G Memory
3000GB HD Space

Amazon: 1534.00
r3.4xlarge
16 CPUS
122.0 Memory
SSD 1 x 320
3000GB HD Space

Amazon: 1679.00
c3.8xlarge
32 CPUS
60.0 Memory
SSD 2 x 320
3000GB HD Space

None of the above include egress bandwidth charges. Ingress is free.

Softlayer: ~815 (with 72GB memory ~ 950)
16 Cores
RAID 10
4TB (4 2TB drives)
48GB Memory

Softlayer: ~1035 (with 72GB memory ~ 1150)
16 Cores
RAID 10
3TB (6 1TB drives, I also looked at 8-750GB and the price was the same. Lastly I also looked at using 2TB drives but the cost is all about the same)
48GB Memory

If I had to name one thing that surprised me the most back when I started messing with C and PostgreSQL, I'd probably name memory contexts. I never met this concept before, so it seemd rather strange, and there's not much documentation introducing it. I recently read an interesting paper summarizing architecture of a database system (by Hellerstein, Stonebraker and Hamilton), and there's actually devote a whole section (7.2 Memory Allocator) to memory contexts (aka allocators). The section explicitly mentions PostgreSQL as having a fairly sophisticated allocator, but sadly it's very short (only ~2 pages) and describes only the general ideas, without going discussing the code and challenges - which is understandable, because the are many possible implementations. BTW the paper is very nice, definitely recommend reading it.

But this blog is a good place to present details of the PostgreSQL memory contexts, including the issues you'll face when using them. If you're a seasoned PostgreSQL hacker, chances are you know all of this (feel free to point out any inaccuracies), but if you're just starting hacking PostgreSQL in C, this blog post might be useful for you.

Now, when I said there's not much documentation about memory contexts, I was lying a bit. The are plenty of comments in memutils.h and aset.c, explaining the internals quite well - but who reads code comments, right? Also, you can only read them when you realize how important memory contexts are (and find the appropriate files). Another issue is that the comments only explain "how it works" and not some of the consequences (like, palloc overhead, for example).

Motivation

But, why do we even need memory contexts? In C, you simply call malloc whenever you need to allocate memory on heap, and when you're done with the memory, you call free. It's simple and for short programs this is pretty sufficient and manageable, but as the program gets more complex (passing allocated pieces between functions) it becomes really difficult to track all those little pieces of memory. Memory allocated at one place may be passed around and then freed at a completely different part of the code, far far away from the malloc that allocated it. If you free them too early, the application will eventually see garbage, if you free them too late (or never), you get excessive memory usage (or memory leaks).

And PostgreSQL is quite complex code - consider for example how tuples flow throught execution plans. The tuple is allocated at one place, gets passed through sorting, aggregations, various transformations etc. and eventually sent to the client.

Memory contexts are a clever way to deal with this - instead of tracking each little piece of memory separately, each piece is registered somewhere (in a context), and then the whole context is released at once. All you have to choose the memory context, and call palloc/pfree instead of malloc/free.

In the simplest case palloc simply determines the "current" memory context (more on this later), allocates appropriate piece of memory (by calling malloc) and associates is with the current memory context (by storing a pointer in the memory context and some info in a "header" of the allocated piece) and returns it to the caller. Freeing the memory is done either through pfree (which reverses palloc logic) or by freeing the whole memory context (you can see it as pfree loop over all allocated pieces).

This offers multiple optimization options - for example reducing the number of malloc/free calls by keeping a cache of released pieces, etc.

The other thing is granularity and organization of memory contexts. We certainly don't want a single huge memory contexts, because that's almost exactly the same as having no contexts at all. So we know we need multiple contexts, but how many? Luckily, there's a quite natural way to split memory contexts, because all queries are evaluated through execution plans - a tree of operators (scans, joins, aggregations, ...).

Most executor nodes have their own memory context, released once that particular node completes. So for example when you have a join or aggregation, once this step finishes (and passes all the results to the downstream operator), it discards the context and frees the memory it allocated (and didn't free explicitly). Sometimes this is not perfectly accurate (e.g. some nodes create multiple separate memory contexts), but you get the idea.

The link to execution plans also gives us hint on how to organize the memory context - the execution plan is a tree of nodes, and with memory contexts attached to nodes, it's natural to keep the memory contexts organized in a tree too.

That being said, it's worth mentioning that memory contexts are not used only when executing queries - pretty much everything in PostgreSQL is allocated within some a memory context, including "global" structures like various caches, global structures etc. That however does not contradict the tree-ish structure and per-node granularity.

MemoryContext API

The first thing you should probably get familiar with is MemoryContextMethods API which provides generic infrastructure for various possible implementations. It more or less captures the ideas outlined above. The memory context itself is defined as a simple structure:

typedefstructMemoryContextData{NodeTagtype;MemoryContextMethods*methods;MemoryContextparent;MemoryContextfirstchild;MemoryContextnextchild;char*name;boolisReset;}MemoryContextData;

Which allows the tree structure of memory contexts (by parent and first/next child fields). The methods describe what "operations" are available for a context:

typedefstructMemoryContextMethods{void*(*alloc)(MemoryContextcontext,Sizesize);/* call this free_p in case someone #define's free() */void(*free_p)(MemoryContextcontext,void*pointer);void*(*realloc)(MemoryContextcontext,void*pointer,Sizesize);void(*init)(MemoryContextcontext);void(*reset)(MemoryContextcontext);void(*delete_context)(MemoryContextcontext);Size(*get_chunk_space)(MemoryContextcontext,void*pointer);bool(*is_empty)(MemoryContextcontext);void(*stats)(MemoryContextcontext,intlevel);#ifdefMEMORY_CONTEXT_CHECKINGvoid(*check)(MemoryContextcontext);#endif}MemoryContextMethods;

Which pretty much says that each memory context implementation provides methods to allocate, free and reallocate memory (alternatives to malloc, free a realloc) and also methods to manage the contexts (e.g. initialize a new context, destroy it etc.).

There are also several helper methods wrapping this API, forwarding the calls to the proper instance of MemoryContextMethods.

And when I mentioned palloc and pfree before - these are pretty much just additional wrappers on top of these helper methods (grabbing the current context and passing it into the method).

Allocation Set (AllocSet) Allocator

Clearly, the MemoryContext API provides just the infrastructure, and was developer in anticipation of multiple allocators with different features. That however newer happened, and so far there's a single memory context implementation - Allocation set.

This often makes the discussion a bit confusing, because people mix the general concept of memory contexts and the (single) implementation available.

Allocation Set implementation is quite sophisticated (aka complex). Let me quote the first comment in aset.c:

... it manages allocations in a block pool by itself, combining many small allocations in a few bigger blocks. AllocSetFree() normally doesn't free() memory really. It just add's the free'd area to some list for later reuse by AllocSetAlloc(). All memory blocks are free()'d at once on AllocSetReset(), which happens when the memory context gets destroyed.

To explain this a bit - AllocSet allocates blocks of memory (multiples of 1kB), and then "splits" this memory into smaller chunks, to satisfy the actual palloc requests. When you free a chunk (by calling pfree), it can't immediately pass it to free because the memory was allocated as a part of a larger block. So it keeps the chunk for reuse (for similarly-sized palloc requests), which has the nice benefit of lowering the number of malloc calls (and generally malloc-related book-keeping).

This works perfectly once you have palloc calls with a mix of different requests sizes, but once you break this, the results are pretty bad. Similarly, it's possible to construct requests that interact with the logic grouping requests into groups (making it easier to reuse the chunks), resulting in a lot of wasted memory.

There's another optimization for requests over 8kB, that are handled differently - the largest blocks (part of the block pool) are 8kB, and all requests exceeding this are allocated through malloc directly, and freed immediately using free.

The CurrentMemoryContext

Now, let's say you call palloc, which looks almost exactly the same as a malloc call:

char*x=palloc(128);// allocate 128B in the context

So how does it know which memory context to use? It's really simple - the memory context implementation defines a few global variables, tracking interesting memory contexts, and one of them is CurrentMemoryContext which means "we're currently allocating memory in this context."

Earlier I mentioned that each execution node has an associated context - the first thing the memory node may do is setting the associated memory context as the current one. This however is a problem, because the child nodes may do the same, and the execution may be "interleaved" (the nodes are passing tuples in an iterative manner).

Thus what we usually see is this idiom:

MemoryContextoldcontext=MemoryContextSwitchTo(nodecontext);char*x=palloc(128);char*y=palloc(256);MemoryContextSwitchTo(oldcontext)

which keeps the current memory context set to the original value.

Summary

I tried to explain the motivation and basic of memory contexts, and hopefully direct you to the proper source files for more info.

The main points to remember are probably:

• Memory contexts group allocated pieces of memory, making it easier to manage lifecycle.
• Memory contexts are organized in a tree, roughly matching the execution plans.
• There's a generic infrastructure allowing different implementations, but nowadays there's a single implementation - Allocation Set.
• It attempts to minimize malloc calls/book-keeping, maximize memory reuse, and never really frees memory.

In the next post I'll look into the usual problems with palloc overhead.

The Portal Project hosted at PSU is near & dear to our hearts here at PDXPUG. (It’s backed by an almost 3TB Postgres database.) We’ve had several talks about this project over the years:

eXtreme Database Makeover (Episode 2): PORTAL– Kristin Tufte
Metro Simulation Database– Jim Cser
R and Postgres– Chris Monsere (I think this is where we first heard about bus bunching)
Extreme Database Makeover – Portal Edition– William van Hevelingin

Kristin Tufte most recently spoke at the PDXPUG PgDay about current development on this project, which is now in its 10th year. Future plans include data from more rural locations, more detailed bus stats, and possibly a new bikeshare program. We look forward to hearing more about it!

• schema_name, table_name, index_name: the obvious
• index_cols: a comman-delimited list of index columns
• indexdef: a CREATE statement for how the index was created, per pg_indexes view
• index_scans: the number of scans on this index per pg_stat_user_indexes

A specially extended feature frenzy for FOSS4G 2014 in Portland. Usually I only frenzy for 25 minutes at a time, but they gave me an hour long session!

PostGIS Feature Frenzy — Paul Ramsey from FOSS4G on Vimeo.

Thanks to the organizers for giving me the big room and big slot!

Tonight I presented a talk on using JSON in Postgres at the Toronto Postgres users group. Pivotal hosted the talk at their lovely downtown Toronto office. Turnout was good with a little over 15 people attending (not including the construction workers banging against some nearby windows).

I talked about the JSON and JSONB datatypes in Postgres and some idea for appropriate uses of NoSQL features in a SQL database like Postgres.

My slides are available for download

We are thinking of having lighting and ignite talks for the next meetup. If anyone is in the Toronto area and wants to give a short (5 minute) talk on a Postgres related topic let me know.

Last week I explained (or attempted to) the basics of memory contexts in PostgreSQL. It was mostly about the motivation behind the concept of memory contexts, and some basic overview of how it all works together in PostgreSQL.

I planned to write a follow-up post about various "gotchas" related to memory contexts, but as I was writing that post I ended up explaining more and more details about internals of AllocSet (the only memory context implementation in PostgreSQL). So I've decided to split that post into two parts - first one (that you're reading) explains the internals of allocation sets. The next post will finally deal with the gotchas.

The level of detail should be sufficient for understanding the main principles (and some tricks) used in the AllocSet allocator. I won't explain all the subtle details - if you're interested in that (and the code is quite readable, if you understand the purpose), please consult the actual code in aset.c. Actually, it might be useful to keep that file opened and read the related functions as I explain what palloc and pfree do (at the end of this post).

blocks

The introductory post mentioned that Allocation Sets are based on "block pools" - that means the allocator requests memory in large blocks (using malloc), and then slices these blocks into smaller pieces to handle palloc requests.

The block sizes are somewhat configurable, but by default the allocator starts with 8kB blocks and every time it needs another block it doubles the size up to 8MB (see the constants at the end of memutils.h). So it first allocates 8kB block, then 16kB, 32kB, 64kB, ... 8MB (and then keeps allocating 8MB blocks). You may change different sizes, but the minimum allowed block size is 1kB, and the block sizes are always 2^N bytes.

To make the management easier, each block is decorated with a header, represented by this structure defined in aset.c:

typedefstructAllocBlockData{AllocSetaset;/* aset that owns this block */AllocBlocknext;/* next block in aset's blocks list */char*freeptr;/* start of free space in this block */char*endptr;/* end of space in this block */}AllocBlockData;

Each block references the memory context it's part of, has a pointer to the previous block (to make a linked list of allocated blocks) and pointers delimiting the free space in the block. The header is stored at the beginning of the block - the AllocBlockData is ~32B if I count correctly, so this overhead is negligible even for the smallest block size (~3%).

Chunks

So the allocator acquires memory in blocks, but how does it provide it to the palloc callers? That's what chunks are about ...

Chunks are pieces of memory allocated within the blocks in response to palloc calls. Just like blocks, each chunk is decorated with a header (this is slightly simplified, compared to the actual structure definition in aset.c):

typedefstructAllocChunkData{void*aset;/* the owning aset */Sizesize;/* usable space in the chunk */}AllocChunkData;

Each chunk mentions which memory context it belongs to (just like blocks), and how much space is available in it.

Just like block sizes, the chunk sizes are not entirely arbitrary - sure, you may request arbitrary amount of memory (using palloc), but the amount of memory reserved in the block will be always 2^N bytes (plus the AllocChunk header, which is usually 16B on 64-bit platforms).

For example, if you request 24 bytes (by calling palloc(24)), the memory context will actually create a 32B chunk (because 32 is the nearest power of 2 after 24), and hand it back to you. So you will (rather unknowingly) 32B of usable space. The smallest allowed request is 8B, so all smaller requests get 8B.

Chunk overhead

Chunks are the main source of overhead in memory contexts.

Firstly, because of the 2^N sizes, there'll usually be some unused space. The bottom 1/2 of the chunk is always utilized (otherwise a smaller chunk would be used), and the upper half is 50% utilized on average, assuming a random distribution of requested sizes. Thus ~75% utilization and 25% overhead on average. That's not negligible, but it's probably a fair price for the features memory contexts provide (and compare it to overhead in garbage-collected languages, for example).

Secondly, there's the chunk header. A chunk with 32B usable space will actually occupy 48B. The impact of this really depends on the request size. For large requests it gets negligible (similarly to the block header), but as you can imagine for very small requests it gets very high (e.g. 200% for 8B, which is the smallest allowed request).

Chunk reuse

But why are the chunk sizes treated like this? The answer is "reuse." When you call pfree, the chunk is not actually returned to the system (using free). It actually can't be, because it's only a small part of the block (which is what we got from malloc), and there may be more chunks on it. So we can't just free the chunk or block.

Instead, the chunk is moved to a "freelist" - a list of empty chunks, and may be used for future palloc calls (requesting chunk with the same size). There's one freelist for each chunk size, so that we can easily get chunk of the "right" size (if there's one). Without using these "unified" chunk sizes, this would not be really possible - the size matching would be more expensive or less efficient.

Still, there are ways this can fail tremendously (e.g. when the assumtion of random distribution of sizes does not hold, or when the reuse does not work for some reason). More on these failures in the next post.

Oversized chunks

If there's a maximum block size, what happens to requests exceeding this limit? The current implementation creates a special one-chunk block for each such oversized chunk, with almost exactly the right size (no 2^N rules, just some basic alignment tweaks). The purpose of the block pool is to prevent overhead with allocating too many tiny pieces, so allocating large blocks directly is not a big deal (before you allocate enough of them to notice the overhead, you'll likely run out of memory).

More precisely - oversized chunks are not chunks exceeding max block size, but chunks exceeding 8kB (because the number of freelists is limited) or exceeding 1/8 of max block size (so that a stream of such chunks results in 1/8 overhead).

Anyway, the thing you should remember is that large chunks are special-cased - are allocated in dedicated blocks with the right size, and pfree immediately frees them.

Now, let's see what the two main methods - palloc and pfree do.

palloc (AllocSetAlloc)

Assuming the CurrentMemoryContext is set to an instance of allocation set allocator (and it's difficult to break this assumption, because there are no other implementations), the flow after calling palloc (see mcxt.c for implementation) looks about like this:

palloc(size)->CurrentMemoryContext->alloc(CurrentMemoryContext,size)->AllocSetAlloc(CurrentMemoryContext,size)

That is, the palloc looks at CurrentMemoryContext, and invokes the alloc method (which is part of MemoryContextMethods API, described in the previous post). As it's an AllocSet instance, it invokes AllocSetAlloc which then does all the work with block and chunks.

In short, it does about this:

• Checks whether the size is exceeding the "oversized chunk" limit - in that case, a dedicated block (of just the right size) is allocated.
• Checks whether there's an existing chunk in the freelist with the proper size, and if yes then uses it.
• Checks whether there already is a block, and if there's enough space in it (that's what the block header is for). A new block is allocated (using malloc) if necessary.
• Reserves sufficient space in the block for the chunk, and returns it to the caller.

There are a few more tricks (e.g. what would you do with the remaining space in a block?) - the code is easy to read and quite well commented, althouth rather long. Not sure what happened to the "Every function should fit on a screen" recommendation - either they had extremely large screens, or small font size ;-)

pfree (AllocSetFree)

The pfree does not need to consult the CurrentMemoryContext - and it can't, because the chunk could have been allocated in a completely different context. But hey, that's what the chunk header is for - there's a pointer to the proper AllocSet instance. So the flow looks about this:

pfree(chunk)header=chunk-16B->header->aset->free(header->aset,header)->AllocSetFree(header->aset,header)

And AllocSetFree doesn't do much:

• Check if this is "oversized" chunk (those can and should be freed immediately).
• If it's a regular chunk, move it to a freelist, so that it can be reused by subsequent palloc calls.

Summary

• There's a single memory context implementation - Allocation Set.
• It requests memory in "large" blocks, with configurable min/max sizes. The smallest block size is 1kB, the sizes follow 2^N rule.
• The blocks are kept until the whole context is destroyed.
• The blocks are then carved into chunks, used to satisfy palloc requests.
• Both chunks and blocks have a header linking them to the memory context.
• Chunks sizes follow the 2^N rule too, palloc requests are satisfied using the smallest sufficient chunk size.
• Chunks are not freed, but moved to a freelist for reuse by suceeding palloc calls.
• Large requests (over 8kB or 1/8 of max block size) are handled using dedicated single-chunk blocks.

• Use the -j multiprocess option for pg_restore (and, on 9.3, for pg_dump as well).  Ideal concurrency is generally two less than the number of cores you have, up to a limit of 8.  Users with many ( > 1000) tables will benefit from even higher levels of concurrency.
• Doing a compressed pg_dump, copying it (with speed options), and restoring on the remote server is usually faster than piping output unless you have a very fast network.
• If you're using binary replication, it's faster to disable it while restoring a large database, and then reclone the replicas from the new database.  Assuming there aren't other databases on the system in replication, of course.
• You should set some postgresql.conf options for fast restore.

• The below assumes that the restored database will be the only database running on the target system; they are not safe settings for production databases.
• It assumes that if the pg_restore fails you're going to delete the target database and start over.
• These settings will break replication as well as PITR backup.
• These settings will require a restart of PostgreSQL to get to production settings afterwards.

• you want to set maintenance_work_mem as high as possible, up to 2GB, for building new indexes.  However, since we're doing concurrent restore, you don't want to get carried away; your limit should be (RAM/(2*concurrency)), in order to maintain somewhat of an FS buffer.  This is a reason why you might turn concurrency down, if you have only a few large tables in the database.
• checkpoint_segments should be set high, but requires available disk space, at the rate of 1GB per 32 segments.  This is in addition to the space you need for the database.

At FOSS4G we gave two presentations. The videos from other presentations are FOSS4G 2014 album. I have to commend the organizers for gathering such a rich collection of videos. Most of the videos turned out very well and are also downloadable in MP4 in various resolutions. It really was a treat being able to watch all I missed. I think there are still some videos that will be uploaded soon. As mentioned lots of PostGIS/PostgreSQL talks (or at least people using PostGIS/PostgreSQL in GIS).

I’ve always worked on PgJDBC, the JDBC Type 4 driver for PostgreSQL, with just a terminal, ant and vim. I recently had occasion to do some PgJDBC debugging work on Windows specifics so I set up Eclipse to avoid having to work on the Windows command prompt.

As the process isn’t completely obvious, here’s how to set up Eclipse Luna to work with the PgJDBC sources.

If you don’t have it already, download JDK 1.8 and Eclipse Luna.

Now download JDK 1.6 from Oracle (or install the appropriate OpenJDK). You’ll need an older JDK because the JDK doesn’t offer a way to mask out new classes and interfaces from the standard library when compiling for backward compatibility – so you may unwittingly use Java 7 or Java 8-only classes unless you target the Java 6 VM.

(If you want to work with the JDBC3 driver you need Java 5 instead, but that’s getting seriously obsolete now).

Now it’s time to import the sources and configure Eclipse so it knows how to work with them:

1. Register JDK 1.6 with Eclipse:
   • Window -> Preferences, Java, Installed JREs, Add…
   • Set JRE Name to “JDK 1.6″
   • Set JRE Home to the install directory of the JDK – not the contained JRE. It’ll be a directory named something like jdk1.6.0_45 and it’ll have a jre subdirectory. Select the jdk1.6.0_45 directory not the jre subdir.
   • Finish…

   and OK the preferences dialog.

2. Import from git and create a project:
   • File -> Import, Git, Projects from Git, then follow the prompts for the repo
   • After the clone completes, Eclipse will prompt you to “Select a wizard for importing projects from git”. Choose Import as a General Project
   • Leave the project name as pgjdbc and hit Finish
3. Configure ant
   • Get properties on the “pgjdbc” project
   • Go to the Builders tab
   • Press New…
   • In the resulting dialog’s Main tab set the Buildfile to the build.xml in the root of the PgJDBC checkout and set the base directory to the checkout root, e.g. buildfile = ${workspace_loc:/pgjdbc/build.xml} and base directory = ${workspace_loc:/pgjdbc}
   • In the Classpath tab, Add… the maven-ant-tasks jar, lib/maven-ant-tasks-2.1.3.jar from the PgJDBC lib dir
   • In the JRE tab, choose Separate JRE and then select the JDK 1.6 entry you created earlier
4. Build the project – Project -> Build Project

Done! You can now modify and debug PgJDBC from within Eclipse, including setting breakpoints in your PgJDBC project and having them trap when debugging another project that’s using the same jar.

It isn’t completely automagic like with a native Eclipse project, though. To debug the driver from another project, you’ll need to add the driver from the jars/ directory produced by the build to your other project’s build path. It’ll be named something like postgresql-9.4-1200.jdbc4.jar. Add it under the project’s Properties, in Java Build Path -> Libraries -> Add External JARs.

Once you’ve added the driver JAR you can use your modified JAR, but breakpoints you set in the driver sources won’t get tripped when debugging another project. To enable that, tab open the newly added driver JAR in the Libraries tab and edit “Source attachment…”. Choose “Workspace Location” and, when prompted, your PgJDBC driver project.

The project sources are now linked. You can set breakpoints either by exploring the PgJDBC jar from the project that uses PgJDBC to find the package and source file required, or by opening the source file in the PgJDBC project. Either way it’ll work.

You’ll probably also want to set the loglevel=2 parameter to PgJDBC so that it emits trace logging information.

Many Thanks

The Talks

Not Just for DBAs

Milwaukee's Best

Postgres People