CTE is a great feature.
We also need to rewrite Q20 extensively. We haven't really tried out best, and we believe we can unroll the three subqueries into outer joins, but it is mind boggling. The WITH clause (Common Table Expression) comes to rescue. CTE allows one to write simple, clear, easy to understand steps in SQL. This is very much like writing query in Pig [1] (or Quel, for old timers), except with a better syntax (and it is standard).
What one need to beware, is that WITH clause is a optimization barrier in PostgreSQL, that is, the planner will optimize each CTE separately and will not consider global optimizations. This is both good and bad. One can use this feature to reduce planner search space and "force" a plan (for example, join ordering). But on the other hand, more than often, the optimizer (or planner) is smarter than the programmer.
Let's look at the following simple example,
create table t as select i, i + 1 as j from generate_series(1, 10000000) i;
create index ti on t(i);
[1] http://pig.apache.org
We also need to rewrite Q20 extensively. We haven't really tried out best, and we believe we can unroll the three subqueries into outer joins, but it is mind boggling. The WITH clause (Common Table Expression) comes to rescue. CTE allows one to write simple, clear, easy to understand steps in SQL. This is very much like writing query in Pig [1] (or Quel, for old timers), except with a better syntax (and it is standard).
What one need to beware, is that WITH clause is a optimization barrier in PostgreSQL, that is, the planner will optimize each CTE separately and will not consider global optimizations. This is both good and bad. One can use this feature to reduce planner search space and "force" a plan (for example, join ordering). But on the other hand, more than often, the optimizer (or planner) is smarter than the programmer.
Let's look at the following simple example,
create table t as select i, i + 1 as j from generate_series(1, 10000000) i;
create index ti on t(i);
ftian=# explain select * from t where i = 10 and j = 100;
QUERY PLAN
-----------------------------------------------------------------------
Bitmap Heap Scan on t (cost=927.50..48029.26 rows=250 width=8)
Recheck Cond: (i = 10)
Filter: (j = 100)
-> Bitmap Index Scan on ti (cost=0.00..927.43 rows=50000 width=0)
Index Cond: (i = 10)
(5 rows)
ftian=# explain with x as (select * from t where j = 100) select * from x where i = 10;
QUERY PLAN
--------------------------------------------------------------
CTE Scan on x (cost=169247.81..169247.83 rows=1 width=8)
Filter: (i = 10)
CTE x
-> Seq Scan on t (cost=0.00..169247.81 rows=1 width=8)
Filter: (j = 100)
(5 rows)
Note that in the second query, planner is not able to utilize the index.
Closely related to WITH clause is view. One can create views one by one to achieve what WITH clauses do, but this is much more intrusive in the sense that it modifies catalog. On the other had, view is not an optimization barrier. For example,
create view vt as select * from t where j = 100;
ftian=# explain select * from vt where i = 10;
QUERY PLAN
------------------------------------------------------------
Index Scan using ti on t (cost=0.43..8.46 rows=1 width=8)
Index Cond: (i = 10)mi
Filter: (j = 100)
(3 rows)
The index ti is used, no problem. But can someone explain why this produces a different plan? It uses Index Scan instead of Bitmap Index Scan. Well, optimizer is really fun.
Frighteningly, it is not that uncommon for people to turn off fsync to get a performance gain out of their PostgreSQL database. While the performance gain is dramatic, it carries the risk that your database could become corrupt. In some cases, this may be OK, but these cases are really rather rare. A more common case is a database where it is OK to lose a little data in the event of a crash. This is where synchronous commit comes in. When synchronous commit is off, the server returns back success immediately to the client, but waits to flush the data to disk for a short period of time. When the data is ultimately flushed it is still properly sync to disk so there is no chance of data corruption. The only risk if the event of a crash is that you may lose some transactions. The default setting for this window is 200ms.
New activities include:
November 7th 2014 was the