A little while ago I whipped up a quick buildfarm module to run basic pg_upgrade testing using the builtin "make check" target for pg_upgrade, or its MSVC equivalent. There's only one problem with this: it doesn't check what pg_upgrade is really for, which is upgrading from different versions. And I am seeing pg_upgrade used more and more, so testing cross-version upgrade is really quite important, and not having it is a serious hole in our testing regime.
Our standard testing framework isn't really able to cope with cross-version testing. But the buildfarm knows all about handling different versions. Almost all buildfarm animals regularly build and test multiple versions of postgres. The only trouble is that they throw away all their work product at the end of each run.
So some time ago I started creating a framework that would keep the work product from the latest run on each branch, and then a week or so ago I started reworking that into a module that would both do that and use it to test upgrading to different versions. There is still some work to go, but it's basically working, using logic similar to the builtin pg_upgrade test. Here is sample output from a run on my test animal this morning:
The complete module will be published in an upcoming buildfarm client release. The downside is that to use it you need to have a bit of extra disk space to spare - probably a couple of gigabytes or more to be on the safe side.
The module is limited to testing upgrades from PostgreSQL 9.0 and up. The reason for this is the extra_float_digits mess (see pg_upgrade's TESTING document for details).
One problem is that we check for success by comparing the diff size between pre-upgrade and post-upgrade dumps. At the moment the expected sizes (numbers of lines) are hard coded. That means that a change in the code that works could well lead to a break in the buildfarm, which is rather against our principles, so I'm thinking of publishing the expected diff sizes on the buildfarm server, just as we do for the list of branches of interest, and having the module fetch it. That poses a few problems, especially for the Mingw environment, but I think it can be solved.
Our standard testing framework isn't really able to cope with cross-version testing. But the buildfarm knows all about handling different versions. Almost all buildfarm animals regularly build and test multiple versions of postgres. The only trouble is that they throw away all their work product at the end of each run.
So some time ago I started creating a framework that would keep the work product from the latest run on each branch, and then a week or so ago I started reworking that into a module that would both do that and use it to test upgrading to different versions. There is still some work to go, but it's basically working, using logic similar to the builtin pg_upgrade test. Here is sample output from a run on my test animal this morning:
Tue Nov 27 10:33:01 2012: buildfarm run for crake:HEAD starting [10:33:01] checking out source ... [10:33:01] checking out FileTextArrayFDW checkout complete [10:33:01] checking if build run needed ... [10:33:01] creating vpath build dir pgsql.31025 ... [10:33:01] copying source to ...file_text_array_fdw.31025 [10:33:01] running configure ... [10:33:20] running make ... [10:34:29] running make check ... [10:34:55] running make contrib ... [10:35:02] running make install ... [10:35:08] running make contrib install ... [10:35:09] building FileTextArrayFDW [10:35:10] installing FileTextArrayFDW [10:35:10] setting up db cluster (C)... [10:35:13] starting db (C)... [10:35:14] running make installcheck (C)... [10:35:37] saving files for cross-version upgrade check [10:35:42] checking upgrade from REL9_0_STABLE to HEAD ... ***SUCCESS! [10:36:06] checking upgrade from REL9_1_STABLE to HEAD ... ***SUCCESS! [10:36:31] checking upgrade from REL9_2_STABLE to HEAD ... ***SUCCESS! [10:37:07] install-checking FileTextArrayFDW [10:37:08] stopping db (C)... [10:37:09] OK
The complete module will be published in an upcoming buildfarm client release. The downside is that to use it you need to have a bit of extra disk space to spare - probably a couple of gigabytes or more to be on the safe side.
The module is limited to testing upgrades from PostgreSQL 9.0 and up. The reason for this is the extra_float_digits mess (see pg_upgrade's TESTING document for details).
One problem is that we check for success by comparing the diff size between pre-upgrade and post-upgrade dumps. At the moment the expected sizes (numbers of lines) are hard coded. That means that a change in the code that works could well lead to a break in the buildfarm, which is rather against our principles, so I'm thinking of publishing the expected diff sizes on the buildfarm server, just as we do for the list of branches of interest, and having the module fetch it. That poses a few problems, especially for the Mingw environment, but I think it can be solved.