Jagadeesh Panuganti: Migration Management and schema version control in PostgreSQL

Version control of any code is essential for every organization. All organizations use their own preferred tools like git, svn, perforce etc. As I was working on a new requirement, which is about porting the current CI/CD pipeline to PostgreSQL, I needed to look for a tool which will take care of the schema changes. In this use case, the customer is maintaining all the schema in a separate directory, and creating the objects/schema during the application bootstrap process. This is the behavior most of the applications do follow, but having a dedicated schema change management system is something unique. In this article, we shall discuss about managing version control schema in PostgreSQL using node-pg-migrate.

There are a good number of open source tools for the schema change management requirements. As a NodeJS developer, I was looking for a tool written in NodeJS for schema version control in PostgreSQL. Fortunately, we found a tool called node-pg-migrate which is written in NodeJS. This tool also follows it's own style of defining the SQL statements. The SQL statements can be managed by a set of predefined javascript functions. By using these functions, we can easily perform the create/alter/drop operations for the SQL objects.

Okay, now let us understand how to perform Schema version control or migration management for PostgreSQL in detail.

Installing node-pg-migrate

We need the latest NodeJS, npm to be installed on the server to install node-pg-migrate. Following commands can be used to perform the installation.

# Installing latest NodeJS
curl -sL https://rpm.nodesource.com/setup_14.x | sudo bash -
sudo yum -y install nodejs

# Installing node-pg-migrate with dependent pg library 
sudo npm install -g node-pg-migrate pg

Creating the First Release

To understand this tool better, let us start with a first release which has only one table in it. You can use the following commands to create the first release.

# Creating a directory for all the SQL release files
$ mkdir releases
$ cd releases/

$ node-pg-migrate create first_release
Created migration -- /home/jagadeesh/releases/migrations/1634906885247_first-release.js

As you see in the above output, this tool actually created a file 1634906885247_first-release.js for this release. We have to edit this file with the list of DDL changes that we are planning to deploy with this release. Now, let us see the contents of this file.

$ cat migrations/1634906885247_first-release.js
/* eslint-disable camelcase */

exports.shorthands = undefined;
exports.up = pgm => {};
exports.down = pgm => {};

In this file, we need to focus on 2 exported objects : up, down that can be used for the following use cases.

up      -> List of SQL statements which have to deploy
down    -> List of SQL statements which will revert this deployment

Now, let us write the first deployment as to deploy only one table and also write the rollback behavior to drop that table. This means, the up behavior is CREATE and the down behavior is DROP.

Find the following up, down object definitions, which we have updated to the file : /home/jagadeesh/releases/migrations/1634906885247_first-release.js

exports.up = (pgm) => {
  pgm.createTable("test", {
    id: "id",
    name: { type: "varchar(1)", notNull: true },
    createdAt: {
      type: "timestamp",
      notNull: true,
      default: pgm.func("current_timestamp"),
    },
  });
};

exports.down = (pgm) => {
  pgm.dropTable("test");
};

In the above file content, we have not used any standard SQL syntax, rather we used node-pg-migrate's functions to create/drop the objects. Also, the JSON style of declaring the table definition is really cool. In the above example, we defined the PostgreSQL table as JSON object with multiple columns : id, name, createdAt and defined the column's specific properties like data type, default value and not null.

And in the down object, we used node-pg-migrate specific function dropTable, dropping the table test to enable the capability of a rollback.

Performing the First Deployment

To perform the deployment and use up command for the node-pg-migrate as below.

$ export DATABASE_URL=postgres:postgres@localhost:5432/postgres
$ node-pg-migrate up
> Migrating files:
> - 1634906885247_first-release
### MIGRATION 1634906885247_first-release (UP) ###
CREATE TABLE "test" (
  "id" serial PRIMARY KEY,
  "name" varchar(1) NOT NULL,
  "createdAt" timestamp DEFAULT current_timestamp NOT NULL
);
INSERT INTO "public"."pgmigrations" (name, run_on) VALUES ('1634906885247_first-release', NOW());

Migrations complete!

From the above results, as you see that the JSON is deserialized as SQL statement, and created the table at the mentioned database location.

Creating the Second Release

As part of the second release, we wanted to add another column to the test table. Following are the steps we followed for performing the same.

# Creating second release
$ node-pg-migrate create second_release
Created migration -- /home/jagadeesh/releases/migrations/1634916250633_second-release.js

# Adding second release changes
$ cat migrations/1634916250633_second-release.js
/* eslint-disable camelcase */

exports.shorthands = undefined;

exports.up = pgm => {
pgm.addColumns('test', {
    another_column: { type: 'text', notNull: true },
 })
};

exports.down = pgm => {
        pgm.dropColumns('test', ['another_column']);
};

Performing the Second Deployment

Now, let us execute the deployment and apply changes to the test table by using the up command.

$ export DATABASE_URL=postgres:postgres@localhost:5432/postgres
$ node-pg-migrate up
> Migrating files:
> - 1634916250633_second-release
### MIGRATION 1634916250633_second-release (UP) ###
ALTER TABLE "test"
  ADD "another_column" text NOT NULL;
INSERT INTO "public"."pgmigrations" (name, run_on) VALUES ('1634916250633_second-release', NOW());

Migrations complete!

Performing a Rollback of the Second Deployment

Let us assume that we decided to rollback the second release. In our example, the rollback of the second release involves reverting the test table changes. By using the node-pg-migrate down command, we can revert the last release changes as seen in the following block.

$ export DATABASE_URL=postgres:postgres@localhost:5432/postgres
$ node-pg-migrate down
> Migrating files:
> - 1634916250633_second-release
### MIGRATION 1634916250633_second-release (DOWN) ###
ALTER TABLE "test"
  DROP "another_column";
DELETE FROM "public"."pgmigrations" WHERE name='1634916250633_second-release';

Migrations complete!

From the above results, we could noticed that the column another_column added as part of the second release has been dropped from the database.

Performing a Rollback of the First Deployment

Assume that you wished to revert the changes done in the first release too. To perform this action, we simple do node-pg-migrate down which will revert the first deployment changes as well.

$ export DATABASE_URL=postgres:postgres@localhost:5432/postgres
$ node-pg-migrate down
> Migrating files:
> - 1634908948307_first-release
### MIGRATION 1634908948307_first-release (DOWN) ###
DROP TABLE "test";
DELETE FROM "public"."pgmigrations" WHERE name='1634908948307_first-release';

Migrations complete!

Whenever we do any operation up and down, then node-pg-migrate will be tracking the release points. Whenever the consecutive up or down happen, then it will follow from the last release point and will execute the release scripts accordingly.

Async Releases

As you have seen so far, the node-pg-migrate is a NodeJS tool, where up and down is calling some set of pre-defined javascript functions. We can also leverage this behavior by calling Promise functions in the up and down sections of the release file.

This async release behavior gives additional flexibility by allowing us to perform releases of all the PostgreSQL related deployments in parallel with other service related changes. For more details about the usage of async releases, please refer to the official documentation of thenode-pg-migrate.

Conclusion

As part of the CI/CD requirement, we explored multiple SQL code change management tools. I personally found that node-pg-migrate is developer friendly and a good asset for the CI/CD pipeline. Thanks to it's extensibility, where we can extend it's behavior by writing our own javascript functions. This tool provides many pre-defined PostgreSQL related functions enabling smoother deployments. As of now, I found this tool very reliable and it is doing it's job as expected.

Migrating to PostgreSQL ?

If you are looking to migrate to PostgreSQL or looking for support in managing PostgreSQL databases, please contact us or fill the following form.

The post Migration Management and schema version control in PostgreSQL appeared first on MigOps.