Suman Michael: Organizing PostgreSQL codebase using templates in Golang

While migrating to PostgreSQL, have you ever wondered whether you could make changes to SQL-only code without triggering a build, re-deployment, or even a restart? Have you ever attempted packaging software as a binary and allowing clients to execute their own compatible queries on any DB engine at no additional cost? In such cases, a better approach to organizing SQL queries in Golang is required. We recently developed an in-house micro-service using golang and PostgreSQL by leveraging the go's template language feature. In the article, we are going to discuss about Organizing PostgreSQL codebase using templates in Golang. For the purpose of this article, some basic utilities are written leveraging Go templates as seen in the MigOps repository : GoTSQL, as a Golang package.

Quick setup of GoTSQL

1. Installation through Go Get command.

   $ go get github.com/migopsrepos/gotsql

2. Initialize the GoTSQL object

   import "github.com/migopsrepos/gotsql"
   ...
   g := gotsql.GoTSQL{}

3. Load the template file or directory with template files. (Templates are discussed in detail in next section)

   g.Load("library/")
   ...
   g.Load("library_dev/")

4. Get the query using namespace and use it in Golang

   query, err := g.Get("library/books/select/getAllBooks", nil)
   ...
   rows, err := db.Query(query, ...)

That's all; with GoTSQL, separating SQL code from the Go source and maintaining it is now simple. Let's take a closer look into GoTSQL Templates in details.

GoTSQL Templates

Templates are text files that are used to generate dynamic content. Go templates and text/template template engine are in use here. Go templates are a great way to modify output in any way you want, whether you're making a web page, sending an email, or it can be even extended to maintain an isolated and modular SQL codebase.

In this blog, we’re going to take a quick look on how to use the Go templates and write GoTSQL files, as well as how to integrate them with an application. We'll use a simple SQL Codebase as an example, which is just a directory with GoTSQL files (Go template files with SQL arranged in a hierarchy. The file structure is as shown below.

- library
   - books
      - select.gotsql
      - stats.gotsql
   - authors
      - select.gotsql
      - filter.gotsql
   - maintenance.gotsql
   - stats.gotsql

Before we learn how to implement it, let’s take a look at the template's syntax. Templates are provided to the appropriate functions either as strings or as “raw string” or in this case a plain text file with extension .gotsql.

Here's a sample .gotsql file which has PostgreSQL template query. You'll learn more as we progress through this blog.

-- FILE: library/authors/select.gotsql
{{ define "getAllAuthors" }}
SELECT * FROM authors 
{{ if .Offset }}OFFSET {{ .Offset }} {{ end }}
{{ if .Limit }}LIMIT {{ .Limit }} {{end}};
{{ end }}

...

{{ define "getAuthor" }}
SELECT * FROM authors WHERE book_id = $1
{{ end }}

...

Actions in templates represent the data evaluations, functions, or control loops. They’re delimited by {{ }}. Other, non-delimited parts are left untouched. By passing a string constant to the define action, you can name the template that is being constructed.

The GoTSQL examples and its corresponding Go snippets mentioned below will assist you in learning how to use it.

Data Evaluations

When you use templates, you usually bind them to a data structure (such as a struct or a map) from which you'll retrieve data.

{{ define "getAllBooksWithOrder" }}

SELECT * FROM books ORDER BY {{ .OrderBy }};

{{ end }}

In the above example, to obtain data from a map, you can use the {{ .OrderBy }} action, which will replace it with the value of keyOrderBy of a given map, on parse time.

query, err := g.Get("library/books/select/getAllBooksWithOrder", map[string]interface{
        "OrderBy": "ISBN",
})

You can also use the {{.}} action to refer to a value of a non-struct types.

Conditions

You can also use if-else statements in templates. For example, you can check if .Offset and .Limit are non-empty, and if they are, append its value to query.

{{ define "getAllAuthors" }}

SELECT * FROM authors 
{{ if .Offset }}OFFSET {{ .Offset }} {{ end }}
{{ if .Limit }}LIMIT {{ .Limit }} {{end}};

{{ end }}

query, err := g.Get("library/authors/select/getAllAuthors", map[string]interface{}{
        "Limit":  "10",
        "Offset": "100",
})

Go templates do support else, else if, conditions and nested statements in actions.

Loops, Functions and Variables

You can loop across a slice using the range action. {{range .Member}} ... {{end}} template is used to define a range action.

The range action creates a variable that is set to the iteration's successive elements. A range action can declare two variables that are separated by a comma. Any variable is prefixed by $.

Here in the below example, let's look at the case of extracting certain columns.. You can use $index to get the index of the slice and $col to get the current iterating element of the Columns slice and implement the logic in the template itself. len and slice are functions. Go also supports custom functions in templates via FuncMap.

Explore more about Pipelines, Variables and Functions in Golang's text/template.

{{ define "getAuthor" }}

{{ $comma_count := len (slice .Columns 1) }}
SELECT
{{ range $index, $col := .Columns}} 
    {{ $col }}{{ if lt $index $comma_count }}, {{ end }} 
{{end}}
FROM authors WHERE author_id = $1

{{ end }}

columns := []string{"firstname", "lastname"}
query, err := g.Get("library/authors/select/getAuthor", map[string]interface{}{
    "Columns": columns,
})
...
rows, err := db.Query(query, authorId)

The general rule of thumb is to treat all user input as untrusted. Hence, It is strongly recommended to use parameterized queries in Golang. database/sqldeals with it effectively and is designed to do so. Here is a quick primer on how to avoid SQL Injection Risk.

This code is meant to generate and maintain the parameterized queries. Hence, it is not advised to parse the user input data using this templates. Notice the WHERE clause, which is parameterized to avoid SQL injection through user input.

To put this into context, we should consider utilizing this tool to produce only parameterized queries to ensure optimum safety.

Why this approach?

SQL is a text-based language that usually belongs in a text file. Storing SQL as text, outside of compiled code, eases maintenance significantly. For applications that rely heavily on databases, separating Go and SQL source code is essential. And your database administrator will appreciate it.

The following are a few of the advantages of this approach:

• Simplified SQL codebase maintenance that isolates code changes and, most crucially, provides a framework for enabling Version Control (Git Integration), multiple SQL dialects, and optimizations with the same Go code.
• Composing queries with a template syntax made easier than writing them explicitly in Go code.
• Having SQL segregated from code makes it much easier for new developers to browse through SQL statements and grasp the database structure. And it allows those who are familiar with SQL to contribute directly to a project without having to learn about all of the other tools involved.
• It reduces the negative consequences of database modifications. For example, If the name of a table or column changes, the fix is typically as simple as a search and replace in a single text file.
• It dramatically decreases development cost by allowing users to rapidly import files from an organized template repository and make nominal changes to integrate them into similar projects.
• If an older application needs to be migrated to a new framework, then the SQL is readily available for porting to the new application.
• If necessary, Changes to SQL can be done after deployment by updating the text-based template file and restarting or refreshing the server (assuming interfaces are the same).
• Separation between programmer and DBA responsibilities. There is no confusion about who is responsible for what. The code is done by developers, while the scripts are written by database developers. This gives DBAs a better feasibility to write SQL with higher performance (faster run-time) than using an ORM tool. If both do a decent job and the database design is correct, the application will outperform all other ORM frameworks.
  
  

Cons, Really?

• The fact that the queries are stored in another file appears to obfuscate it from source code, encouraging the programmer to be ignorant of the database. That could be a useful abstraction or a smart division of labor. This problem may be mitigated if the SQL templates were organized and named properly.
• Some may argue that ORM (Object-Relational-Mapping) should be used instead. ORM frameworks are designed to let you deal with objects and query abstractions. However, if your application is heavily reliant on databases and requires DBA support, ORM won't help you, but it will make life difficult for DBAs. Read this to know why you shouldn't always look up to ORM ?
• Refactoring could be hindered by a wider separation between Go and Database SQL code.
• Because SQL isn't hidden in the compiled code, it's easier for a third party to read and reverse engineer the application, which may or may not be a concern for a commercial company.
• When you rename, delete, or add a field, multiple files can become out of sync. Is there a chance you'll encounter a compilation error? Will you be able to locate all of the areas where you need to make changes? Or Is that anything you have to look for in testing ?
  
  

Conclusion

In this article we showed how to use Go templates and maintain PostgreSQL codebase using GoTSQL. The examples provided in this article can be used as a reference to build your own GoTSQL templates. While we have a lot of advantages with this approach, the challenges related to the SQL code being obfuscated from the source code, may be minimal.

What's next?

GoTSQL's next release will feature native support for registering native golang functions in template actions both globally and at the template level. This will greatly enhance the capability of generating SQL Queries with custom logic. Please watch out for my next article to see how it works.

Migrating to PostgreSQL ?

If you are looking to migrate to PostgreSQL or looking for support in managing PostgreSQL databases, please contact us or fill the following form.

The post Organizing PostgreSQL codebase using templates in Golang appeared first on MigOps.