Monitoring Postgresql with a Background Worker
pgantenna and pgsampler comprise an experimental Postgreqsql monitoring framework. This post explores how they work, and what problems they aim to solve.
Framework Overview
pgsampler is a background worker which collects data in a Postgresql cluster. It can log this data to CSV files or ship the metrics to a pgantenna instance over a tcp connection.
pgantenna is an application shipped as a Docker image which receives pgsampler data. It provides a web interface for live monitoring, checks for alerting conditions, and allows for psql access to a historical database of cluster metrics.
Motivation
There are a number of high quality monitoring and performance analysis tools for Postgresql. Many of these involve a remote service which connects to Postgresql as a regular client, or an application that parses log files.
The presented framework uses a background worker to ship statistics to a remote service. It aims to solve a grab bag of real or imagined problems discussed below. Of course, this approach presents it’s own problems and is thus best characterized as an experiment.
Live Monitoring
Data is sent from the cluster in a polling loop at second intervals. Different metrics can be tuned to desired sampling rates.
Using Postgres to Monitor Postgres
Dashboard plots and alert conditions are all written directly in SQL. For example, alert conditions are triggered whenever a cron-executed query returns a NULL in the first field in the first record. Plots are rendered with plotpg.
Historical Analysis with SQL
pgantenna provides a containerized remote cluster which stores historical data separate from transactional systems. The history is just a Postgresql database that can be queried with familiar tools.
Easy to Configure
The background worker uses direct access to identify and connect to databases automatically. Security concerns notwithstanding, this allows for very little configuration minutae to get started with comprehensive monitoring.
Close to metal
A background worker lives and dies with postmaster. One of the foundational alerting conditions is the receipt of a heartbeat from the background worker.
Extensible
Because the metrics collector is a background worker, it may prove to be able to collect data that other monitoring approaches could not reach. For example, while developing I considered several approaches to collect a notion of statements/second. I was thwarted by a reluctance to implement executor hooks or divine this information from shared memory data structures due to limited experience with Postgres internals. But is it possible, or a good idea? Maybe.
Trying it Out
This is an experimental prototype so it’s not appropriate for critical clusters.
To get started first launch a pgantenna instance which can be as simple as a single command on a system with Docker installed.
docker run -p 24831:24831 -p 80:80 no0p/pgantennaNext install pgsampler and update postgresql.conf with an entry pointing pgsampler to the pgantenna instace.
pgsampler.output_network_host='localhost'
