Aislan Wendling: Audit Log and Partitioning

Hi all,

For those who have to log high load databases, and worse, need to keep this logs for long time and always available, partitioning is the way.

Of course you can create various log tables, splitting by hand, but it is not efficient.

To solve this demand, I’ve used two very well explained tutorials with some modifications. The first is the Audit Trigger from the official Wiki. The second is the Scaling PostgreSQL Performance Using Table Partitioning from Engine Yard Blog.

In this tutorial I will create a log that records every data change on the tables where the audit trigger is enabled and it will be partitioned by month and year. Feel free to use the original tutorials, as they cover more situations, specially the Engine Yard’s one, that has a solution to store the old data in files after some time.

Creating the Audit Log

Keep in mind that almost all steps here are extracted from the original tutorials, with little modifications to satisfy my demand.

First of all, you will need to create an extension:

CREATE EXTENSION IF NOT EXISTS hstore;

This extension install a new data type, hstore, which is similar to JSON. Documentation here. I advise you to learn about it because it has a different syntax to retrieve the values.

Now, to organize, lets create the log structure in a separated schema and fix some permissions:

CREATE SCHEMA audit;
REVOKE ALL ON SCHEMA audit FROM public;

CREATE TABLE audit.logged_actions (
 event_id BIGINT DEFAULT nextval('audit.logged_actions_event_id_seq'::regclass) NOT NULL,
 schema_name TEXT NOT NULL,
 table_name TEXT NOT NULL,
 session_user_name TEXT,
 action_tstamp_stm TIMESTAMP WITH TIME ZONE NOT NULL,
 action TEXT NOT NULL,
 row_data public.hstore,
 changed_fields public.hstore,
 statement_only BOOLEAN NOT NULL,
 login_user INTEGER,
 transaction BIGINT,
 CONSTRAINT logged_actions_pkey PRIMARY KEY(event_id),
 CONSTRAINT logged_actions_action_check CHECK (action = ANY (ARRAY['I'::text, 'D'::text, 'U'::text]))
) 
WITH (oids = false)
TABLESPACE audit;

CREATE INDEX logged_actions_action_idx ON audit.logged_actions
 USING btree (action COLLATE pg_catalog."default");

CREATE INDEX logged_actions_action_tstamp_tx_stm_idx ON audit.logged_actions
 USING btree (action_tstamp_stm);

CREATE INDEX logged_actions_idx_transaction ON audit.logged_actions
 USING btree (transaction)
 TABLESPACE audit;

CREATE INDEX logged_actions_table_name_idx ON audit.logged_actions
 USING btree (table_name COLLATE pg_catalog."default");

REVOKE ALL ON audit.logged_actions FROM public;

In this example, I’ve changed and omitted some columns from the original, as I will not need it. The biggest change was the inclusion of the field login_user, which is to record our app’s user. If you don’t need to log it, just remove. Now lets create the function:

CREATE OR REPLACE FUNCTION audit.if_modified_func (
)
RETURNS trigger AS
$body$
DECLARE
 audit_row audit.logged_actions;
 include_values boolean;
 log_diffs boolean;
 h_old hstore;
 h_new hstore;
 excluded_cols text[] = ARRAY[]::text[];
BEGIN
 IF TG_WHEN <> 'AFTER' THEN
 RAISE EXCEPTION 'audit.if_modified_func() may only run as an AFTER trigger';
 END IF;

 audit_row = ROW(
 nextval('audit.logged_actions_event_id_seq'), -- event_id
 TG_TABLE_SCHEMA::text, -- schema_name
 TG_TABLE_NAME::text, -- table_name
 session_user::text, -- session_user_name
 statement_timestamp(), -- action_tstamp_stm
 substring(TG_OP,1,1), -- action
 NULL, NULL, -- row_data, changed_fields
 'f', -- statement_only
 function_to_retrieve_user(), -- login_user
 txid_current()
 );

 IF TG_ARGV[1] IS NOT NULL THEN
 excluded_cols = TG_ARGV[1]::text[];
 END IF;
 
--IF (audit_row.session_user_name in ('xxx')) THEN-- RETURN NULL;--ELSE 
 IF (TG_OP = 'UPDATE' AND TG_LEVEL = 'ROW') THEN
 audit_row.row_data = hstore(OLD.*) - excluded_cols;
 audit_row.changed_fields = (hstore(NEW.*) - audit_row.row_data) - excluded_cols;
 IF audit_row.changed_fields = hstore('') THEN
 -- All changed fields are ignored. Skip this update.
 RETURN NULL;
 END IF;
 ELSIF (TG_OP = 'DELETE' AND TG_LEVEL = 'ROW') THEN
 audit_row.row_data = hstore(OLD.*) - excluded_cols;
 ELSIF (TG_OP = 'INSERT' AND TG_LEVEL = 'ROW') THEN
 audit_row.row_data = hstore(NEW.*) - excluded_cols;
 ELSIF (TG_LEVEL = 'STATEMENT' AND TG_OP IN ('INSERT','UPDATE','DELETE')) THEN
 audit_row.statement_only = 't';
 ELSE
 RAISE EXCEPTION '[audit.if_modified_func] - Trigger func added as trigger for unhandled case: %, %',TG_OP, TG_LEVEL;
 RETURN NULL;
 END IF;
 INSERT INTO audit.logged_actions VALUES (audit_row.*);
 RETURN NULL;
--END IF; -- Uncomment the red code if you want to log all db users
 
END;
$body$
LANGUAGE 'plpgsql'
VOLATILE
CALLED ON NULL INPUT
SECURITY DEFINER
COST 100
SET search_path = pg_catalog, public;

I’ve written as a comment an example of how to not log one or more db users. If you have a special user that has a lot of processes, most probably your log will increase with useless data, so it is a way to not log it. There are more three functions to create:

CREATE OR REPLACE FUNCTION audit.audit_table (
 target_table pg_catalog.regclass
)
RETURNS void AS
$body$
SELECT audit.audit_table($1, BOOLEAN 't', BOOLEAN 't');
$body$
LANGUAGE 'sql'
VOLATILE
CALLED ON NULL INPUT
SECURITY INVOKER
COST 100;

CREATE OR REPLACE FUNCTION audit.audit_table (
 target_table pg_catalog.regclass,
 audit_rows boolean,
 audit_query_text boolean
)
RETURNS void AS
$body$
SELECT audit.audit_table($1, $2, $3, ARRAY[]::text[]);
$body$
LANGUAGE 'sql'
VOLATILE
CALLED ON NULL INPUT
SECURITY INVOKER
COST 100;

CREATE OR REPLACE FUNCTION audit.audit_table (
 target_table pg_catalog.regclass,
 audit_rows boolean,
 audit_query_text boolean,
 ignored_cols text []
)
RETURNS void AS
$body$
DECLARE
 stm_targets text = 'INSERT OR UPDATE OR DELETE';
 _q_txt text;
 _ignored_cols_snip text = '';
BEGIN
 EXECUTE 'DROP TRIGGER IF EXISTS audit_trigger_row ON ' || target_table;
 EXECUTE 'DROP TRIGGER IF EXISTS audit_trigger_stm ON ' || target_table;

 IF audit_rows THEN
 IF array_length(ignored_cols,1) > 0 THEN
 _ignored_cols_snip = ', ' || quote_literal(ignored_cols);
 END IF;
 _q_txt = 'CREATE TRIGGER audit_trigger_row AFTER INSERT OR UPDATE OR DELETE ON ' || 
 target_table || 
 ' FOR EACH ROW EXECUTE PROCEDURE audit.if_modified_func(' ||
 quote_literal(audit_query_text) || _ignored_cols_snip || ');';
 RAISE NOTICE '%',_q_txt;
 EXECUTE _q_txt;
 stm_targets = 'TRUNCATE';
 ELSE
 END IF;

 _q_txt = 'CREATE TRIGGER audit_trigger_stm AFTER ' || stm_targets || ' ON ' ||
 target_table ||
 ' FOR EACH STATEMENT EXECUTE PROCEDURE audit.if_modified_func('||
 quote_literal(audit_query_text) || ');';
 RAISE NOTICE '%',_q_txt;
 EXECUTE _q_txt;

END;
$body$
LANGUAGE 'plpgsql'
VOLATILE
CALLED ON NULL INPUT
SECURITY INVOKER
COST 100;

Now, lets test it:

SELECT audit.audit_table('schemaname.tablename');

And to disable the trigger:

drop trigger audit_trigger_row on schemaname.tablename;
drop trigger audit_trigger_stm on schemaname.tablename;

Partitioning the Table

Now, we can create the function who will redirect the new rows to the right table. As I’m doing this to log transactions, the date will be the statement’s CURRENT_TIMESTAMP.

As I did on the log function, the partition function was modified too. I did a lot of changes from the original code to adapt on more situations, but you can use the original as you please, of course.

CREATE OR REPLACE FUNCTION
audit.log_partition_function()
RETURNS TRIGGER AS 
$BODY$
DECLARE

originaltable text;
schemaname text;
newdata timestamp;
tablename text;
startdate text;
enddate text;

BEGIN

originaltable := 'logged_actions';
schemaname := 'audit';

newdata := NEW.action_tstamp_stm;

-- Verify date to record on the right table
startdate := to_char(newdata,'MM')||'_'||to_char(newdata,'YYYY');
tablename := 'logged_actions_'||startdate;

-- Verify if the table exists
PERFORM 1
FROM pg_catalog.pg_class c
JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind = 'r'
AND c.relname = tablename
AND n.nspname = schemaname;

-- If not, creates it
IF NOT FOUND THEN
enddate := to_char((newdata + INTERVAL '1 month'),'MM')||'_'||to_char((newdata + INTERVAL '1 month'),'YYYY');
EXECUTE 'CREATE TABLE '||schemaname||'.'||tablename||'
(CHECK ( '||quote_literal(newdata)||' != ' ||quote_literal(enddate)|| '))
 INHERITS ('||schemaname||'.'||originaltable||')';
EXECUTE 'ALTER TABLE '||schemaname||'.' || tablename ||' OWNER TO postgres';
-- Table permissions are not inherited from the parent.
-- If permissions change on the master be sure to change them on the child also.
EXECUTE 'ALTER TABLE '||schemaname||'.' || tablename ||' ADD PRIMARY KEY (event_id)';
EXECUTE 'GRANT ALL ON TABLE '||schemaname||'.'||tablename||' TO dba';

-- Indexes are defined per child, so we assign a default index that uses the partition columns
EXECUTE 'CREATE INDEX '||tablename||'_indx1' || ' ON '||schemaname||'.'||tablename||' USING btree (action COLLATE pg_catalog."default") TABLESPACE audit';
EXECUTE 'CREATE INDEX '||tablename||'_indx2' || ' ON '||schemaname||'.'||tablename||' USING btree (action_tstamp_stm) TABLESPACE audit';
EXECUTE 'CREATE INDEX '||tablename||'_indx3' || ' ON '||schemaname||'.'||tablename||' USING btree (transaction) TABLESPACE audit';
EXECUTE 'CREATE INDEX '||tablename||'_indx4' || ' ON '||schemaname||'.'||tablename||' USING btree (table_name COLLATE pg_catalog."default") TABLESPACE audit';
END IF;

-- Insert the current record into the correct partition, which we are sure will now exist.
EXECUTE 'INSERT INTO '||schemaname||'.'||tablename||' VALUES ($1.*)' USING NEW;
RETURN NULL;
END;
$BODY$
LANGUAGE plpgsql;

CREATE TRIGGER log_partition
BEFORE INSERT ON audit.logged_actions
FOR EACH ROW EXECUTE PROCEDURE audit.log_partition_function();

Now you have a log that auto splits by month, which is very useful for maintenance and in the case of purges, you can just truncate a child table.