Postgres

This component buffers & batches data as shown in the diagram above. You’ll notice that Vector treats these concepts differently, instead of treating them as global concepts, Vector treats them as sink specific concepts. This isolates sinks, ensuring services disruptions are contained and delivery guarantees are honored.

Batches are flushed when 1 of 2 conditions are met:

1. The batch age meets or exceeds the configured timeout_secs.
2. The batch size meets or exceeds the configured max_bytes or max_events.

Buffers are controlled via the buffer.* options.

In order to insert data into a PostgreSQL table, you must first create a table that matches the json serialization of your event data. Note that this sink accepts log, metric, and trace events and the inserting behavior will be the same for all of them.

For example, if your event is a log whose JSON serialization would have the following structure:

{
    "host": "localhost",
    "message": "239.215.85.26 - AmbientTech [04/Mar/2025:15:09:25 +0100] "DELETE /observability/metrics/production HTTP/1.0" 300 37142",
    "service": "vector",
    "source_type": "demo_logs",
    "timestamp": "2025-03-04T14:09:25.883572054Z"
}

And you want to store all those fields, the table should be created as follows:

CREATE TABLE logs (
	host TEXT,
	message TEXT,
	service TEXT,
	source_type TEXT,
	timestamp TIMESTAMPTZ
);

Note that not all fields must be declared in the table, only the ones you want to store. If a field is not present in the table but it is present in the event, it will be ignored.

When inserting the event into the table, PostgreSQL will do a best-effort job of converting the JSON serialized event to the correct PostgreSQL data types. The semantics of the insertion will follow the jsonb_populate_record function of PostgresSQL, see PostgreSQL documentation about that function for more details about the inserting behavior. The correspondence between Vector types and PostgreSQL types can be found in the sqlx crate’s documentation

Practical example

Spin up a PostgreSQL instance with Docker:

docker run -d --name postgres -e POSTGRES_PASSWORD=password123 -p 5432:5432 postgres

Create the following PostgreSQL table inside the test database:

CREATE TABLE logs (
	message TEXT,
	payload JSONB,
	timestamp TIMESTAMPTZ
);

And the following Vector configuration:

sources:
  demo_logs:
    type: demo_logs
    format: apache_common
transforms:
  payload:
    type: remap
    inputs:
      - demo_logs
    source: |
      .payload = .      
sinks:
  postgres:
    type: postgres
    inputs:
      - payload
    endpoint: postgres://postgres:password123@localhost/test
    table: logs

Then, you can see those log events ingested in the logs table.

Composite Types

When using PostgreSQL composite types, the sink will attempt to insert the event data into the composite type, following its structure.

Using the previous example, if you want to store the payload column as a composite type instead of JSONB, you should create the following composite type:

CREATE TYPE payload_type AS (
	host TEXT,
	message TEXT,
	service TEXT,
	source_type TEXT,
	timestamp TIMESTAMPTZ
);

And the table should be created as follows:

CREATE TABLE logs (
	message TEXT,
	payload payload_type,
	timestamp TIMESTAMPTZ
);

Then, you can see those log events ingested in the logs table and the payload column can be treated as a regular PostgreSQL composite type.

Ingesting metrics

When ingesting metrics, the sink will behave exactly the same as when ingesting logs. You must declare the table with the same fields as the JSON serialization of the metric event.

For example, in order to ingest Vector’s internal events, and only take into account counter, gauge, and aggregated_histogram metric data, you should create the following table:

create table metrics(
	name TEXT,
    namespace TEXT,
	tags JSONB,
 	timestamp TIMESTAMPTZ,
	kind TEXT,
	counter JSONB,
	gauge JSONB,
 	aggregated_histogram JSONB
);

And with this Vector configuration:

sources:
  internal_metrics:
    type: internal_metrics
sinks:
  postgres:
    type: postgres
    inputs:
      - internal_metrics
    endpoint: postgres://postgres:password123@localhost/test
    table: metrics

You can see those metric events ingested into the metrics table.