SQL polling

When log-based capture is not available (no replication privileges, a managed database that hides its log, SQLite), the sqlpoll source re-reads a table on an interval and picks up rows whose cursor column moved forward.

Requirements

A driver: pgx (PostgreSQL; postgres and postgresql are aliases), mysql (MySQL and MariaDB; alias mariadb) or sqlite (alias sqlite3).
A primary key, used to page through rows: table … has no primary key; sqlpoll needs one to page through rows otherwise.
A cursor column that grows with every insert and update: an updated_at timestamp maintained by the application or a trigger, or a version number bumped on every write.
SELECT on the table. No other privilege is needed.

Example for PostgreSQL, keeping updated_at current with a trigger:

ALTER TABLE products ADD COLUMN updated_at timestamptz NOT NULL DEFAULT now();
CREATE INDEX ON products (updated_at, id);

CREATE FUNCTION touch_updated_at() RETURNS trigger AS $$
BEGIN NEW.updated_at := clock_timestamp(); RETURN NEW; END $$ LANGUAGE plpgsql;
CREATE TRIGGER products_touch BEFORE UPDATE ON products
  FOR EACH ROW EXECUTE FUNCTION touch_updated_at();

For MySQL: updated_at TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6). An index on the cursor column (followed by the key) keeps each poll cheap.

Connection settings

Driver	DSN example
`pgx`	`postgres://reader:secret@db.internal:5432/shop`
`mysql`	`reader:secret@tcp(db.internal:3306)/shop` (`parseTime` is enabled automatically)
`sqlite`	`file:/var/data/app.sqlite`

Task fields

Field	Default	Meaning
`table`	required	The table to poll.
`schema` (API only)	driver default	Schema of the table.
`cursorColumn`	required	The column that grows with every write. The wizard ranks candidates (`updated_at`, `modified_at`, `last_modified`, `changed_at`, `version`, `row_version`, `rowversion`, then timestamps and integers).
`pollInterval`	`2s`	How often to look for changes. The wizard offers 500 ms to 1 minute; any duration works through the API.

How it works

Snapshot. Keyset pages of 1,000 rows by primary key.
Polling. Every interval, the source reads rows whose (cursor, primary key) is greater than the last one delivered, 1,000 at a time, and keeps reading until a page comes back short. Each row becomes an upsert.
Deletes need soft deletes. A hard DELETE is invisible: the row simply stops being returned. Mark rows deleted instead (a deleted flag or a deleted_at column that also bumps the cursor) and set the task's soft delete column, so those rows delete their documents.

Limitations

Rows whose cursor is NULL are never streamed (the snapshot still copies them).
A write that commits with a cursor lower than one already polled is missed: for example a long transaction that stamped updated_at at its start, or sequence values allocated out of order. Prefer a cursor assigned at commit time, or keep transactions short compared with the poll interval. A periodic diff and repair catches anything missed.
On SQLite prefer an integer cursor: time values are bound as text and compare lexically against however the column was written.
No truncate detection.

Positions and resume

A position is JSON with the cursor and primary key of the last row delivered, for example {"cursor":"2026-10-26T10:00:00.123456Z","pk":[42]}. A null cursor means "from the beginning" (the table was empty when the position was taken). A position that cannot be decoded is treated as lost and the task resyncs.