Declarative tasks file

Keep sync tasks in Git: list them in a YAML or JSON file, point CDC_TASKS_FILE (or -tasks) at it, and the service creates or updates them every time it starts.

How it is applied

• The file is read once at startup, after stored tasks are resumed.
• Before parsing, every ${VAR} (or $VAR) is replaced with the environment variable of that name, so credentials stay out of the file. An unset variable becomes an empty string. Expansion happens on the whole file, so a literal $ written in the file (inside a password, say) is expanded too: keep such values in environment variables.
• Every task needs an id (letters, digits, - and _, at most 64 characters). It is the key that matches a task in the file to a stored task.
• A task with a new id is created; it starts right away when desiredState: running.
• A task whose id exists is updated with the file's definition (a running task restarts). Then its desiredState is applied: running starts it, stopped or paused stops it, and an omitted value leaves it as it is.
• If the source table, schema, database, connector or the index uid or destination changed, the checkpoint is cleared and the next start runs a full snapshot.
• Tasks that are not in the file are left alone; removing a task from the file does not delete it.
• Any error (unreadable file, invalid YAML, a missing id, a validation error) stops the startup: the service logs stopped with error with the reason (for example task products: name is required) and exits with status 1.

Tasks created from the file are ordinary tasks: they appear in the dashboard and can be operated there. Edits made in the dashboard are overwritten by the file at the next start.

Format

The file has a single top-level key, tasks, holding a list of task definitions. The fields are the JSON fields of a task, described on Create a task; YAML and JSON are both accepted (JSON is valid YAML). Durations are strings like 500ms or 2s.

The demo file (deploy/tasks.yaml)

# Declarative tasks, applied at startup (created or updated by id).
# ${VAR} is expanded from the environment, so secrets stay out of this file.
tasks:
  - id: products
    name: Postgres products
    desiredState: running
    source:
      type: postgres
      dsn: ${PG_DSN}
      table: products
    mapping:
      softDeleteColumn: deleted_at
      timeFormat: unix
      geo: {lat: lat, lng: lng}
    index:
      uid: products
      manageSettings: true
      settings:
        searchableAttributes: [name, description, category]
        filterableAttributes: [category, price, in_stock, _geo, updated_at]
        sortableAttributes: [price, updated_at, _geo]
        typoTolerance: {enabled: true}
    sync:
      batchSize: 5000
      flushInterval: 500ms
      maxInflight: 8
      safety:
        maxDeleteRatio: 0.3
        zeroDowntimeResync: true

The Compose demo sets PG_DSN=postgres://shop:shop@postgres:5432/shop?sslmode=disable and CDC_TASKS_FILE=/etc/cdc/tasks.yaml, and mounts the file read-only.

A fuller example

Two tasks: one using an inline connection, one reusing a connector created in the dashboard (its id is shown on the connector page), writing to a named destination.

tasks:
  - id: orders-search
    name: MySQL orders
    description: Orders searchable by customer and status
    tags: [shop, mysql]
    desiredState: running
    source:
      type: mysql
      dsn: ${ORDERS_DSN}          # cdc:secret@tcp(db.internal:3306)/shop
      table: orders
      serverId: 4242              # unique among the server's replicas
    mapping:
      primaryKey: [id]
      exclude: [internal_notes]
      rename: {customer_email: email}
      static: {tenant: acme}
      timeFormat: unix
    index:
      uid: orders
      writeMode: replace
      manageSettings: true
      settings:
        filterableAttributes: [status, tenant, created_at]
        sortableAttributes: [created_at]
    sync:
      mode: snapshot_stream
      batchSize: 2000
      flushInterval: 1s
      maxInflight: 4
      retry: {maxAttempts: 8, initialBackoff: 200ms, maxBackoff: 30s}
      safety:
        maxDeleteRatio: 0.2
        minDocsForDeleteGuard: 1000
        allowTruncate: false
        zeroDowntimeResync: true
        dumpBeforeResync: false

  - id: catalog-eu
    name: Catalog to the EU instance
    desiredState: running
    source:
      connectorId: pg-catalog     # a saved connector
      schema: public
      table: products
    index:
      uid: products
      destinationId: eu           # empty = the default destination

Running with a tasks file

export ORDERS_DSN='cdc:secret@tcp(db.internal:3306)/shop'
meilisearch-cdc -tasks /etc/meilisearch-cdc/tasks.yaml
# or
CDC_TASKS_FILE=/etc/meilisearch-cdc/tasks.yaml meilisearch-cdc

On success the service logs applied tasks file with the path and the number of tasks.

Import and export

Besides the startup file, tasks can be moved between installations as JSON: GET /api/v1/tasks/export returns every task with secrets masked, and POST /api/v1/tasks/import creates the tasks of a JSON array (stopped). Both are on the task list page as well. See Import and export.