Performance and tuning
Meilisearch's own indexing is usually the bottleneck: the service itself uses about a third of one CPU core while feeding tens of thousands of documents per second. Most tuning is about batching.
Measured throughput
The numbers come from the load test in test/bench, which runs the real service in-process against real Meilisearch and PostgreSQL containers. They are end to end: a row changes, and the document is confirmed and searchable in Meilisearch.
| Setup | |
|---|---|
| Machine | 12-core Apple M-series, containers under OrbStack |
| Source | PostgreSQL 17 (postgres:17-alpine): pgoutput replication slot, keyset-paged snapshot copy |
| Destination | Meilisearch (getmeili/meilisearch), single node, gzip on |
| Dataset | products: 10 columns (bigint id, text name/description/category, numeric price, boolean, text[] tags, jsonb meta, timestamptz, int version) |
| Task settings | batchSize 10,000, maxInflight 8, flushInterval 200ms |
Ranges are the lowest and highest result across repeated runs.
| Workload | Rows | Throughput |
|---|---|---|
| PostgreSQL snapshot (pgoutput slot + keyset copy) | 200,000 | 83k–92k docs/s |
| PostgreSQL streaming (500-row transactions) | 50,000 | 39k–46k docs/s |
| In-memory source snapshot | 200,000 | 64k–73k docs/s |
| In-memory source streaming | 50,000 | 26k–29k docs/s |
| Writer pipeline alone (fake Meilisearch) | — | ~2.5M docs/s |
| Footprint | Value |
|---|---|
| Idle process RSS, dashboard included | ~47 MiB |
| Binary size | ~55 MiB |
Tuning knobs
| Knob | Default | Effect |
|---|---|---|
sync.batchSize | 1000 | Larger batches mean fewer Meilisearch tasks and better indexing throughput, at the cost of more memory per in-flight batch. 5,000–10,000 suits snapshots. |
sync.maxInflight | 4 | Batches waiting for Meilisearch confirmation. Meilisearch auto-batches consecutive tasks, so 8 helps bulk loads. |
sync.flushInterval | 1s | Upper bound on how long a change waits in the buffer. Lower it for lower latency on quiet tables. |
index.writeMode | replace | upsert (partial update) sends only the fields present, for sources that omit unchanged columns. |
Destination gzip | on | Saves bandwidth to remote instances; turn it off on loopback or a LAN to save CPU. |
GOMEMLIMIT | unset | Set it to about 80% of the container memory limit, for example GOMEMLIMIT=400MiB. The garbage collector then works harder near the limit instead of the process being killed by the OOM killer. |
GOGC | 100 | Raising it (for example 200) trades memory for less GC CPU on busy instances. |
Sizing
- Memory scales with
batchSize × maxInflight × average document sizeper running task, plus about 1 MB of logs and metrics history per task. Snapshot reads are chunked (5,000 rows for the SQL sources), and back pressure stops reading from the source while the writer is saturated. - Exceptions: the Kafka and Redis snapshots fold the whole topic or stream in memory, and a diff keeps one hash per document while it runs.
- CPU is mostly garbage collection of short-lived row maps plus network I/O; a fraction of a core per busy task is typical.
- Disk:
cdc.dbholds definitions, checkpoints, dead letters, reports, the alert history and up to 20,000 audit entries; it is usually small. - One process per data directory. Scale up (bigger batches, more in-flight batches) rather than out; for isolation, run several installations with their own data directories.
What profiling changed
Profiles were taken with go test -tags bench -cpuprofile/-memprofile during the load test:
- Documents are encoded to NDJSON by a hand-written encoder and sent with the SDK's NDJSON calls: encoding is 3.8× faster with 23× less memory and 13× fewer allocations; the writer's CPU dropped 45% and its allocations 58% end to end.
- Encode buffers are pooled and pre-sized from the previous batch; retries resend the same bytes.
- The transform copies nested maps and slices only when something inside needs converting: 27% fewer bytes and 3 fewer allocations per row.
- argon2id key derivation uses OWASP's 19 MiB / 2-pass profile instead of 64 MiB: 45 MiB less allocated at startup.
- In total, process CPU dropped 23% (2.26 s to 1.74 s for 250k documents) in the end-to-end test.
Profile-guided optimisation
cmd/meilisearch-cdc/default.pgo merges the CPU profiles of both load tests and go build uses it automatically. In A/B runs it stayed within run-to-run noise (the service is I/O- and GC-bound); it is kept because it costs nothing and becomes useful once refreshed from production traffic:
CDC_PPROF=true ./meilisearch-cdc
curl -H "Authorization: Bearer $TOKEN" -o prod.pprof "http://host:8080/debug/pprof/profile?seconds=60"
make pgo PROFILES="prod.pprof" # merges into cmd/meilisearch-cdc/default.pgoReproduce
make bench # micro benchmarks (transform, batching, encoder, writer)
make load # end-to-end load tests against containers; prints docs/s and writes profiles
BENCH_ROWS=1000000 make load