SQLStreams

the messaging platform that is just Postgres

You last visited on 9999-99-99 Show what's new since then

0779 — The idle fleet's cost is the manager tick, and its losing claims convoy on the shared system rows

Edit this page
Posted: 2026-09-12 · Report this thread
brandon Site Admin brandon profile Posts: 677

Context

ROADMAP asked for an idle-fleet benchmark before any fix, then a rung on the ladder: row poll_rate; idle backoff in the tick runner; LISTEN/NOTIFY. The idle-fleet-16/160/1600 family under .bench declares one group per stream and produces nothing; the observer samples pg_stat_statements per statement shape (it drops the leading owner comment) and the checker prints each shape’s calls/s and share of statement time.

Read on 2026-09-12 (8-core Postgres 18.4, ten-minute holds):

streamsrowsreplicasstatements/sPostgres coreslosing claim
1612611,1340.2099/s, 0.02 ms
1612632,7630.31259/s, 0.02 ms
16099019,1250.62814/s, 0.01 ms
160990324,1531.652,275/s, 4.1 ms, 92% of statement time
16009,63017,3568.1, saturated213/s, 31 ms; fleet never fully registered
16009,63033,6497.7, saturated115/s, 800 ms; fleet never fully registered

Every Consume session holds an uncapped group-scoped manager row that ticks every second: a chain listing, two sweep DELETEs, and for each capped row on its chain another process holds, a schema assert, SELECT ... FOR UPDATE, INSERT ... SELECT count, and a rollback. The chain carries three system-scoped rows (consumer group janitor, schedule producer, metrics collector), so every group manager re-locks the same three rows each second. Row poll_rate paces only a winner’s work ticks.

Decision

  • Rung 1 is rejected by measurement: it does not reach the term.
  • Rung 2 is the pick: a tick that made no progress backs off toward ten times the row’s poll_rate; any progress snaps it back. A manager tick’s progress is a reconcile change or a swept row. It divides the CPU term by about ten and moves the convoy’s onset from about 480 managers to about 4,800.
  • Beside rung 2, a losing claim stops taking the row lock: it reads target_instances and the live count first, locks only with room.
  • The group manager’s chain drops the three system-scoped rows; the system manager, one winner per deployment, reconciles them. The shared rows the convoy formed on are then claimed by R processes, not R times the groups.
  • Rung 3 is not earned.

Consequences

  • One replica with a thousand rows idles at 0.6 Postgres cores today; ten thousand rows do not start. The three changes are one ROADMAP item.
  • CONVENTIONS “The literal” overstates pg_stat_statements; not corrected.
  • Rejected: rung 1; a knob on the manager row’s poll_rate.

Amended. [0781] supersedes the rung-2 clause: the claimant backs off in the manager’s pool along ManagerConfig.ClaimRetry, and live instances keep their poll rate. The rest stands.