Consumer groups

Producers write messages to a stream. Consumers read those messages and run your application’s processing logic. An application may run several consumer instances to share the work, while other applications need to process the same stream independently. SQLStreams needs to track which work each application has completed across all of its instances.

A consumer group holds that shared progress. Instances in the same group divide the work. Different groups keep their own progress through the stream.

Because progress belongs to the group, an instance can stop without losing the record of completed work. If completion was never recorded, another instance can process the message again.

Instances in a group share work

Suppose the payments.requested stream contains four messages, with ids 1–4. A payment application runs two consumer instances, A and B. Both use the charge-cards group on that stream.

Each instance claims work from the group. If A claims ids 1–2, B can claim ids 3–4. A claim gives the instance a time-limited lease on its claimed range of message ids. While the lease is valid, another instance in the group cannot claim the same range.

An additional instance shares the charge-cards group’s work. It does not get its own pass through every message.

The group is the shared identity stored in Postgres. An instance is a running consumer that claims and processes work under that identity.

Different groups keep independent progress

Now another application needs to audit the payments. It uses a second group, audit-payments, on the same stream. For this example, both groups start before id 1 and read all four messages without routing filters.

The charge-cards group can finish ids 1–4 while audit-payments has only finished ids 1–2. The audit group still has its own work to do on ids 3–4. It does not share the other group’s claims or completion state.

Finishing a message in the charge-cards group does not delete it from the stream or mark it complete for the audit-payments group.

Reading old messages still depends on retention: a group can only read messages that remain in the stream.

Progress survives an instance

If instance A stops, the charge-cards group remains in Postgres. Starting another instance under that name uses the existing group’s progress. Work whose completion was recorded does not become new work just because a process restarted.

Registering the group again preserves its cursor. Its configured starting position applies when the group is created. Changing that setting does not rewind an existing group. A new group name creates independent progress with its own starting position.

A restart can still repeat work whose completion was not recorded.

A message can run again

Consider message id 1, a request to charge a card. The payment provider records the charge. SQLStreams separately records the message as complete for the charge-cards group in Postgres.

A crash can occur between those two writes.

1. Instance A claims ids 1–2 for the charge-cards group and calls the payment provider for id 1. The provider accepts the charge.
2. The consumer handler returns nil, reporting success to the consumer. Before SQLStreams records completion in Postgres, A crashes. The charge has already happened.
3. A’s lease on message ids 1–2 eventually expires. On a later claim, instance B can reclaim the unresolved work, including id 1.
4. B runs the consumer handler for id 1 again. Without application-level deduplication, it can charge the card again.

For this payment, both attempts need the same payment-provider idempotency key, such as the payment’s business id. The provider can then recognize the second request as the same operation.

Limiting retries alone cannot tell the application whether the first charge happened.

Related topics

• Consumer leases — How long claimed work can wait and run.
• Retries and failed messages — How returned errors lead to retries or permanent failure.
• Make message processing safe to repeat — Prevent a repeated message from applying a database change twice.