Worker liveness alert (SQL0096)

worker_liveness

Reports worker rows without a live instance on a stream, including workers owned by its consumer groups. The alert belongs to the stream.

Definition

Property	Value
Code	`SQL0096`
Severity	`info`
Owner	Stream.
Evidence	Retained `sqlstreams.stream.workers.unclaimed` measurements and their worker details.
Unhealthy condition	At least one unclaimed worker.
Default activation	Consecutive unhealthy samples spanning `2m`.
Default evaluation schedule	`@every 1m`.
Active reminders	Every `4h` by default while the condition remains active.

Activation logs at INFO, which the default WARN-level logger hides. The alert is still published to __system.alerts and available through the API and CLI. Its routing key uses the info severity segment.

Read the latest report

Latest(ctx context.Context) (*Alert, error)

Read the latest retained report for the payments.requested stream using an initialized client and the example payload. Cancelling ctx or reaching its deadline interrupts the database reads.

payments := client.Stream[PaymentRequestedV1]("payments.requested")
report, err := payments.Alerts().WorkerLiveness().Latest(ctx)
if err != nil {
  return err
}
if report == nil {
  return errors.New("no retained worker liveness report for payments.requested")
}
fmt.Println(report.Status, report.At)

Condition	Return value
A retained report exists	`(report, nil)`. Its status can be active or resolved.
Stream exists but no report is retained	`(nil, nil)`.
Stream is absent or the read fails	`(nil, err)`. See Errors.

Configuration

Set SystemConfig.WorkerLivenessAlert when registering the system. nil and unset fields use the defaults below. These settings control checks for all streams in that system.

WorkerLivenessAlertConfig

Field	Type	Default	Description
`ScheduleExpression`	`string`	`@every 1m`	Schedule for evaluating the alert. Must be a valid supported schedule expression.
`PendingDuration`	`time.Duration`	`2m`	Required span of consecutive unhealthy samples. Positive after defaults.
`MaximumGap`	`time.Duration`	`2m`	Maximum permitted gap between samples and maximum age of the newest sample. Positive after defaults.
`DisablePending`	`bool`	`false`	Allows activation from fresh unhealthy evidence without waiting for the pending span.

Queued checks retain the policy they were produced with.

The active reminder interval belongs to the alert worker’s RepeatInterval setting and is not a field of WorkerLivenessAlertConfig.

Evaluation and resolution

Evidence	Result
Consecutive unhealthy samples cover `PendingDuration`, with acceptable gaps and age	Activate the alert.
Fresh unhealthy evidence with `DisablePending` enabled	Activate without waiting for the pending span.
Fresh healthy evidence	Resolve an active alert.
Missing, stale, or invalid evidence	Do not change the recorded alert state.

For the payments.requested stream, samples can show a worker owned by charge-cards without a live instance. With the defaults, those unhealthy observations must span at least two minutes, with no gap or newest-sample age greater than two minutes. Waiting two minutes without new samples does not satisfy the requirement.

Evidence must meet these conditions:

The measurement has the expected metric name, gauge kind, and worker unit.
The count is a finite, nonnegative integer.
Worker details account for the count.
Each listed worker has a name, belongs to the evaluated stream, and has a nonzero target instance count.

Reported fields

Reports use the shared Alert fields. The owner identifies the stream, with group_id: 0.

The message gives each affected resource one sentence listing its missing components: message consumer, retry consumer, consumer manager, or the named maintenance workers. The consumer manager starts and restarts the group’s workers.

For example, when the charge-cards group’s message consumer, retry consumer, and consumer manager are stopped:

alert_message="Consumer group \"charge-cards\" has no running message consumer, retry consumer, or consumer manager."
hint="Start a consumer for each affected group, or delete any group you no longer need."

Maintenance reports suggest sqlstreams manager run. A report containing both missing application consumers and maintenance includes both actions. Deleting a consumer group removes its saved progress. Use that action only when the group is no longer needed.

Detail is empty. The activation log omits it. Worker names and instance targets remain in Data.

Data

Field	Type	Description
`unclaimed_count`	integer	Number of unclaimed worker rows in the finding.
`workers`	array	Corresponding worker details from the same stored sample.

Active reports include this evidence. Resolved reports have no Data value.

Each workers entry has these fields:

Field	Type	Description
`worker`	string	Worker name.
`owner`	string	Stream or consumer-group name owning the worker.
`owner_kind`	string	Kind of resource that owns the worker.
`target_instances`	integer	Declared instance target for the worker.

Active and resolved reports share the same alert identity. A later resolved report replaces the active report as the latest retained value.

CLI

sqlstreams alert latest worker_liveness --stream payments.requested

Prints the latest report and exits with status 0 when one is retained. No retained report prints an absence message and exits with status 1. See alert commands for output formats and flags.

Errors

Error	When
`ErrStreamNotFound`	`Latest` is called for a stream that is not registered.
`ErrNotRegistered`	System alert storage is not registered.
Database error	The retained alert cannot be read.
Context cancellation or deadline error	The database operation is interrupted by its context.

Worker is not running — Inspect worker ownership, suspension, and failures.
Background managers and workers — How managers start workers and replace expired claims.
Stream metrics — Read the unclaimed-worker measurement used by this alert.