SQLStreams

the messaging platform that is just Postgres

You last visited on 9999-99-99 Show what's new since then
Posted: 2026-09-12 · Report this thread
brandon Site Admin brandon profile Posts: 677

.Metrics() on the system, a stream, or a consumer handle returns that scope’s metrics handle: no I/O, no failure. Metrics split live state from collected history. Snapshot(ctx) asks the source tables what is true now; Latest(ctx) and History(ctx, limit) read observations already written to __system.metrics.

groupMetrics := client.Stream[PaymentRequestedV1]("payments.requested").Consumer("charge-cards").Metrics()
live, err := groupMetrics.Snapshot(ctx) // *sqlstreams.ConsumerGroupSnapshot, the source tables now
if err != nil {
	return err
}
collected, err := groupMetrics.CursorBacklog().Latest(ctx) // *sqlstreams.Measurement, newest retained
if err != nil {
	return err
}
history, err := groupMetrics.CursorBacklog().History(ctx, 20) // newest first

Verbs

On every scope:

verbreturnsnotes
Definitions()[]MetricDefinitionthe built-ins SQLStreams can produce for the scope, ordered by SQL code; in-memory, never Postgres
a built-in selector*MetricHandleno I/O; the stream and group names on the resource handle supply its attributes

On the system handle:

verbreturnsnotes
Latest(ctx)[]*Measurementthe newest retained point for every series ever produced, built-in and user, ordered by name then attributes; nil until the first collection
Metric(name, attributes)*MetricHandlethe string escape hatch: a user-produced series, a built-in selected from a definition, a consumer-session series
Producer()*MetricProducerHandleits Register(ctx, cfg) returns a MetricProducerInstance for your own measurements
Consumer(name)*ConsumerHandle[Measurement]a consumer group on __system.metrics; Bindings takes metric names or patterns such as billing.*

On a stream or group handle:

verbreturnsnotes
Snapshot(ctx)*StreamSnapshot, *ConsumerGroupSnapshotlive, and the existence check: ErrStreamNotFound, ErrConsumerNotFound

On a MetricHandle:

verbreturnsnotes
Latest(ctx)*Measurement(nil, nil) when the exact series has no retained point
History(ctx, limit)[]*Measurementnewest first; limit must be positive; empty for the same absence

The built-in selectors, one per declared metric:

// system levels
systemMetrics.CollectorCompletedTimestamp()
systemMetrics.UnclaimedWorkers()
systemMetrics.OldestUnclaimedAge()
systemMetrics.FailingWorkers()
systemMetrics.OverdueSchedules()
systemMetrics.OldestDueAge()
systemMetrics.SuspendedSchedules()
systemMetrics.ActiveAlerts()
systemMetrics.ResolvedAlerts()

// one built-in check; the alert name becomes the series' `alert` attribute
systemMetrics.CheckStreamsEvaluated("partition_count")
systemMetrics.CheckStreamsFailed("partition_count")
systemMetrics.CheckPublishedAlerts("partition_count")
systemMetrics.CheckResolvedAlerts("partition_count")

// stream level
streamMetrics.Compacted()
streamMetrics.Partitions()
streamMetrics.UnclaimedWorkers()

// consumer-group levels
groupMetrics.CursorHead()
groupMetrics.CursorClaimed()
groupMetrics.CursorCommitted()
groupMetrics.CursorBacklog()
groupMetrics.CursorInflight()
groupMetrics.ReadyExceptions()
groupMetrics.InflightExceptions()
groupMetrics.DeferredExceptions()
groupMetrics.DeadExceptions()
groupMetrics.OldestUnresolvedAge()
groupMetrics.OpenLeases()
groupMetrics.AbandonedRoutinesOutstanding()
groupMetrics.AbandonedRoutinesTotal()
groupMetrics.AbandonedRoutinesSelfClearLatencyAverage()

CLI

sqlstreams metric list --builtin lists retained measurements whose names start with sqlstreams., across system, stream, and consumer scopes. --user selects user-produced measurements; omit both flags to list both.

metric latest <name> reads the latest measurement per matching attribute set; metric history <name> --limit 10 reads each set’s history. Both accept repeatable --attribute key=value filters and --series-limit to bound the number of attribute sets.

These CLI reads select multiple series. The client’s Metric(name, attributes) selects one exact series; CLI attributes are partial filters. For example, sqlstreams metric latest billing.queue_depth shows the latest point for each retained attribute set under that name. Adding --attribute region=us-east keeps sets carrying that pair. JSON keeps a series array in both latest and history, with attributes and measurements per set. series_total reports the count before --series-limit truncation. No matching series gives exists: false, an empty array, and exit 1.

Scopes

A definition is metadata, never a value: Code, Name, Kind, Unit, Description, Scope, AttributeKeys. The scopes and their required attributes are fixed:

scoperequired attributes
MetricScopeSystemnone, except alert-check series require alert
MetricScopeStreamstream
MetricScopeConsumerGroupstream, group
MetricScopeConsumerSessionstream, group, version, session
MetricScopeExporternone; collection-local OTel health, never stored

Built-in observations come from the manager’s metrics collector, every 30 seconds by default with jitter; the rate is SystemConfig.MetricCollector.PollRate (System). Consumer-session series have no typed handle yet and are read through Metric(name, attributes).

CollectorCompletedTimestamp() selects the Unix-second timestamp recorded after every read and measurement write in a collector pass succeeds. Startup and partial passes do not refresh it. No retained value means no retained completion evidence, not a timestamp of zero. The independent collector-progress alert combines completion with retained manager lease coverage (0700).

Your own measurements

producer, err := client.System().Metrics().Producer().Register(ctx, nil)
if err != nil {
	return err
}
measurement, err := sqlstreams.NewMeasurement(
	"billing.queue_depth", sqlstreams.MetricKindGauge, 7, "",
	map[string]string{"region": "us-east"}, time.Now(),
)
if err != nil {
	return err
}
if _, err := producer.Produce(ctx, measurement); err != nil {
	return err
}

The producer derives the routing key from the metric name and the message key from the name plus attributes; compaction keeps the newest value per series. Names starting with sqlstreams. are reserved and rejected. A user series appears in Latest and never in Definitions.

Gotchas

  • Measurement.Metadata is optional JSON attached to the observation. Latest, History, metrics consumers, and CLI JSON expose it. It does not change the series key or become OTel labels. Built-in partition measurements carry compaction status; stream unclaimed-worker measurements carry the named workers.

  • Latest means newest retained, not fresh. The collector writes backlog 27 at 10:03, work finishes, and at 10:07 Snapshot says 19 while Latest still says 27 with Measurement.At at 10:03. A caller with a freshness requirement compares At to its own threshold; the library never turns an old collected value into live state.

  • A collected read proves nothing about the resource: a series can outlive its stream, and a new group can exist before the collector reaches it. Snapshot is the existence check.

  • Consuming __system.metrics is not a guarantee of every intermediate value: compaction applies there too.

  • The optional otel module exports these through OpenTelemetry and Prometheus. Every collection reads the latest retained values; counters export their stored cumulative totals, so reading 120 twice exports 120. Two collection-local gauges, sqlstreams_otel_source_read_success and sqlstreams_otel_measurements_rejected, say whether Postgres was read and how many retained rows were omitted; the warning measurements cannot be exported names the family.

  • Run one export target per installation. Replicas read the same deployment totals, so summing them counts every measurement once per replica.