Why SQLStreams

SQLStreams is designed to be a poor man’s Kafka. It is for smaller projects that want the rich features that a message log provides but don’t want the hassle of setting up and maintaining an unfamiliar message broker.

SQLStreams is not designed for many enterprise use cases. While latency and throughput should, in theory, be sufficient for most use cases at low to medium scale, if you have extreme reliability, throughput, or governance constraints, look elsewhere.

Always weigh your options for current and future success. And give these other messaging platforms some love, as they heavily inspired much of what SQLStreams is:

• Kafka
• NATS
• River

So, with that out of the way, let’s really answer the question “Why SQLStreams?”

Background

Let’s first set some understanding. Queues and logs are different and they each excel at different things.

Queues are conceptually the easier thing to understand. They are your distributed arrays. You push things to them and pop things off of them. That simplicity allows for some pretty amazing features like delay, retry backoff, priority, and many more.

Logs, on the other hand, are append-only, immutable, distributed arrays. You push your data to the log, and even if something reads that data, it stays there. This naturally leads to consequences like a cursor to keep track of where the reader of the log is. These two seemingly simple differences from a queue lead to some interesting outcomes like replay, independent consumers, and history.

It’s a message log

SQLStreams, at its core, is a message log. As such, you get all of the beautiful features a log provides.

If you happen to be unfamiliar with a message log or Kafka in general, let me set the stage.

It’s a cruel world for the uninformed, but fortunately for you, you have SQLStreams.

When you start off using a message log, the emails, activity feed, and promotional campaigns all derive from a single stream, each with its own consumer group. As long as you’ve retained that history, you can use it to add new features or rebuild existing ones as requirements change.

It’s a message log… AND a retry queue

By design, queues and logs solve similar but distinct problem sets, and SQLStreams tries to utilize both where they each naturally shine.

A stream in SQLStreams is both a message log AND a retry queue. When you produce a message onto a stream, it is stored in an append-only message_log_<stream_id> table. From here, you can have many independent consumer groups read and act on that data in real time. However, if an error occurs in your consumer, the message, by default, will be stored in an exception_queue_<stream_id> table and retried until it either succeeds or is marked dead.

The automatic combination of these two gives, in my opinion, the most natural user experience for how a messaging platform should function. Natural is the key word here, though. There are consequences of this design like no automatic circuit breaker and, by default, out-of-order retry processing.

SQLStreams is opinionated and aims to give you a message log without having to build the retry queue yourself.

Postgres my love

How do I love thee? Let me count the ways.

You’re the people pleaser’s database. JSONB queries, RLS, and now a graph database???

Oh, you don’t have that one obscure thing I want? Let me just write an EXTENSION.

Your MVCC design is inspired, even if it forces me to deal with my archnemesis, autovacuum.

…

Oh, you’re still here? Yeah, you get to use Postgres as well.

More

If you want more convincing:

• Transactional produce
• Ordering and concurrency
• Routing bindings
• Message compaction
• Schedules