Events on the bus, truth in the database

Inside snakken's backend, things that happen are announced as events: a post created, a warning published, a comment worth a push. Other parts of the system subscribe to those facts and react. Before the first line of configuration we set two rules:

• First: the database stays the only source of truth. An event may announce a change, but it will never be the change.
• Second: the bus lives under the same privacy line as our public API. Our servers never see coordinates, only coarse H3 cells the phone computes for itself; a message broker doesn't get to weaken that just because it's internal.

This post is about what those two rules do to an event architecture, and what the whole thing costs us.

The other way to do it: tables and clocks

Our first design for asynchronous work was a database table and a clock. A comment lands, a row goes into a notifications table, and a sweep with a lease picks it up on its next tick — which meant a push for that comment spent real seconds doing nothing, waiting for a timer. Officially published warnings drained through an outbox table the same way. So did event reminders. Three features, three hand-built copies of the same pattern, and every new "when X happens, do Y" cost another poller, another lease, another interval to tune.

The pattern itself was never wrong. A row written in the same transaction as the thing it describes is exactly how you make side effects reliable, and we kept that part — more on the outbox below. What stopped scaling was everything around it: reactions were tick-bound, and the leases pinned all background work to a single runner.

Choosing a broker, and the sizing lesson

For the broker itself we wanted the smallest thing that does the job: self-hosted, running in the EU, persistence and replay built in. A managed cloud queue was never on the table; our sovereignty promise doesn't come with an asterisk for infrastructure. The heavyweight streaming platforms would have worked too, at an operations bill sized for traffic we don't have — and wouldn't for years on even our most optimistic curve.

That claim is checkable, because we did the arithmetic before choosing. We modelled growth from write rates per user per day, up to an ambition we would be lucky to grow into. Even there the peak stays at a modest number of events per second, orders of magnitude below what a small replicated broker persists without noticing. Social apps read far more than they write, and only the writes become events.

That is the lesson worth publishing: for a product like ours, don't size the broker — size everything around it. What changes as a system like this grows is how many services consume the events, how the database is read, and where team boundaries fall; the broker's configuration is the part that stays put. If your event-architecture plan spends most of its pages on the message system, it's probably answering the wrong question.

Commit first, publish after

The dual-write problem is the trap every event architecture falls into once: you commit a row to the database and then publish the event — and one of the two fails. Now the system disagrees with itself, and no retry policy can put it back together, because there is nothing left that knows both halves.

So publishing directly from a request is banned in our codebase, enforced by an architecture test. A state change and its event are written to the same database in the same transaction, the event into an outbox table. A relay reads that table and publishes to the broker, marking rows only after the broker has acknowledged them. The database even hands us the wake-up for free: an insert fires a notification, the relay listens for it, and a slow fallback poll catches anything a dropped connection missed. Notify wakes; poll reads; the table stays the truth.

Delivery is at-least-once, which means duplicates are not an error condition — they are the contract. Every consumer proves idempotency, most simply with a unique key over the event id, a pattern we already trusted: it's how a burst of likes on one post collapses into a single push instead of a dozen.

An envelope that carries almost nothing

Every event travels in the same thin envelope: what kind of fact, when, an opaque token for who, an H3 cell for roughly where — and a payload of almost nothing. For a created post the payload is the post's id and whether it has media. Not the text, not the images, not a username, and never a coordinate, because the servers producing these events don't have coordinates to leak.

Thin payloads buy more than privacy. Streams keep messages for a few days, and the content itself only ever lives in the database. So a GDPR deletion request stays what it has always been for us: a database question with one answer, instead of an archaeology project across every stream that ever mentioned the user.

Why the database keeps the crown

The first rule sounds obvious, but it is a choice, and there is a respectable school of design that makes the opposite one. In event sourcing the log is the source of truth: every change is an event kept forever, and the database is just a cached projection you can rebuild. What that buys is real: a perfect audit trail, and the ability to replay history into shapes you hadn't thought of when you wrote it. For ledgers and anything accountants care about, it is often the right call.

We turned it down for two reasons that are specific to what snakken is. The first is deletion. A log that is the truth has to keep its events essentially forever, so a person's data ends up woven through an immutable history, and honouring an erasure request under the GDPR becomes an exercise in key-shredding ceremonies or log rewrites. Because our streams forget within days and content lives in exactly one place, deleting a person stays a database operation with a result you can check. The second is enforcement. Everything we promise about who can read what is enforced as row-level security inside the database, so a second source of truth would be a second place to get those rules wrong. There is also a third, less noble reason: event sourcing is a permanent tax on schema evolution, because every event version ever written must stay readable, and we are a small team.

So nobody here replays the event log to reconstruct state, ever. Events are integration events — announcements between parts of the system — and the moment one becomes load-bearing for state, the first rule has been broken.

Facts in, facts out

Moderation is the flow that shows the shape. When someone posts, that fact goes onto the bus. The checks subscribe to it, and when a post breaks the rules they publish a fact of their own: the post comes down, and its author gets a notice with the reason, on the same path a moderator's takedown takes.

Warnings give a second example. An official alert, once published for an area, is a fact on the bus like any other, and the delivery that gets it onto phones in the affected cells is simply its subscriber. We wrote about why warnings belong in a neighbourhood feed before; the bus is what spares that feature a bespoke pipeline of its own.

The quiet payoff is what didn't have to change: the code that creates a post or publishes a warning announces its fact and is done. The takedown is a subscriber; the buzzing phone is another; whatever we build next stands beside the write path instead of being wired into the middle of it.

What it costs

We now run a stateful message broker. It is small and boring, and it is still one more thing that pages someone. At-least-once delivery, once chosen, is forever: every consumer we ever write must tolerate duplicates, and that discipline can't be retrofitted onto a sloppy one later. Ordering has its own catch as it is guaranteed per subject, but not globally. So any logic that quietly assumes global order will be wrong in ways that only show up under load.

The latency win is real but honest: reactions that used to wait for the next sweep now happen as the fact arrives. Nice, not existential. The bigger admission is that we built plumbing ahead of the features that justify it. The sizing model says the foundations are right — and the model is made of assumptions about users we don't have yet.