Back to Blog
·AlgoX2 Team·Engineering

When a Node Crashes: Replication Built From One Subscription

Replication is the claim that makes a streaming platform a platform. Without a story for what happens when a node crashes, everything else is marketing.

Most systems answer with machinery of their own: a replication protocol between brokers, an ACK back to the producer, and a consensus service such as Raft or ZooKeeper bolted on the side. AlgoX2 has none of these. Inside the cluster there are no special protocols at all. Every node talks to every other node the same way a user talks to the platform: it publishes to a stream, and someone subscribes.

The clearest way to see how that holds up is to follow one message.

Any gateway, one appender

A producer opens a named stream, ABC say, and appends a message. The message lands on a gateway. Any gateway will do; none is special.

The gateway works out where the message belongs. If its own node is the appender for that partition, the message is published locally. If another node is, the gateway publishes the message to an intermediary stream, and the appender’s node is subscribed to it.

That is the entire cross-node protocol. AlgoX2 reuses publish-subscribe: one node publishes, the other subscribes. Every node subscribes to every other, an n × (n − 1) matrix of subscriptions. Underneath, on the backplane, everything travels as lossy datagrams. There are no TCP connections inside X2.

A message passing from producer to gateway to appender, then read by committers and by subscribers at different ACK levels producerany gatewayappenderintermediary streamlossy datagramsnode 1node 2m₀m₁m₂m₃m₄m₅committercommittercommitterthree copies, each a subscriberevery subscriber names an ACK levelplatform · ACK 1consumer · ACK 3control plane · ACK n/2 + 1
Scroll sideways for the whole diagramA gateway forwards the message to the partition's appender over an intermediary stream, and the appender sequences it. Committers, the platform, consumers and the control plane all read that stream as subscribers; each names an ACK level, the number of nodes that must hold a message before it sees it.

Three coordinates per message

The appender receives the request, assigns the message a sequence number, and the message is published. By now it carries three identities:

  1. The stream ID and sequence number the producer gave it.
  2. The lane ID and sequence number the forwarding gateway assigned.
  3. The partition ID and sequence number the appender assigned.

The producer’s own stream ID and sequence number are also what exactly-once publishing is built on.

Committers are subscribers too

Once published, the message is held in a memory cache, in case a receiver misses it.

The committers that store it are ordinary subscribers. If the partition is configured for three copies, three nodes receive the message. Each keeps a pointer to how far it has read, requests batches to catch up, and gets real-time messages delivered as any subscriber does. Where the network allows it, delivery uses multicast. In AWS, where it does not, the same send goes to each subscriber in turn.

No ACK to the producer

So when does the ACK go back to the producer? It never does.

An ACK is a separate fact about a message, and a separate fact can fall out of step with the publish it describes. That is how idempotency gets lost. A producer that wants to know what happened to its stream does not wait for an ACK. It subscribes to the stream and reads the message itself.

ACK levels

This is where crashes come in. Every subscriber in AlgoX2 names an ACK level when it subscribes: how many nodes must be proven to hold a message before the subscriber sees it.

At ACK level one, the message exists: someone published it and is holding it. At level two, a second node, one the message could be recovered from, has put it in durable in-memory storage and announced how far along it is. At level 45, the message could be recovered from 45 nodes.

The whole system is assembled from this one number:

  1. Applications subscribe at the level they need. A consumer that wants only durable data subscribes at level three, and never sees a message a node failure could still cancel.
  2. The platform itself subscribes at level one, because it is the thing implementing the higher levels and needs every message it can get.
  3. The control plane is not Raft and not ZooKeeper. It is a plain stream, subscribed at level n/2 + 1. That is a majority of the cluster, which is what Raft is.

One number spans everything from cluster consensus to an application’s durability, and the same primitive builds the streaming system underneath both. AlgoX2 does not bolt consensus onto replication. Both are ways of reading a stream.

Watch the video

Watch on YouTube

Want more context on the platform? Read about the product, or get in touch with us on the contact page.