Exchange Architecture on Streams

An exchange must put every message it receives into one sequence. Years later, it must be able to show, from that sequence alone, why it did what it did. This article builds an exchange out of the ordinary parts of a streaming platform. The platform is AlgoX2, called X2 below, and the construction needs no exchange-specific subsystem.

This article describes the design: what the exchange guarantees, how the parts fit together, and what the design costs. It does not describe the sample code that implements it.

The problem in one order

A bid for 100 shares rests in the book. Two participants, A and B, each send a sell order for 100 shares at that price. The two orders arrive within a microsecond of each other. Only one of them will trade.

Which one is not a matter of opinion. The answer may not depend on which CPU core happened to be free. A regulator may ask, years later, for proof that the exchange’s own published rules produced this answer.

Every exchange faces this question, and the answer shapes the whole system. There has to be one sequence of every message the market received. That sequence has to be recorded. Everything the market did has to be recoverable from it.

Our answer is to fix that sequence first and let everything else follow from it. One stream carries every order, cancel and reference-data change the market received, in one sequence. The book at any point is then a function of a prefix of that stream. So is every trade, every acknowledgment, every market-data update and every position.

A matching engine that reads that stream adds no information of its own. It only computes what the stream already implies. Two engines reading the same stream compute the same result. An engine restarted from a checkpoint computes the same result. An auditor’s own implementation, run five years later, computes the same result again.

What an exchange must guarantee

Return to the two sell orders. The exchange must decide which one trades, and the decision has four properties. Each is easy to state. Getting all four together is the hard part.

• Single-valued. There is exactly one answer, and “either, depending” is not allowed. Price-time priority makes the answer a function of the arrival sequence, so the exchange has to have such a sequence.
• Reproducible. A record of what happened is not enough. The exchange needs a record of the inputs, complete enough to derive the outcome again. Regulation asks for this from two sides: rules on the resilience and auditability of the trading system, and rules on the accuracy of the clocks that timestamp its events.
• Ordered against every effect. A trade has many consumers: two acknowledgments, the book update on the public feed, a drop copy to each back office, a position change at the clearing house. Every consumer has to see cause before effect. A consumer comparing two feeds must never see an inversion.
• Survives failure. Machines fail during the microsecond in question. The answer may not change because of that, and the market may not stop.

A fifth obligation belongs on the list, and systems writing usually leaves it off. It is not about one order. It is about the two ways a market has to look at all of them.

Matching looks at orders by instrument. No order ever matches an order in a different instrument. A matcher needs every order for one instrument, in one sequence, and nothing else.

Delivery looks at the same orders by participant. A participant sends orders across many instruments. It is owed one answer stream that carries its acknowledgments, fills and cancels in a definite order. It also asks for operations that span every instrument it trades, such as “cancel all my resting orders”.

Neither view can be dropped, and no single arrangement of the data serves both. A market has to hold the same records under two keys and keep them consistent. This is a structural obligation, and it constrains the architecture as much as the other four. The repartitioning network below shows how it is met.

Notice what is not on the list. Nothing here asks for concurrent execution. Matching against a book is sequential by the price-time rule: to know what an order matches, you have to know every order that came before it. Concurrency is not a requirement of this problem. A mechanism whose whole purpose is to allow safe concurrency has no purpose here.

Sequencing is the transaction

Start with the one platform primitive the whole design rests on. An X2 partition is a stream that other streams embed into. The embedded streams share its sequencing, and exactly one process may assign positions in it. That process is the partition’s appender.

Now put every participant’s order-entry stream into one partition, and watch what happens when an order reaches the appender and receives position k. The prefix 0…k is fixed from that instant. The book after k is a function of that prefix. So is everything anyone would want to ask: does this order trade, against which resting order, at what price, for what quantity, and what remains resting afterwards. Nothing that happens later can change any of it, and no process other than the appender was consulted.

The exchange’s transaction is therefore the increment. It is indivisible, because a position is assigned once and to one record. It is a decision, because it is where the arrival sequence becomes fact. It is the only step in the whole system with the authority to make the market’s outcome one thing and not another. Everything after it is computation over a value already determined. The matching engine is a decoder: it reveals what the stream implies, and nobody asks it what the outcome should be.

One precision is needed, or the claim would be too strong. The appender assigns positions faster than they are committed to disk, so if the appender dies, its newest unsettled positions can be lost. The platform’s repair procedure collects the committed end of every stream in the partition and records that table as an epoch boundary on the cluster’s command stream. So the death of an appender can change where the partition ends. It can never change what the partition says below that point. A replacement appender is chosen by an election among the platform’s sequencing processes. That election runs only when an appender is assigned or replaced, so it sits off the path from an order to its fill, and nothing on that path votes. A failure introduces one uncertainty, the length of the prefix, and a recorded cut resolves it. A participant that wants to know whether its order survived reads its own stream back at a replication level of its choice. The matcher never sees two different histories for one position.

Three moments, not one commit

A database commit fuses three distinct events. That is why it is expensive, and why so much exchange engineering consists of taking it apart by hand. Naming the three separately is the analytic step this model makes available.

In a database these coincide at commit. Here they are three events, they happen in this order, and each one can be observed separately. Determination is one process’s local act and needs no round trip. Durability is not awaited but discovered. Every committer advertises how far its disk has reached, so commitment is a level that rises behind the data. A reader asks for a position only once K committers have passed it. Revelation depends on how fast the decoder runs. A slow decoder delays knowledge without endangering the outcome, because the outcome was fixed before it read anything.

The practical result is that each obligation lands on the party that has a stake in it. A participant that must know its order will survive a data-center failure reads its stream back at that level and pays that latency itself. The market’s sequencing does not pay it. A market-data consumer that prefers speed reads at the live edge and knows it is reading an unsettled tail. In the fused model these are one dial, set once, for everybody.

What each ACID letter becomes

Walk the four letters. Each dissolves in a different way, and one survives.

• Atomicity. One input message occupies one position. There is no partial position and no rollback: the unit of atomicity is the act of sequencing. An order that produces four trades and a cancel is atomic because it is one input whose whole consequence is determined at once. The four trades are not four transactions to coordinate. They are four records the decoder emits while processing one position. Cross-instrument atomicity comes from the same place: a mass cancel over a hundred instruments is one input at one position, and the decoder that reads it holds every book it affects.
• Consistency. The derived state has exactly one writer, the decoder, and the decoder is the only interpreter of the rules. No concurrent writer can break an invariant of the book, because there is no concurrent writer. The invariant lives in the decoder, where anyone can read it.
• Isolation. There is nothing to isolate. Execution is serial by construction, so the property that concurrency control exists to achieve holds trivially. This is the letter that costs a database the most, and the one that disappears most completely here. Readers of derived data cannot conflict with the decoder, because what they read is an immutable published prefix.
• Durability. This one survives, but it is no longer bundled with the decision. It becomes a quantity, how many committers hold this position, that a reader names when it reads.

Time must be in the stream

A decoder must be a function of its input stream alone. That sounds mild, and it has one sharp edge. A matching engine wants to stamp each order with a time, because time priority is part of the book’s key and participants expect a timestamp on their fills. If the engine reads a clock to get that time, it is no longer a function of its input. Two copies of it disagree about the timestamps. Worse, they produce records that differ only in a field the deduplication does not examine, so the disagreement is invisible in the output.

The resolution is that the sequencing time is data. Every sequenced record in X2 carries two instants: the event time, on the publisher’s clock, and the sequencing time, stamped by the appender when it assigned the position. The appender is one process, so its clock is one clock for the whole partition, with no skew between publishers. The matcher sets its notion of “now” to the sequencing time of the record it is processing, and never calls a clock function. The timestamp on a fill is then a fact about the stream, and anyone replaying the stream reproduces it.

The same rule covers every other way a decoder can go non-deterministic, and they are all the same mistake: any input that is not in the stream is forbidden. A random number, a configuration file read at startup, the wall clock, a query to another service, the iteration order of a hash table. Each one turns the decoder into a function of something no replay can supply. Where an exchange needs an external event, the event enters as a message. A scheduled auction is a message on the input, not a timer inside the engine.

One symbol group

This section builds one symbol group end to end: the order entry for a set of instruments, the sequence over it, the pair of matchers that reads it, and the trade tape they produce. That is the unit. A market is many of them, and a later section assembles the whole exchange from copies of it. The platform’s sample exchange builds exactly this unit, on one node, as one small program that drives the platform’s ordinary verbs. The platform contains no exchange-specific mechanism, and that is the point of showing it.

The streams

Order entry is one stream per participant, and each participant may write only its own. A participant opens its stream in append mode. The open returns the stream’s current end, the participant numbers its orders upward from there, and the platform’s positional rule absorbs a retry and rejects a gap. A participant’s own stream is therefore a permanent, self-describing submission record. “My order number 41” is a name that survives the connection, the process and the machine. A participant that reconnects after any failure reads back its own end and continues.

Those streams are created against one nominated partition, and nothing else is placed on that partition. The interleave is not an added mechanism. Merging embedded streams is what a partition’s appender already does for every stream in the cluster. The partition itself is the market’s sequence: every participant’s orders, in arrival order, with nothing else mixed in. The matchers read the partition, not any individual stream.

The trade tape is a stream on a second reserved partition, so the market’s latency-sensitive output does not share an appender with order entry. Each matcher also keeps a checkpoint, which names where to resume, and a snapshot stream of its own, which carries its book. Those live off the order-entry partition. A matcher reads that whole partition, so anything written there comes back as input, and a book snapshot is bulk state that no other reader of the market’s sequence wants.

The decoder

The matcher receives one input record at a time, dispatches on its type, updates its in-memory book, and emits zero or more output records: a trade for each fill, a resting order for the remainder, a cancel when a resting order is exhausted. It publishes each output record at the position its own output has reached, which is a count of what it has emitted. The output position is a function of how much input has been processed, never of when the publish happened.

Three properties of the matcher carry weight. It takes time from the record it is processing, as described above. It separates creating a book entity from announcing it, so a resume can rebuild state without re-emitting announcements the tape already carries. And it writes its book to its snapshot stream in an order a restore can consume. The identity of a symbol is its position among the symbols, so creation order is part of the state.

Restarting from a checkpoint

Losing one matcher of a pair costs nothing in correctness, as the next section shows. It does cost time. The dead matcher’s book, every resting order in every instrument it matched, was in memory and is gone. The survivor carries the tape meanwhile, so the market does not stop. But until a replacement has rebuilt an equal book, the market is one failure away from having no matcher at all. The plain way to rebuild is to read the partition from position zero, and for a market that opened this morning that is every order of the session. So how fast a decoder can restart is a property worth designing for.

A restart needs two things that must agree: a copy of the processed state, and the input position that state accounts for. If they disagree by one record, the restarted decoder applies that record twice or never. So a checkpoint is not a snapshot. It is a snapshot together with the position it corresponds to. A decoder may read several inputs and writes an output of its own, so it is one position per stream, all taken at one moment. X2 calls this a cut and makes it a first-class object with a name.

A cut holds one position per stream, and each position is stated on both axes: the stream position, where a consumer resumes, and the partition position, where retention acts. A checkpoint is a cut where one of the named positions holds the state, and the other positions name the history that state accounts for. A durable name in the platform’s namespace points at the latest cut. Naming it is also what protects it: the platform frees no message at or above any position a referenced cut names. Publishing a checkpoint and pointing the name at it therefore pins exactly the input history the checkpoint does not account for. Retention and recovery stop being two concerns that have to be kept consistent by hand.

Two properties follow without extra work. The name stays put while the cut behind it is replaced by every new checkpoint, so a restarting matcher asks the name and receives whatever was current at the moment of the restart. And a cut may name only settled positions, so a checkpoint can never point at data a failure could cancel.

Resuming is one call. The matcher opens its checkpoint name, receives the streams the cut names and the position each stands at, loads the snapshot, and reads forward. A checkpoint that was never written names nothing, which is exactly what a cold start means. The first run and the thousandth recovery are the same code path. The regenerated output lands on positions the tape already holds and is absorbed. The cost of a restart is bounded by the checkpoint interval, not by the age of the market.

Each matcher of the pair keeps its own checkpoint and its own snapshot stream, because a resume position and the state it accounts for belong to one process. Only a replica tag differs. That per-matcher snapshot stream has a second use, which the next section comes to.

Redundancy without agreement

Fault tolerance is where the model pays most visibly, so start with what the two matchers do not do. They do not exchange messages. They do not hold a lease, take a lock, or use a fencing token. Neither is primary, neither is standby. There is no election, no view change, and no detection of the other’s death. No component compares their outputs. Each one reads the partition and publishes what it computes.

What makes this safe is that both publish the same record at the same address. Each opens the tape in append mode under a distinct replica tag, which tells the platform they are redundant copies of one producer and not one producer reconnecting. The partition’s appender is the one point where both copies are seen. It applies the rule it applies to everything: a position that is already filled is filled. So the second copy of position k is dropped. When one matcher dies, its twin’s records were already landing at some positions, and they simply become the ones landing at all of them. The survivor’s stream was never a substitute for the dead matcher’s stream. It was the same stream.

The contract

The safety of this rests on the producers, not on the platform. The contract has three clauses.

1. Both copies are deterministic transformers of one input stream. For each input record, each copy emits the same output records, with the same contents, and therefore at the same positions. This is what lets two producers append to one stream together with nothing to exchange.
2. Both copies ignore the end the tape reports at their open. There is no useful synchronization there. While one copy is appending, the value the other one reads is already stale. A copy publishes from where its own computation says. What is already present is absorbed, and the first genuinely new record lands at the cursor.
3. Restart is replay, not negotiation. A copy with checkpoints resumes from its checkpoint and the input position it was taken at. A copy without checkpoints replays its input from the beginning. Either way the regenerated prefix is absorbed and the continuation lands. One discipline applies: a checkpoint must never record output the cluster has not confirmed. A checkpoint ahead of the committed tape would make the restarted copy skip positions, and a skip is the one thing the positional rule rejects instead of absorbing.

What the tape cannot show

The first clause is the producers’ obligation, and the platform does not verify it. This should be stated as bluntly as possible. If the two copies disagree about what record belongs at position k, the appender still keeps exactly one record for position k, because it selects by position and never by contents. The resulting tape has the right length, one record per position, and no record anywhere that anything went wrong. A silent corruption is available to anyone who lets a decoder read a clock.

What can expose a disagreement is whatever each copy writes alone. For a checkpointing matcher that is its snapshot stream, which carries the book its output is computed from, at positions both copies share. Two copies that agree write byte-identical snapshots, and comparing them is a real check. Equal snapshots demonstrate equal state, not equal emitted records. Demonstrating that a pair emits identical records needs matchers that do not share a tape, each writing its own output stream, and a comparison of the two tapes. The platform’s own test suite does that separately, under repeated kills.

Compared with a standby, a vote, and a consensus log

A primary with a hot standby needs failure detection, a lease or fence to prevent two primaries, and a promotion step. The standby’s output is not the primary’s output, so the switch is visible downstream and has to be handled. Here there is no switch, because there was never one writer.

A quorum of engines with an arbitrator votes on each result. This is the shape described in published patents of a large derivatives exchange. Voting tolerates a wider class of fault: it survives an engine that computes the wrong answer, which positional deduplication does not. That is a real advantage. Its cost is a component on the critical path that must itself be correct and available, plus a round of collection per result. This design takes the other trade. It detects engine disagreement out of band, by comparing state each engine writes alone, and it keeps the result path free of any component that has to decide anything.

State machine replication over a consensus log is the same deterministic-execution idea with a different way of establishing the log. The difference that matters is the shape of the commit. A consensus leader appends an entry, waits for a majority to acknowledge, and only then declares it committed, so the client’s answer waits on that round. X2 runs no such round. The appender sequences and moves on, and commitment is a level discovered from positions the committers advertise anyway. Determination does not wait for durability, which is the same separation the three-moments table made, seen from the replication side.

Fan-out over an immutable prefix

An exchange’s visible output is not one tape but a family of derived feeds: public book updates, the last-sale feed, per-participant drop copies, the clearing house’s trade records, surveillance, end-of-day files, risk aggregation. Build those against a mutable book and each one becomes a reader of its state, which makes each one a transaction that contends with the writer and with the others. Snapshot isolation, read replicas and change data capture are the standard remedies for that contention, and all of them exist to reduce a cost the arrangement itself created.

Here every derived feed is another decoder over a published prefix, and a published prefix has no writers. A record that has been sequenced and committed cannot change: not its contents, not its position, not its neighbors. Reading it needs no coordination with anything, because there is no possible interference to coordinate against. A reader takes no lock, holds no snapshot, and cannot be made to wait by another reader or by the producer. The number of consumers is a resource question, not a correctness question. Consumers may be added, removed, restarted or replayed from the beginning without affecting the market or each other.

Cause and effect in one sequence

A second property is easy to miss, because it looks like a detail of stream placement. The matchers publish the trade tape into a partition, so the trades are themselves sequenced records with positions. In the general form of the construction they can be published into the same partition that carried the orders. Then one partition holds the market’s inputs and outputs together. The matcher sees its own trades come back as input and skips them by record type. A consumer that must see cause before effect gets that from the order it reads records in, with no join, no timestamp comparison, and no reconciliation of two feeds. A surveillance system asking “which order caused this trade” is asking about a position, not performing a correlation.

The alternative, outputs on a separate bus, forces every downstream consumer to rebuild the ordering that already existed upstream, usually by comparing timestamps from two different clocks. That is the standard source of the standard bug, and here it is avoided by structure instead of by care.

What fan-out costs

The claim that fan-out parallelizes freely is a claim about correctness, not about cost. A sealed record crosses the network to a node once, however many consumers that node hosts. What multiplies is the node-local work after arrival: one copy per subscriber per record, and the wakeup of each subscriber. At high rates the per-delivery bookkeeping, not the bytes, is what shows up.

The platform’s own remedies are engineering: stream-scoped wakeups, batching under backlog, and direct delivery into a local subscriber’s lane. The more effective lever belongs to the application and follows from the model. A consumer that needs a subset should subscribe to a stream that carries the subset. An embedded stream shares its partition’s sequencing. So a decoder that republishes a filtered projection (one instrument, one participant’s fills, one message class) gives its consumers the same ordering guarantees on a fraction of the traffic. Those projection decoders can run as pairs by the same rule as the matchers.

The exchange as a repartitioning network

One symbol group is the unit, not the whole. A market runs many of them, and this section is about what joins them. It is where the fifth obligation is finally paid, and where the design’s most consequential structure appears.

Two keyings, and neither is negotiable

Matching needs symbol locality. To know what an order matches, a matcher must see every order for that instrument in one sequence, and it needs nothing else. So the natural keying of order entry is by symbol, and along that axis the work shards perfectly. Partition the instruments, give each partition its own appender and its own pair of matchers, and every partition interleaves every participant’s orders for its own instruments into one sequence.

Delivery needs user locality. A participant sends orders across many instruments and wants one answer: its acknowledgments, fills and cancels, in a definite order, on one subscription. Assembling that means gathering records from every symbol partition the participant touched. That is exactly the keying matching does not use.

Neither requirement can be given up. The first is what matching is. The second is what a participant connects for. So an exchange is not a sharded system that happens to need a consolidated view added afterwards. It is a repartitioning network by nature: keyed by symbol on the way in, keyed by user on the way out. The only question is what the repartitioning costs.

Repartitioning is publish-then-sequence

Here is the construction to avoid, and it is the obvious one. Give each participant a consolidator that reads every symbol partition the participant trades and merges the records into one feed. That is a decoder with many inputs. The order in which records from two partitions reach it is their arrival order, and that order is not a function of the two partitions. So the consolidator’s output positions are not a function of its input positions. Two copies of it interleave differently, and by the rule above the two may not share an output stream. The consolidator is therefore a single writer, and a single writer in the delivery path brings back the lease, the failure detection, the promotion and the visible seam that the matcher pair existed to avoid. One stage of the wrong shape undoes the property the whole design is built on.

The merge does not need to be performed by a reader at all. A partition is a stream that other streams embed into, and its appender interleaves them as they arrive. Inventing an interleave is exactly what an appender is for, so that is where the merge belongs. For each symbol partition, run a user partitioner: a deterministic decoder that reads that one partition and publishes each record into a stream of its own, embedded in the user partition of the record’s participant. Every user partitioner then has exactly one input. So every one of them is a deterministic transformer whose output positions are a function of its input positions. By the same arguments as the matcher, it is exactly-once, restartable from a checkpoint, and able to run as a pair. The user partition’s appender merges their streams, and a participant subscribing to that partition receives everything about itself in one sequence.

The difference is a rule, and it decides where every repartitioning in such a system belongs. Read-then-merge makes the interleave a reader’s computation, and a computation over several partitions is not determined by them. Publish-then-sequence makes the interleave an appender’s decision, and the appender is the one component in the system licensed to invent order. Route every repartitioning through an appender and the network contains no deterministic merge anywhere. Every stage is a single-input decoder, so every stage is redundant by the same mechanism as every other.

Read the diagram edge to edge. A participant’s connection ends at a gateway. The gateway publishes each order into the partition that owns the order’s symbol. That partition’s appender assigns the position, and that is the moment the outcome is determined. A matcher pair reads the partition and publishes the trades back into it, so each instrument’s inputs and outputs share one sequence. A user partitioner pair per symbol partition republishes each record under its participant’s key. The destination user partition’s appender merges those streams, and the participant reads one stream that carries its fills across every instrument it trades. Surveillance, clearing and risk attach as further decoders, on the symbol partitions where they want per-instrument order, on the user partitions where they want per-participant order.

Every box on that path is one of two things: an appender assigning positions, or a decoder reading one stream and publishing another. There is no third kind of component. In particular there is no router, broker, coordinator or arbitrator anywhere in it.

Non-blocking edge to edge

Count the round trips on that path. There are none. Every edge is a positional publish. The gateway checks a position and forwards. The appender assigns a position and moves on. The matcher reads and publishes. The user partitioner reads and publishes. The committers write and answer nothing in particular. No stage sends a request and waits for a reply, and no stage’s progress is gated on an acknowledgment from the stage after it. What bounds a stage is the free point, the level below which every record is committed and consumed, rising behind it at commit speed. That is backpressure, not a handshake.

So the network’s latency is the sum of its stages’ work, not the sum of its stages’ round trips, and that is what makes it safe to add a stage. A participant that wants commit confirmation still pays for one, by reading its own stream back at an acknowledgment level. But it pays alone, and no part of the market’s path waits with it.

Scaling the two axes independently

The two keyings give two independent scaling axes. Symbol partitions scale matching. Instruments never interact, so throughput grows with the number of sequencing processes the partitions are spread across. One sequencing process per node serves every partition homed on that node, so partitions on one node share one sequencing thread. Matching scales by adding nodes, or by spreading partitions across the nodes there are. User partitions scale delivery and fan-out. The user partitioners scale with the product, one deterministic pair per symbol partition, and each one’s work is proportional to the traffic of the single partition it reads.

Nothing in that is a global bottleneck. The only serial element anywhere is one partition’s appender, and it is serial only over the instruments assigned to it. That is the one place where the market’s rules require a single order. A market that outgrows a partition splits its instruments across two. A market that outgrows its delivery capacity adds user partitions. Neither change touches the other axis, the protocol, the publish rule, the deduplication or the recovery story.

Cross-symbol operations

A participant with resting orders in four hundred instruments detects that its own pricing has gone wrong and sends one message: cancel everything of mine. What does the market guarantee?

The cancel is published into every symbol partition that could hold that participant’s orders, and each partition sequences it at a position of its own. In each book the cancel is ordered against exactly the orders it could affect, and any order sequenced after it in that partition meets a book without those resting orders. There is no global instant at which the cancel took effect, and none is needed. An order can only trade in its own book, so being ordered ahead of it in that book is the whole of what the guarantee needs.

Independent clusters per symbol group would not break ordering. They would break the participant: N identities, N connections, N recovery procedures, N namespaces, and no consolidated answer stream anywhere. Building that stream outside the clusters means writing exactly the multi-input consolidator this section rejected, now with no platform underneath it. The argument for one cluster with many partitions is that the consolidation can be built as a stage of the system, with the same exactly-once and redundancy properties as every other stage.

One case stays hard, and it should be stated plainly. A pre-trade credit limit that must be exact across every instrument is a cross-partition invariant, and nothing here makes it free. The user partition helps, because a risk decoder reading it holds an exact, ordered position for that participant across all instruments. But that sits behind the match, not in front of it. An exact check in front of the match needs either credit pre-allocated per symbol partition, so each partition spends against a local budget, or a check that accepts bounded staleness. The model sharpens the choice. It does not remove it.

Where a deterministic merge is still wanted

Publish-then-sequence removes the multi-input decoder from the network. It does not remove it from every consumer a market might want. A surveillance replay that must present two instruments’ flows in one sequence, identically on every run, is asking for something the appenders never established. The two partitions were sequenced independently, and no fact about either says how they interleave.

Such a consumer must supply the missing order itself, from data: order the records by a key both partitions carry, and emit only below a position both inputs have passed. That is deterministic, so it may run as a pair, and it costs latency bounded by the slower input. The platform supports the shape, since a cut carries a position per input and a multi-input decoder resumes exactly. What the platform cannot supply is an interleave nobody ever chose. This is also why matching itself stays inside one partition: a matcher must be a function of one sequence, and two partitions do not share one. The symbol group, not the symbol, is the unit of partitioning, and the grouping is chosen so that no rule of the market spans a group.

The record an auditor can use

Look at the model from the outside. The property that makes redundancy free is the same property an audit needs, and that coincidence is the strongest practical argument for the design.

The complete record of what the market did is the input log plus the decoder. Anyone holding both re-derives every trade, every acknowledgment and every timestamp, byte for byte. This is stronger than “the exchange retains its audit trail”, because it does not depend on the exchange’s output being complete or correct. A disputed trade is settled by recomputation from inputs the participants themselves produced and numbered, not by consulting a record the exchange wrote about its own behavior. The timestamps are part of what re-derives, because sequencing time is data on the record, not a clock read at execution.

The storage engine adds a mechanical convenience. Its index is append-only and every write produces a new root, so a read can be addressed to a historical root and the store answers as of that point in time. Reconstructing the book as of a position is a read of a prefix, not a restore of a backup. Append-Only Storage describes that engine.

Two limits keep this honest. Re-derivation reproduces the decoder that was running, bugs included. It establishes what the system did and what its rules, as implemented, required. It does not adjudicate between the implementation and the published rulebook. And it requires that the decoder’s build be identified as part of the record, since “the decoder” is a specific binary and a different one is a different function.

How this compares

The design leaves out a lot, so this section places it beside the families it descends from and the ones it is most often confused with.

Three points follow from the table.

Serializability names no sequence. The transactional model promises that the outcome equals the outcome of some serial execution. It does not say which one, it does not promise the same one on replay, and it hands nobody a name for it. For a bank ledger that is exactly right, because the bank does not care in which order two independent deposits were applied. For an exchange it answers the wrong question. The sequence is the thing being sold. A participant’s claim to a fill is a claim about its position in the sequence, so an exchange that could only certify “some order existed” could not say whose fill it was. And in the transactional account the outcome becomes real when the log record is safe, so the moment the market knows who traded is bounded below by a disk or a quorum. Every low-latency exchange fights that coupling. The three-moments table shows the coupling is an artifact of the model, and taking it apart is most of what this design gains.

The deterministic database is the closest ancestor. Partitioned run-to-completion databases removed concurrency control by running each partition single-threaded. Deterministic databases took the next step: agree on the input order first, and deterministic execution makes the replicas agree on the output for free. That is the same insight as here, and the remaining difference is which artifact is the system of record. There the database is authoritative and the input log is machinery for keeping replicas in step. For an exchange the ranking is reversed: the stream of what participants sent is the primary record, and the book is a summary of it. Once the stream is authoritative, the batching epoch, the lock manager and the separate replication of executed state all fall away.

Voting tolerates a wrong engine; positional deduplication does not. The sequencer-plus-voting design starts from the same insight: a single point of determinism that stamps arrival order. It then reintroduces agreement on the output path. Every result passes through an arbitrator that must be right about which answer is right, and that arbitrator is on the critical path and can fail. Here two engines’ copies of one result collide at one address, and keeping the first arrival is not a judgment about correctness. It is the platform’s ordinary rule that a position is filled once. Arbitration by identity replaces arbitration by vote. What is given up is tolerance of an engine that computes a different answer, and that has to be caught out of band, as described above.

One more relative deserves a sentence. The most widely deployed log orders records within one topic-partition and nowhere else. Two topics have no defined order between them, and none can be recovered afterwards. The construction here depends on a merge of many participants’ streams being a first-class object with its own dense numbering, and that is what an X2 partition is.

Limitations

• One sequence has a throughput ceiling. One partition is sequenced by one process, so a market’s single sequence cannot be scaled by adding machines to it. The answer is more sequences, and the cost is that whatever must be matched against one sequence must fit in one.
• Determinism is unchecked. Deduplication selects by position, never by contents, so two decoders that disagree corrupt their shared output silently. Comparing independently written state detects it, and doing so is a discipline the operator adopts, not a guarantee the platform gives.
• A deterministic bug is exactly reproducible. Excellent for diagnosis, unhelpful in production: a record that crashes the decoder crashes its twin at the same position, and both restarts crash again. The operational answer, quarantine of the offending position, has no equivalent in a system where a retry might simply work.
• Time and every other outside input must be injected. A decoder may not read a clock, a file or another service. Logic that depends on elapsed time needs time to enter as messages.
• A cross-partition interleave has to be invented. A consumer that wants a reproducible interleave of two partitions must supply the order itself from a key both carry, and wait on the slower input.
• An exact pre-trade limit across instruments is not free. In front of the match, the choice is a per-partition budget or bounded staleness.
• Fan-out has a node-local cost. Correctness-free fan-out is not cost-free fan-out. The mitigation is to publish the projections consumers actually want.
• Not evaluated here. This article argues a design and points at a worked exchange that a test exercises. It reports no throughput or latency measurements for a production-scale market, and the comparison with a serializable or a voting architecture is analytic, not experimental.

---

Want more context on the platform? Read Everything is a Stream and Append-Only Storage, learn about the product, or get in touch with us on the contact page.