Introducing the X2 SDK: how the streaming operating system works, and how to start
Unix made everything a file. X2 does the same thing with streams. Log in and you find folders that organize your streams, and streams that manage the system itself, the way /sys does on Linux.
That is not a metaphor laid over the top. The system really is built out of streams. Creating a topic is a record on a stream. Monitoring comes out of the system as a stream. Learn a few principles and the rest follows from them.
The first alpha of the X2 SDK is now available, and you can join the closed beta on the developers page. This article walks through how X2 works under the hood, the two bets behind it, and what you get when you start.
The system is made of streams
Cluster changes, alarms, monitoring and coordination between nodes are all streams. There is no external metadata system to run: no ZooKeeper, and no separate KRaft quorum as Kafka now has.
One partition is special. Every node replicates partition 0 and the nodes vote on its leader. That leader sequences the metadata that keeps the cluster consistent. When a new stream has to be known to every node, the change is just a record on partition 0.
This also makes the system easy to inspect. Which node is the leader? Read the stream that says so. Most of the cluster’s state can be found by interrogating its streams.
Meet teams where they are
Nobody moves a whole industry to a new system in a day. So X2 supports Kafka, NATS, MQTT and Redis Streams in the same cluster, with high fidelity and good performance. Teams bring the code they already have and grow into the rest.
X2 also has a native protocol, and it has one clear advantage: performance without client tuning. Kafka gets much of its speed by batching in the client, behind settings like linger.ms and the maximum batch size. Tune them wrong and performance suffers. Across hundreds or thousands of applications, managing those settings team by team is overwhelming.
X2’s clients stay simple. The server batches optimistically, and because the protocol does not have Kafka’s back-and-forth of acknowledgements, the client does not have to batch at all.
Two bets
The first bet is commodity streaming: the produce-and-consume workflow that moves data across an organization. Teams do remarkable things with streaming, but at a high cost in dollars and in having to learn new models. We want streaming to have its S3 moment, where storage became effectively infinite and behaviour changed faster than any new mental model could have changed it. Lower cost helps. Fewer sharp edges matter more: lots of streams and lots of consumers should not knock the system over or force you to overprovision. We wrote more on this in The State of Streaming.
The second bet is a new class of applications. Write-ahead logs sit at the heart of most data systems, and distributed ones at the heart of many. Kafka and NATS have existed for years, yet those systems rarely build on them. They were not fast enough, so teams wrote their own specialized logs and lost the ability to reuse a common one. Our hypothesis is that you can build more ambitious systems on X2, especially with compute running inside the cluster.
Performance isn’t a goal in and of itself. An order of magnitude lower latency lets us think about using streams for new use cases, cases where you couldn’t do it before.
Three roles
X2 divides the work along natural seams. It is the shape most streaming systems evolve towards.
- Gateways handle connections and take messages off the wire. They are more or less stateless, and a client can talk to any of them and get the same result. Kafka clients have to find the right broker. An X2 client can be placed on the best gateway to save a network hop, but it does not have to be.
- Transaction nodes,
TXN, are the sequencers. Each leads a partition and gives it one order. - Committers follow the sequence and write the log durably to storage.
The connections between them are streams too, called lanes: one from every gateway to every transaction node.
Streams are logical, partitions are physical
A partition in X2 is not a Kafka partition. First-generation streaming systems were built for a few very high-throughput data sets, such as logs from every node in a fleet. But a business has far more kinds of data than that, and the many small kinds together add up to more than the few large ones.
When every stream costs a physical file on disk, modelling the real business gets odd performance. X2 breaks that link. A stream is a logical sequence of records, and many streams are multiplexed onto a smaller number of physical partitions. You can have thousands, hundreds of thousands, even millions of streams that match real concepts, and the performance stays the same however many you create.
Fewer physical partitions also let X2 use a thread-per-core design all the way through, learned from building the NYSE exchange.
A log you can look up
A plain log of bytes is a great primitive, but ambitious applications need a little more. If many streams share a partition, the system already has to find one stream’s records inside it. So X2 keeps an index over its streams.
You can ask for every record for one account ID, and X2 retrieves them without a scan. It is not a general-purpose key-value store, and reading in sequence is still what it is built for. But it makes many things simpler than scanning the whole log.
The index is built in memory and serialized into the same physical partitions, with every pointer pointing back to earlier data. If a node fails, it falls back to the last index checkpointed in the sequence and rolls forward. Your own materialized views and snapshots can use the same technique. Inside LST covers the storage engine behind it.
Idempotency across failures
Each message gets a stable number, and X2 tracks how it moves: through which lane, and to which position in its physical partition. Any of those can be a key in the index.
Kafka has an idempotent producer, but the state that removes duplicates lives in the broker. If that broker fails, duplicates can still get through. X2 keys on the message’s path through the system, so idempotency holds over a broad class of failures, including client and cluster failures. Exactly-once semantics without the latency tax goes deeper.
Hardware does the fan-out
In most systems the leader writes every message to each replica itself. Where the network supports it, X2 hands that work to the network: a multicast group, or shared memory over InfiniBand.
The baselines, on four nodes with relatively few cores:
p99 Produce Latency
- on high-end hardware
- <5µs
- in the cloud
- <600µs
Measured from publish to acknowledgement, with no client-side batching delay.
p99 End-to-End Latency
- on high-end hardware
- <15µs
- in the cloud
- <1ms
Measured from publisher to subscriber through the cluster.
Throughput (4-Node Cluster)
- messages/sec delivered
- 180M
- messages/sec produced
- 20M
Multiple readers can follow the same stream at high fan-out without slowing down producers.
Ten microseconds, 200 million messages a second explains why numbers like these matter.
Compute in the cluster
A userproc is code that runs inside the cluster. Running computation next to streams is not new. What is different is where a userproc sits: in the back plane, with its own transaction on a TXN node, fed at a lower level and in some cases through shared memory. It reacts to the stream as fast as anything can, keeps state in memory and builds views over the log.
Think of a stored procedure in a database that updates in real time.
A multi-user file system
One experiment is a shared file system, not yet in the SDK. A log-structured file system works off a log, and X2 can run one fast enough to feel interactive. A userproc materializes the metadata: inodes, directory trees and the rest. A second userproc compacts small changes, such as appends and line edits, and writes them to S3. To read a file, a FUSE client on your machine is handed an S3 URL, hydrates the file from it, and follows any changes in real time.
Joins without a second database
In Flink, a consistent streaming join has to manage the whole key space as state. It is a complete copy of the data the streams already hold.
With an index over the stream, a join can assume the two streams arrive at roughly the same time and pair them as they come. When a record is missing, it fetches it by key. The stream store acts as a cache instead of another database built on top.
An exchange
Polymarket, our first customer, built its core exchange on X2 with userprocs, and runs a high-performance matching engine for its prediction markets. Time-series databases and RocksDB state inside userprocs are among the other uses we are exploring. How to build an exchange shows the pattern.
Six clients from one model
Clients for a system with smarts in the client tend to drift apart from language to language. We want consistent behaviour and good performance, with idiomatic APIs in each language.
The core client-server interaction is specified as TLA+ models. Together with the canonical C++ implementation, they are the reference. From them comes a state machine that holds all the logic, and each client wraps it with an I/O model and a concurrency model. An acceptance suite tests every client against it, so C++, Go, Rust, Python, TypeScript and Java all get clients of the same quality from the start. Idiomatic high-level wrappers come next, shaped by your feedback.
Getting started
The alpha ships as Linux tarballs or as a Docker image, which stays private during the alpha. The SDK is a GitHub repository, and access is by invitation; if someone you know wants to try it, tell us. Lots of you will point an agent at it, so the repository has an AGENTS.md and an llms.txt next to the quick start and the documentation.
Once it is running, you can poke around it like an operating system:
- One
docker runstarts the X2 supervisor, which creates a temporary cluster and daemonizes it. x2cliopens the root of the stream file system. Under/sysare the command streams that change the system, the alarms it raises about itself, and the streams of eachTXN. Typehelpfor the rest.x2writeandx2readwrite a stream and read it back. The public streams accept anyone, which suits a demo but not production.x2benchmeasures throughput as messages go in and out.x2uiserves a web console with dashboards and the flow of data through the system. You connect unauthenticated at first; a token lets you connect as a user who sees more of the system and can start the demos.
The SDK also has tools for deploying to a real cluster, reading and writing S3, and change data capture. The open-source SDK announcement lists what it contains.
What we want to hear
Compare X2 with the tools you know, such as Kafka and NATS, and tell us where the UX is worse or the performance falls short. We want to know how you would like to consume a stream, and what the best cloud services you use do well. File bugs as GitHub issues, and use GitHub Discussions for everything else. The closed beta is open on the developers page.
A system built on its own idea
X2 runs its own metadata, alarms and monitoring on the same primitive it offers you. The SDK is the chance to build on that primitive yourself, and to tell us where it falls short.
Watch the video
Plays from YouTube, which may set cookies. See the Privacy Policy.
Want more context on the platform? Read about the product, or get in touch with us on the contact page.