Back to Blog
·AlgoX2 Team·Product

Inside the X2 SDK: one client in six languages, keyed substreams and six sample apps

Every language has its own idea of how a program should wait. Rust and TypeScript have async. Our C++ steps through time on a clock. Go, Python and Java each have their own habits too.

A client for a distributed system has to live inside all of them. The obvious way is to rewrite the protocol in each language’s idiom. But there is one correct way to talk to the cluster, the one we designed, debugged and know the properties of. Map it six times and you pick up six sets of bugs you did not expect.

The X2 SDK is now in closed beta, and it ships in C++, TypeScript, Go, Python, Rust and Java at the same time. This article covers how one client stays correct in all six, the streaming features it gives you, and the sample apps that come with it.

One core, an event loop outside it

Most of the client’s logic lives in a single library, the client core. It maps almost one to one onto each of the six languages, so the logic is the same everywhere.

The core does no I/O. An event loop attaches to it from the outside, and all reading and writing happens there. That loop belongs to your application. In TypeScript it can be async; in C++ it can step on the clock.

You never hand control to the client library. You write code the way your language expects, and the code that talks to the cluster is the same tested core in every case.

Application code and its event loop talk to the cluster, and the event loop drives one shared client core idiomatic, owned by the applicationyour codeevent loopasync, or clock stepsall I/OX2 clusterclient corethe protocol logic, written and tested once, no I/Omaps onto each languageC++TypeScriptGoPythonRustJavaone CI test suite runs against all six
Scroll sideways for the whole diagramThe protocol logic lives in one client core that is the same in all six languages. The application's own event loop sits outside it and does all of the I/O, so the client library never takes control away from your code.

One test suite for six clients

Because every client wraps the same core, one test suite can cover them all. The CI swaps in each language’s implementation in turn, and each one is required to pass. All the main functionality is tested this way in every language.

Until recently only a large company could afford a release like this. We built it with a few engineers, because agentic coding gives enormous leverage once the foundation is set up properly. The LLMs translated specifications we wrote into each language. Testing when bots write the code explains the CI that makes this safe, and the SDK deep dive covers the models the core is built from.

Hot-hot and hot-warm

X2 supports two ways to keep a process running when a node dies.

  1. Hot-hot. Two replicas process the same input on two nodes, and the duplicate output is removed. Either one can die and the other carries on. Matching-engine failover shows how.
  2. Hot-warm. The second instance waits in suspended animation and publishes nothing. When the first one dies, the standby gets write access to the output at exactly the point where the first one stopped.

Each has its place. Either way it is a feature of the platform, so you do not have to build it.

A read past the end is a subscription

We spent a lot of time on a natural way to name a range of messages in a stream. A read of a range is one short expression.

If that range runs beyond the current end of the stream, you are reading messages that do not exist yet. So the read becomes a subscription, and those messages arrive when they are written. Reading history and following live data are the same operation.

Keys make substreams

Any message sent to X2 can carry any number of keyed fields, also known as secondary indexes. That makes X2 something like a database. Not quite, but database-ish.

A keyed topic usually raises a question: which value goes with a key? X2 answers all of them. Every value a key has had forms a history, and that history is a substream named by the key. In database terms it is a GROUP BY, computed incrementally as messages arrive, for free.

The same range syntax works on a substream. Ask for the first value, the last, all of them, or read past the end and subscribe.

A stream of messages keyed A, B and C, with the messages keyed A gathered into a substream of their own one stream, messages keyed A, B and Cend of streamA₁B₁A₂C₁B₂A₃not yet sentsubstream AA₁A₂A₃next Afirstlastsubscribeevery value the key has had, kept up to date as messages arrive
Scroll sideways for the whole diagramEvery message on a stream can carry keyed fields. The messages that share a key form a substream named by that key, and the same range syntax reads it: the first value, the last, a range, or a range past the end, which is a subscription.

Many applications reduce to this one idea.

Six sample apps

The SDK ships with sample apps that run real workloads. You can download them and run them now.

  1. An exchange with no single point of failure. Use it to build crypto exchanges, internalizers and dark pools. It loses no messages and stays online, and in some failure scenarios it shows no latency spike at all. How to build an exchange covers the log it reads.
  2. A file system on streams. This one is experimental, a working research subject meant to show what is possible. More on it later.
  3. A vehicle fleet over many protocols. Vehicles publish their locations, and clients subscribe to them through Redis Streams, Kafka, NATS over WebSockets and Postgres. It is one cluster with one configuration, and no architecture to stitch together.
  4. A 3D collaborative whiteboard. Several people drag objects and change their properties. Edits merge through a sequencer instead of data structures that merge automatically, and a sequencer has fewer surprising properties.
  5. Real-time video streaming. X2 streams the metadata, attaching the address of each video segment to a message. Any number of viewers can publish and subscribe, scroll back and forth, and seek to a point in time.
  6. A time-series database with roll-ups at several resolutions. Publish years of prices for the whole stock market, then zoom in to any point on any day for any stock, in real time. It is a small sample app with nothing special done to it.

Open-sourcing the X2 SDK describes what else the SDK contains.

Distributed systems from a few parts

A distributed system usually means half a dozen moving parts stitched together. The SDK gives you one tested client in your language, and a platform that already handles failover, history, subscriptions and grouping by key. What is left is your application.

Join the closed beta on the developers page, try the samples, and tell us what should be better.

Watch the video

Plays from YouTube, which may set cookies. See the Privacy Policy.

Watch on YouTube


Want more context on the platform? Read about the product, or get in touch with us on the contact page.