Most trading firms run a message bus, a database, a replication tool, a stream processor and an archive. Each is a different technology that needs its own expertise. Live and historical systems are built on different infrastructure, and research works from its own copy of the data.
Why AlgoX2
One platform for streaming data. Storage, networking, stream processing, connectors and lakehouse gateways run in one fault-tolerant cluster. Developers build applications on it, operators configure pipelines, and analysts query it through Spark or the lakehouse.
Multi-protocol by design. AlgoX2 is Kafka API compatible: point existing producers and consumers at it by changing the bootstrap address. It also speaks NATS, MQTT, Redis and HTTP/WebSocket. Publish with one protocol and subscribe with another.
Deterministic replay. Orders, fills and prices from every source form one ordered timeline. Any range can be replayed without affecting live consumers. An auditor who wants the book as it was at 10:32 replays it to 10:32.
Any data. Raw binary, such as PCAP or a proprietary feed, goes in as it is, alongside structured messages. Nothing has to be decoded first.
Your code inside the cluster. Decoders, filters and strategy runners run as userprocs: programs the cluster runs and supervises, with streams as input and output. Most of the use cases below are built this way.
| By the numbers | |
|---|---|
| New data to consumers | Within tens of microseconds |
| Your code to its data | Microseconds |
| Protocols | Kafka, NATS, MQTT, Redis, HTTP and WebSocket |
| Native clients | C++, Python, Go and Rust |
The pattern
Every use case below works the same way. Sources write events to streams. A program in the cluster reads them, keeps the state it needs, and writes results to another stream. Desks, dashboards and the lakehouse subscribe to that stream. Output depends only on input, so replay rebuilds it.
Use cases
Pricing data distribution
Pricing touches every internal system, yet in most firms it is spread over many systems with no global ordering.
On AlgoX2, pricing parameters come in on one stream, next to the security master. Pricing applications read them and publish prices, Greeks and sensitivities on a second stream. Every consumer sees the same version, from the colo to risk dashboards, and everything lands in Iceberg or Delta tables on S3.
Dark pool and internalizer
Every desk trade sent to the street pays fees and spread, and shows your hand. Many could have crossed with your own market-making desk.
On AlgoX2, quotes and desk orders arrive in one order. A crossing engine matches them at a price tied to the NBBO or the mid and sends only the rest to the street. First come, first served, so fairness is never in question, and any cross can be replayed.
Security master
Every system depends on the security master, yet most pick up changes from nightly files or by polling a database, so some are always a version behind.
On AlgoX2, each source goes in as it is. A consolidator maps it to one schema and one instrument ID and writes a single ordered tape of changes. Every system subscribes and sees each change as it happens. You can also ask what the reference data was on any past date.
Order and execution management
Most OMS and EMS platforms keep order state, the audit trail and recovery in three systems. Under load they disagree.
On AlgoX2, the order lifecycle is the stream. Every event is sequenced and replicated before anyone acts on it, so the journal is the order state. Two copies of the engine write the same stream, so losing one needs no failover. Venue gateways run inside the cluster, and risk, TCA, surveillance and trader screens read the same stream without loading the order path. RFQs work the same way: small requests are quoted automatically and large ones go to a trader.
Real-time central risk book
Stream every drop copy and every marked price into AlgoX2, and the firm gets one ordered timeline of what it holds and what it is worth.
A position engine in the cluster updates exposure on every fill, per desk and net for the firm. The delta manager hedges the firm’s net position, not each desk’s gross. Limits and kill-switches act on the fill that breaches them. Replaying fills and marks rebuilds the exact book at any past moment.
PCAP ingestion
Packet captures usually sit in files at each colo until a copy job moves them, and live analysis, backtesting and storage each work from their own copy.
On AlgoX2, captures stream from each colo as they are written, hardware timestamps intact. After an outage the stream picks up where the cluster’s copy ends, with nothing lost or duplicated. Live consumers read ordered streams, and the raw packets stay as an immutable record for compliance and backtesting. A decoder bug is fixed by replaying the affected range.
Configuration as a stream
Changing a trading component’s configuration usually means a restart or a database that keeps no history.
On AlgoX2, configuration is a stream. A component reads the latest snapshot when it starts and applies changes while it runs. A kill-switch is one message that reaches every instance. Every change is ordered and recorded, with who made it.
Parallel strategies
To test a new strategy version next to production, every experiment needs identical, ordered input.
On AlgoX2, production publishes its inputs and trades once. Each experiment subscribes to the same input, in the same order, and production does no extra work. Results sit side by side on one stream, and any run can be repeated exactly. Any difference comes from the strategy, so you can promote a new version with confidence.
Model fitting and features
The models are yours. What is usually missing is the plumbing: fitted parameters reach consumers by different paths, and nothing records which inputs produced a fit.
On AlgoX2, fitters publish to the same streams pricing reads. Each feature records the input it was computed from, and research recomputes features with the production code by replaying the inputs. AlgoX2 owns ordering, distribution and history. The mathematics stays yours.
Research-cluster allocation
Most in-house schedulers keep decisions, running state and usage history in different places. When a node dies, jobs are lost or scheduled twice.
On AlgoX2, scheduling is an ordered stream. Jobs post requests, nodes report capacity, and policies set quotas and priorities. The scheduler matches them like an exchange, and a second copy writes the same decisions, so there is no double placement. Usage history lands in the lakehouse for capacity planning and chargeback.
Application telemetry
Telemetry usually runs apart from the trading path, so a latency spike cannot be tied to the order that caused it, and the detail is gone in days.
On AlgoX2, metrics, logs and traces go to streams beside the data they describe. A program in the cluster feeds dashboards and ClickHouse, and the raw stream goes to the lakehouse at full detail. You can test a new alert rule on last week’s data.
What is different about AlgoX2
| A typical trading stack | On AlgoX2 |
|---|---|
| A separate fleet to filter or reformat feeds | Filters run in the cluster, on its spare capacity |
| Every topic and partition adds files | A stream per instrument or desk is fine |
| Connector code or middleware between protocols | Publish over one protocol, read over another |
| Standby failover, where state gets lost | Two copies write the same stream |
| History from the lakehouse, live from the bus | One subscription from the past into live |
| Every new consumer loads the broker | A new consumer costs only its gateway |
| Bus, replication, processing, connectors, archive | One fault-tolerant cluster |
These eleven systems are usually eleven separate projects. On AlgoX2 each is a program that reads streams and writes a stream, with the same ordering, the same replay and the same cheap fan-out. Research works from the data production actually saw.
Running an exchange or a venue instead? See Financial Markets.