The Retention Cliff: What Happens When a Consumer Falls Behind
A data streaming platform promises infinite append-only logs. You append messages to a stream, and anyone can read them later. That is the model, and it is only true in principle.
In reality, the data set has a budget. Anything older than, say, 72 hours, or past some number of messages, has to be discarded. What remains is called retention, and it turns every infinite log into a band of recent messages that moves forward as new ones arrive.
The question this raises is frequently ignored. What happens to a consumer that is reading near the far edge of that band?
Decoupled, and drifting
The whole point of a streaming platform is to decouple producers from consumers. A team can take its consuming application down for a day, reconnect it, and catch up. An application might connect only after 4 p.m., when the markets are closed, because reading during the day would stress the system.
Every one of those consumers falls behind within the band. The slower it is, the closer it drifts to the edge where messages are being dropped. At some point it is about to go over Niagara Falls.
In Kafka, as far as we know, there is no answer. A consumer near the edge has a consumer’s problem: its messages are dropped. The only defence is to log in to the console and raise retention by hand, in time.
Names, links and streams
AlgoX2 answers this through its path namespace. It is Unix-like: a system of names with ACLs attached. A name can be a special file, a reference to a stream by its numeric ID, or a soft link. A soft link can point to a stream or to a directory, and subscribing to a link to a directory subscribes to everything inside it. That is how AlgoX2 handles the question of topic versus topic-partition.
Cuts
Every consumer group keeps a cut in that file system. A cut is a soft link to a set of positions, one per stream, lifted atomically into a single record. The group installs it under a path such as /cut/my-consumer-group, and it says: this stream at this position, that stream at that one. A cut can cover several thousand streams.
The rule is simple. The system never drops messages that would invalidate a cut. The consumer group updates its cut from time to time to say how far it has got, and that it definitely needs nothing below.
The floor
Any number of cuts can exist. The file system layer takes the lowest position across all of them, for every stream it applies retention to, and calls that the floor. Anything below the floor may be evicted. Nothing above it is.
When a cut holds the floor back and the budget wants more space, the two are in conflict. AlgoX2 raises an alert saying someone should look into it. It does not delete the data and break the consumer.
Consistency over quotas
This is a choice, and AlgoX2 makes it in favour of application consistency. If you asked for 7 gigabytes of retention, the platform would rather go above 7 gigabytes, keep your application consistent, and tell you it has done so. An application should not break because of a storage number.
Retention is still a band. The difference is that the consumers decide where its far edge may go, and a slow reader gets an alert instead of a gap.
Append-Only Storage: The Log-Structured Trie describes how the storage engine underneath removes old data without rewriting what remains.
Watch the video
Want more context on the platform? Read about the product, or get in touch with us on the contact page.