Back to Blog
·AlgoX2 Team·Engineering

Testing when bots write the code: CI and the four-legged stool

With agentic coding, you can work 24 hours a day. You give an agent a task, and if you specified it well, it will not come back for an hour or two. You can even tell it not to. That gap works like a branch delay slot in a processor: a fixed stretch of time you can fill with something useful, such as sleep.

AlgoX2 is 16 people, mostly engineers, and all of them write code with LLMs. So the question every team is asking now is ours too. How do you test all this code, and what happens to CI?

Our answer is that testing matters more than it did before the agents arrived, not less.

The limit moves

Bots raise throughput enormously: 50 times, 100 times, close to without limit. As fast as you can say what to build, they can build it.

That moves the weakest link. It is no longer how fast code gets written. It is the orthogonality of the architecture, meaning how cleanly its parts stay independent of each other.

If the architecture is ad hoc, agents running ahead and building on unsynchronized areas will clash. You get inconsistencies that no single change introduced on its own. If the architecture is well thought through and orthogonal, everything stays level as they build.

Think of a skyscraper. Every floor has to sit exactly on top of the one below, or the weight of the building brings it down. The faster the floors go up, the less room there is for one of them to be slightly off.

The rules that keep AlgoX2 orthogonal, such as single-threaded processes that talk only through streams, are covered in Engineering Principles. Much of that structure is generated from specifications by OpenACR.

Nobody can read every merge request

Simply reading the description of every merge request is hard enough. A description can run two or three pages. It mentions things you may not have thought about, and you have to judge each one and fit it into your picture of where the system stands.

We tried. A person cannot keep up. So the testing system has to do the work a human reviewer used to do.

Review bots and the babysit

The first part is a review bot. It reads each submission with a critical eye and:

  1. checks it for consistency with the rest of the system,
  2. raises counterfactuals, asking what happens if an assumption does not hold,
  3. writes test cases that try to break whatever the commit adds,
  4. talks the change through with the agent that submitted it.

The other side of that conversation is what we call the merge request babysit. An engineer leaves a session running for each merge request. It has the context of the change, so it can answer the review bot, make the changes it asks for, and converge on a version it no longer wants to improve because it is acceptable.

The two have to be kept in step. The review bot cannot keep finding something new and burn cycles forever. The babysitter has to know when to stop, when to reject a proposal, and when to finish the task rather than take the weight of the world on its shoulders.

Keeping CI from clogging

More code means more pipelines, and CI clogs too. It needs priorities. Branches have to rebase themselves onto the latest master, so that waiting does not blow up quadratically as merge requests pile up behind each other.

There is a shortcut some teams take for speed. They test each merge request against the master it branched from, and if the tests pass, they merge it, even when it is not a fast-forward of the current master. That policy has a flaw.

The four-legged stool

Take a four-legged stool. One commit saws off leg one. The stool still stands on three legs, and the tests pass. Another commit, written independently, saws off leg two. That stool stands too, and its tests pass as well.

Accept both, and you have a stool on two legs. It falls.

Two commits each remove a different leg of a four-legged stool; merged together, the stool falls masterfour legscommit A saws off leg 1stands on three, tests passcommit B saws off leg 2stands on three, tests passA and B mergedtwo legs, falls
Scroll sideways for the whole diagramEach commit passes its tests against master on its own. Merged together without testing them as one, they leave a stool on two legs.

Two changes that each pass on their own prove nothing about the two together. The only state worth testing is the one that will actually be on master.

Every merge resets everything

So the moment anything merges into the main line, every other merge request has to reset and start again from scratch.

It’s very reminiscent of Bitcoin mining, where everybody is executing some long-running transaction, and when one person wins by minting a block, everybody has to reset and start from scratch.

Alexei LebedevCEO and Co-Founder

Starting from scratch means the entire system, every time. It begins with reformatting all the source code. Then come the performance tests: throughput and latency, on ARM, on macOS and on x86, over Ethernet with multicast and over InfiniBand. Then every failover test.

When you build mission-critical infrastructure, that is what testing has to mean.

The discipline is the product

Agents remove the limit on how fast a system can be built. They do not remove the limit on keeping it consistent. That job falls to the architecture, which has to keep every new floor level, and to a CI strict enough to test exactly what master will become. The faster the bots build, the stricter that has to be.

Watch the video

Plays from YouTube, which may set cookies. See the Privacy Policy.

Watch on YouTube


Want more context on the platform? Read about the product, or get in touch with us on the contact page.