The Agents Were Fast. Our Feedback Wasn’t.

The agents could produce more than I could pay attention to. We built the product four times. Each version technically did what I had defined, but it still did not feel good to use. I needed to see it, try it and give feedback on the experience while there was still a small enough piece of work to change.

Then there was the wait for the automated checks. Continuous integration, or CI, could take an hour or more to tell us whether a change passed. Those checks could help us decide whether to accept the code; they could not tell me whether I wanted to use the result. We needed both. I wanted less waiting between a change, the checks and the conversation about how it actually felt—not simply more code arriving faster.

We are building Solleva as a shared place for people and agents to keep knowledge, the things they create, and the work they do together. Agents also help us build it. Around their work, we had assembled a development system to combine changes, build the application and check its behavior.

Reading Owen Gretzinger’s account of speeding up CI made me realize how much I’d stopped thinking to share. I spend so much time inside this work, I assume people already know these tricks, or have better ones. So this is me starting to build in public — what we tried, what failed, what changed. I’d like to compare notes with people solving the same problems, rather than assume our way of doing it’s obvious, or the best.

We changed what repeated after a failure, what could run ahead, and when a previous successful build could still apply. In a small recent sample, some jobs finished in roughly three minutes; others still took roughly nineteen. The useful story is why they differed, what each shortcut needs to preserve, and how we approached shortening the feedback loop around work small enough to try and judge.

The first problem was easier to see. Two failing tests could make us repeat almost everything.

The first bottleneck

Our earlier behavior-checking CI ordinarily took about forty minutes. One run recorded in the development history took 109. Two scenarios with intermittent failures had triggered retries, but the retry policy restarted the entire task: its infrastructure setup and all 299 specifications describing expected behavior.

The failures were narrow. The work we repeated was not.

A specification that had already passed went through the process again because it shared a task with one that had failed. Before another attempt could tell us anything about those failures, we paid for the surrounding setup again too. The retry policy had made the whole task the unit of recovery, even when the test runner could identify the individual specifications that needed another attempt.

We moved the retry into that narrower scope. Instead of asking CI to restart everything, the runner retried the failed specifications. This did not fix the reason they were flaky. It changed how much work their failure caused us to repeat.

Before adding compute or making the tests themselves faster, ask what exactly happens after a failure. A run’s total duration can hide the answer. The useful record includes which tests failed, what each retry selected, and which setup steps it repeated. If a small failure causes a large body of completed work to run again, the recovery boundary deserves attention.

Moving that boundary also changed what could recover automatically. A failure before the runner started no longer received the old whole-task retry. There would be no list of failed specifications to hand back to a runner that had never begun. That case needed to remain visible as an infrastructure or startup failure, rather than disappear behind the faster retry path.

The retry moves inside the runner; failures before startup remain outside that recovery path.

The change needs testing on both sides of that boundary. In controlled test runs, trigger a failure inside a test and inspect what repeats; separately, interrupt preparation before tests begin and inspect how the failure is reported. Keep limits on retries and retain the failure history. Repeatedly trying until something turns green can conceal a problem instead of giving a trustworthy answer sooner.

The historical record supports this change in retry scope, but does not provide a matched after-duration to quantify the saving.

But a test result was still only one step toward accepting a change. Our development system also had to decide which changes could be combined and which version a passing result actually described.

Speed was not enough

Our custom merge queue assembled proposed changes and coordinated their verdicts. JetBridge, a CI system derived from Concourse, ran the build and test jobs underneath it. One ambition was twenty landings an hour. That was a target, not a measured result, and reaching it would require more than making each test run shorter.

If several changes are waiting, checking and accepting them one at a time leaves later work idle. Checking each independently is not enough either: two changes can pass separately and fail when combined. The historical queue implementation used a more useful approach: test the combinations we expected to accept.

Suppose three changes, A, B and C, are waiting in order. Starting from the shared code, the queue prepares three candidates:

• The shared code plus A.
• The shared code plus A and B.
• The shared code plus A, B and C.

Their checks can run in parallel. If the third finishes first, it waits. Once A passes and lands, the A+B result can support accepting B; then the A+B+C result can support accepting C. The queue does not have to reconstruct and retest each combination simply because the preceding change has landed. That preceding change was already included in the candidate being checked.

This is a familiar pattern, also described in GitLab’s merge-train documentation. The useful question for us was how to preserve its benefit when the expected sequence changed.

Suppose B is withdrawn before it lands. A’s work can remain valid, but the result for A+B+C no longer answers the question we need to ask about A+C. The historical code handles this by preserving A’s work, invalidating later candidates that include B and stopping their obsolete runs. It must establish a result for the new combination before C can be accepted.

That is a more precise rule than “the base changed, so rerun everything.” What matters is whether the passing result still describes the combination being accepted. Expected progress through the chain can preserve that relationship; removing a change from the middle breaks it for the candidates that included it.

The historical implementation and its test definitions were inspected for this account. The tests were not executed, and deployment was not verified; this describes behavior encoded in the source, not a newly demonstrated production result. The tradeoff is visible even without a timing claim: testing ahead can shorten the wait if the expected sequence holds, but spends compute on combinations we may never accept if it changes.

Two controlled cases expose whether the queue preserves that relationship. Let the last candidate finish first and confirm it cannot skip the predecessors it includes. Then withdraw B and confirm that A’s work remains valid while the A+B+C result cannot authorize A+C. Measure both the wait removed and the speculative work discarded. A rising number of parallel runs is not, by itself, evidence of better throughput.

Passing the right checks still answers only part of the product question. The queue could preserve the meaning of a verdict without deciding whether the accepted work was worth doing.

I was encountering that distinction in the application itself. More code could land, but I still had to ask whether I wanted to use what we had built.

Change the size of the work

My first response to an application that felt wrong was to replan the application. I tried to describe everything that needed to change, then let the agents build it again. They fixed the problems I had named, but by the time the next version came back, its behavior still did not feel right.

Breaking the design into smaller pieces helped the look and feel. It did not solve the whole problem, because I still had not been trying the behavior and revising it along the way. I was getting better at specifying a large answer before I had learned enough from using it.

We restarted around smaller, complete pieces of behavior. A “vertical slice” meant enough of the interface and the underlying application to let someone actually do one thing, rather than finishing a layer of code that would become useful later. Real use became part of deciding whether that piece was done. I spent more time planning what success meant, testing and reviewing each increment, and less time asking agents to build the whole application at once.

A later piece of work makes that change concrete. We kept the work on creating a new space in Solleva deliberately small: choose New Space, give it a name, create it and arrive there. I asked us to defer extra details, including an optional visual preview. The point was to get the essential journey into a form we could evaluate.

That boundary matters. A working creation function would not, on its own, show that someone could find the control, enter a name and reach the place they had created. A polished form would not establish that either. The increment had to connect the interaction to its result.

The review still recorded questions about keyboard focus, Escape and duplicate submission. Those are not incidental to the journey: where typing goes, whether someone can back out, and what happens if they submit twice all affect whether the interaction behaves as intended. Keeping the work small did not make these questions disappear. It made them specific enough to stay in the review instead of being lost inside a claim that the application was finished.

The next task can start with that endpoint: what should a person be able to accomplish, and where should they end up? Build enough to walk that path, then use it before expanding the scope. Record the parts that work, the parts that remain uncertain and what the next change needs to address. An agent’s completed task is a prompt for that review, not a substitute for it.

In my own workflow, the review began to feel more like working with a partner. We could iterate on the functionality being added instead of waiting for another whole application and discovering that I still did not want to use it.

But smaller changes made the cost of waiting harder to ignore. I wanted to try what had changed, explain what felt wrong and work through the next adjustment with the agents. A long build-and-test cycle added waiting to that process. Making the checks faster mattered because I wanted to spend more of that loop using and shaping the product, not waiting for another verdict.

Measure the actual wait

For the rebuilt product, we also worked on the local test runner. These were different runs from the earlier behavior-checking CI: one recorded run of forty shell scenarios passed, but took almost fifty-seven minutes. Work around the tests was repeating: builds, launches of Cargo, Rust’s build tool, and environment preparation. We changed the runner to prepare once for the selected work rather than keep paying those costs around individual scenarios.

Even an unchanged file could create unnecessary work. In a separate checkout issue, rewriting files whose contents had not changed invalidated the information used to decide whether a build was still current. Fixing that meant preserving unchanged state, not relaxing the requirement to rebuild when the inputs really changed.

With preparation shared, the runner could execute selected work through a bounded pool of workers. In a separate local experiment comparing execution width, a single execution slot took 541.3 seconds; three slots took 262.6 seconds. That was roughly half the elapsed time for that experiment, with one sample at each setting. It did not tell us how much faster the whole development cycle had become.

Saved validation compared the selected scenarios and their counts, named failures and refusals between serial and parallel execution. A refusal to run unsupported work had to remain a refusal, not turn into a faster green result.

Yet I was still waiting longer than the reported numbers suggested. I challenged a roughly three-and-a-half-minute figure against a wait of more than nine minutes. In a separate fresh-build record, a suite of 630 scenarios took 269.6 seconds to execute, while roughly 323 seconds elapsed before its first folder started. More time had passed before the tests began than the suite itself took.

The suite timer had become a poor stand-in for the wait I cared about.

To find the missing time, record when a change becomes ready for checking, when a worker picks it up, when preparation finishes, when tests begin and end, and when a usable verdict returns. Those measurements distinguish a slow suite from work waiting to start, rebuilding its environment or sitting between completed checks and a decision. Our records exposed some of those intervals; they were not a complete measurement of every stage.

Then change the part that is actually consuming the time. If preparation repeats, establish what can be prepared once and what must invalidate it. If execution dominates, identify work that can run independently, isolate resources the workers would otherwise share, and try a bounded increase in parallelism. Compare both the elapsed time and the answer.

We also had to make sure the runner and CI agreed on what should run. The runner could discover scenario folders, but CI maintained a separate list of six scenario jobs plus a build job. A newly discoverable folder did not automatically become part of that list.

We replaced those jobs with one that selected work through discovery from the project root, keeping two code-convention checks separate. Exclusions became explicit and visible. An empty runnable selection was refused rather than allowed to produce a meaningless success. This removed a second list to maintain; we did not measure a separate speedup from that change.

A newly added scenario is a useful test of the consolidation: verify that CI selects it or reports why it is excluded. Compare the selected work before and after the change, and keep failures and refusals visible. A later path that ran without an interactive interface excluded window checks; it could not stand in for those interactions merely because it finished.

With the scope of the checks explicit, we could return to a large, specific part of the wait: the phone build.

Where we are now

In the rebuilt product’s CI pipeline, one inspected job lasted nineteen minutes eighteen seconds. Its clean phone build took nearly fourteen minutes: about 71% of the job. The test suite took roughly two and a half. Even eliminating the suite’s entire runtime would have left most of the wait.

The next question was whether every change needed that clean build.

We added a decision based on the inputs that determine it. When the tracked inputs matched a recorded, complete successful result, the job could skip the clean phone build. When they changed, or the previous record was missing or malformed, it could not rely on that result and had to do fresh validation. We were making the expensive work conditional, rather than making an identical build execute faster.

The difficult part of that decision is knowing what counts as an input. Looking only at the application’s own files is not enough. A dependency can change underneath unchanged application code, including through another dependency. A previous success then describes a different combination from the one we are about to use.

To test that boundary, leave the application’s source unchanged, change a dependency version, and confirm the system does not reuse the old result. Also remove or corrupt the previous success record and check that uncertainty leads to fresh validation, not an assumed pass. An interrupted or failed attempt must not create the complete-success record that later work relies on.

Saved test cases for our selection rule cover these boundaries, including direct and indirect dependencies, versions and invalid records. Those cases were inspected for this account, not rerun. They identify what the shortcut must preserve; they do not establish that every possible input has been captured. The dependency record itself becomes something the team must maintain as the build changes.

The six successful jobs recorded after the change averaged about eleven minutes twenty-one seconds. Three shorter jobs averaged about three minutes twenty-two seconds; three longer ones averaged about nineteen minutes twenty-one seconds. Direct inspection confirmed that two of the short jobs skipped the clean phone build and one long job performed it. The other three are grouped by duration, not individually verified build steps. These are job start-to-finish times, without queue waiting.

That mixture is the useful result to report. A team making many changes that leave the build inputs untouched can see a different average from one frequently changing them. Reporting only the fastest run would conceal when the expensive work still has to happen; reporting only the combined average would conceal what changed.

We do not yet have a measured account of the compute saved or the maintenance effort required to keep this decision trustworthy. What we have is a way to return an earlier answer for qualifying changes without treating every change as if it qualifies.

What the time is for

The New Space review still had questions about focus, Escape and duplicate submission. A faster build does not answer any of them. It can give us another chance to investigate one before asking the agents to build something else.

Take Escape. We can decide what should happen when someone backs out of creating a space, try that interaction, and compare what the application does with what we intended. If it is wrong, the next piece of work has a clear purpose. If it behaves as specified but still feels awkward, the specification needs another look. That is a much smaller disagreement to resolve than discovering, after another whole-app rebuild, that the experience still feels wrong.

This is where I want the time back: seeing what the agents built, trying it and being able to say what does or does not feel right. The answer may change what we ask for next. I do not need the agents to fill every saved minute with more implementation; I want the conversation about the experience to happen while we can still shape it.

The useful next step may be another feature. It may also be staying with the small interaction in front of us until we have something worth building on.