Tests under the floor: finding the real cost of a slow suite

The problem

The Corpus test suite had become slow enough that waiting for it shaped development. A broad suspicion was that the tests were simply written badly. That explanation was too easy: the suite mixed small tests with larger checks that built realistic working environments, and its performance changed with the host and the way tests were distributed.

The Human asked for a closer look at the process. The product question was where to spend effort so feedback could arrive sooner without discarding checks that protected important behavior. Speed alone would be a poor success measure if it came from reducing what the tests could catch.

The agents’ investigation

Several agent holders continued the investigation across sittings. They profiled representative runs, separated the cost of ordinary tests from a small number of expensive fixtures, and revisited conclusions when a measurement turned out to describe fewer workers than its label implied. They examined test setup, repeated data creation, source control calls, and contention between parallel workers. They also reproduced a regression in one benchmark fixture and repaired the test setup that caused it.

The work led to focused changes rather than a blanket rewrite. Agents reduced repeated setup work and improved data initialization while preserving the checks’ purpose. They identified integration checks that should be understood as larger exercises, rather than judging them by the expectations of a small unit test. They kept wrong turns in the private measurement account so a later investigator could see why a tempting explanation had been rejected.

What changed

The source account reports a later green health reading and considers the suite investigation paid for its period. The measured suite was subsequently archived as the project changed where tests run. The record therefore supports a historical result: specific bottlenecks were found and reduced, and the team gained a more accurate map of the remaining cost. It is not a live benchmark of today’s checkout.

The strongest lesson is about judgment. A suite can be slow for several different reasons at once, and the longest test is not always the mechanism worth redesigning. This investigation made the cost visible enough to choose local repairs and to leave other paths alone when their likely gain was small. It did not prove that every test was necessary or establish a universal target for test speed. A fair comparison would need the same code, host conditions, test selection, and correctness standard.

Editorial note: This is a newly authored public summary grounded in the private Corpus work account of the same name, not a verbatim historical record.