Beyond Quality

An open-source community for deeper enquiry.

Podcast

The erosion thesis

Part of Governance of quality.

Between the governing body and the two functions, product and quality, runs the interface described on interface.md. The top publishes the statement, the short document with the bets, the minimums, the common rule and the risk appetite (value.md). Management decides the everyday cases inside its numbers, five kinds of reports come back up, and at each reevaluation the top corrects the numbers and remakes the bets. People do every part of this, and they do it on two kinds of knowledge about the company’s software: how it works, and why it is the way it is. implementations.md describes what coding agents do to that knowledge, under the ai-era-testing research’s names for the two losses: comprehension debt for the how, intent debt for the why.

This page is about what those losses do to the governance system itself, and about who has to mandate and pay for the countermeasures. The interface keeps the statement’s numbers current. Keeping the people able to run the interface is the other half of the “maintain” the research promises in its purpose statement.

What the interface runs on

implementations.md follows the two debts through the four-step loop of Economics of Testing: in step 1 the managers can no longer say what a failure would cost, and in step 3 QA can no longer say whether a check is credible evidence. The reports carry the same loss to the top. In the refund flow that page follows, a risk size built on tests generated from code nobody has read rests on nothing, and it reaches the top in the same units as every well-supported number, so the top cannot tell the two apart. The assurance check on interface.md traces a reported number back to the raw material behind it, and that check needs a person who can still read the material.

The decision rule on interface.md presupposes the same people. Under the statement a team facing “ship now or test more” works the case out from the published numbers and decides alone when it fits inside the appetite. Reinertsen’s Boeing 777 example makes the same point: any designer was authorized to increase unit cost by up to $300 to save a pound of weight, and five thousand engineers made system-level trade-offs without asking their superiors for permission (The Principles of Product Development Flow). The rule spared the designers the asking, and it did not spare them the judging: each still had to know what their change did to weight and to cost. A published statement spares the team the meeting, and it still needs a team that can say what a failure would cost and whether the checks would catch it.

The value work runs on the same knowledge. The value model on value.md is a set of decisions: which needs get a bet, what level to hold, what holding it is worth, and the top remakes those decisions at each reevaluation. The ai-era-testing research observes that intent debt shows itself when decisions need to be made (research question), and a reevaluation is a set of exactly such decisions. When the top asks whether the fraud rule in the refund flow still protects anything worth its cost, the answer has to come from someone who knows why the rule is there. In the research’s second scenario, which implementations.md retells, the team can describe every line of the code and cannot say that. The bet on that need cannot be checked, and a proposal to retire the rule cannot be judged.

A loss of a different kind

The managers handle a new risk in the ordinary way. When the company adds a second payment provider, say, they write the new risk down in step 1, rank it in step 2, QA covers it in step 3, and its size appears in the next report. The register grows by a row and the work goes on as before. The two debts are a loss in the people who do that work. implementations.md reads intent debt as the wearing away of direction and comprehension debt as the wearing away of oversight, with accountability staying where it is. The standard’s own definition says why that is a loss of a different kind: effective oversight needs a general understanding of the situation (38507 3.2.1, Note 1). The body’s oversight of AI is to rest on a general understanding of how the system is used (6.2). A body whose people no longer understand the software has the same register as before and fewer and fewer people who can tell it what the rows mean, while it still answers for the whole. Its question gains a second half: what are our risks, and can our organization still know them.

The people inside the work cannot see this loss from where they stand, for three reasons.

Two ways a product gets worse

Products get worse in two ways, and the debts belong to the second. A company can decide to lower a quality level, to save the money or because its customers cannot leave. Under the statement that is a bet that lowers a level, with an owner, a return and a review date, so the company at least knows what it chose (value.md). Or nobody decides: a thousand small trade-offs go the same way under deadline pressure, each too small to escalate, and the product decays without a decision anyone could point to. The problem statement’s first two observations describe that second way one decision at a time. Agents add a trade-off of the same kind, made by every developer every day: how much of the work to hand to the agent in one piece. The tools are sold on what one prompt can build and the public benchmarks score the same thing, so the default is set outside the company, and inside it nobody has to decide anything for the debts to grow.

The floor under the recoverable debt

The research holds comprehension debt to be recoverable and intent debt to compound, and implementations.md follows it there. The recovery has a condition the research puts among the four it sets on any answer, keeping comprehension sufficient for supervision, intervention and evolution: somebody has to be able to read the code back. A tester who spends long enough with the refund code learns how it works, if the tester can still read code. The standard has a name for the loss of that ability, “agent atrophy”: people in roles where AI does the routine work are exposed to less and less experience, and their skill hollows out (38507 6.7.4). Agent atrophy is the floor under comprehension debt. While the skill lasts, the debt can be paid off at a cost. Once the skill has gone, nobody can pay that cost, and the recoverable half of the loss stops being recoverable. Intent debt has no such floor, because no skill was ever involved: the why was not written down, and no amount of reading brings it back.

The tool changes underneath

implementations.md notes a difference the ai-era-testing research does not cover: the agent changes while the organization uses it, on the vendor’s schedule, so the assurance over the tool has to be repeated each time it changes. The standard for the governance of IT adds that adaptive AI systems can change the appropriateness of the assurance mechanisms in place, and that more than one mechanism may be needed (38500 7.2.7). One mechanism the company can keep in its own hands is a suite of checks (calibrated evals, in engineers’ terms) its people write and maintain, run against the agent whenever anything in the tooling changes: the same instructions, the same expected results, written by a person who knows what a right result looks like. The suite tells the company whether the tool still behaves as it did, and gives the body a number it can watch over time. Three limits go with it. The expected results have to come from people, because a suite whose answers a model wrote passes while proving nothing, like tests generated from the code they test (lifecycle drift). The suite covers the tasks somebody thought to put in it, so a new kind of work is outside it. And what counts as a right result changes with the product, so the suite decays unless people keep it. Keeping it is a prevention cost like the rest: a body of intent written by people, in test form.

The suite also measures the people. In the building sessions the team accepts each of the agent’s steps or sends it back, and how often it sends steps back can be counted. Suppose that over a year the team sends back one step in twenty where it used to send back one in five. Either the agent has got better, or the people have got worse at telling a bad step from a good one, and the count alone does not say which. The suite does. Its expected results were written by people at a time when they could still tell, and they do not change when the people’s judgment does. Suppose the suite holds the refund rule: for a $600 refund, the expected result is that the request waits for a manager. If the agent’s code now pays such a refund out at once, the suite fails it, and if the team accepted that same code in its sessions, the team has stopped seeing what the suite still sees. The suite has caught the people as well as the tool. No other instrument on this page measures the erosion that directly, and this one works only while someone can still write the expected results.

The size of the risk to knowledge

Under the statement the risk of losing this knowledge is one kind of risk among the others. The top sets an appetite for it, management reports its current size next to that appetite, and a warning comes before a breach (interface.md). Its size cannot be read from self-report, so the body has to ask for indicators. The research proposes some for its own experiment (proposal): how long the team takes to diagnose a fault in code it produced, and whether its members, including those who joined later, can say why each major design choice was made. The first reads comprehension and the second intent. The suite above adds a third, the team’s rejection rate against fixed results, which reads the people’s calibration. Engineering’s numbers in Part 2 of the statement already carry the late version of all three: how long changes take, rework, regressions (value.md). A single report can look fine, two years of trend cannot.

Who mandates and who pays

implementations.md already describes the countermeasure the ai-era-testing research proposes: a building session in which two or more people decide each step while an agent writes the code, which gives the standard’s oversight clause the people it requires. It also says what the session costs. Before agents, people learned the system by doing the work, and nobody paid for the learning; now the learning has to be bought on purpose, as people’s time in the session, as the writing down of what each change was for, as the training of people who can supervise the agent. In the four categories of quality cost the economics research uses, that spend is prevention, money spent so that the problem never arises, and the debts are what accumulates when nobody spends it. What that page leaves open is who pays.

Nobody below the top will pay it on their own, and the reasons are the incentives every function already has. The product manager gets the feature sooner when the developer hands it to the agent whole. The engineering manager answers for delivery this quarter, and the extra diagnosis time the debt causes is paid next year. The QA lead cannot fund a product person’s hours in building sessions out of a testing budget. The developer who hands over a whole feature is using the tool as it was sold. Each of them is right by their own numbers, and the sum is the loss described above. This is the speed-quality fight of the problem statement with the same structure: the win visible now on one side, the cost later and elsewhere on the other, and nobody below the top judged on the balance.

The standards put four acts on the governing body. The first is to name the practice, how the company’s software is built with agents, as something it governs. implementations.md shows that until then nothing in the oversight clause attaches to it. The standard puts the scope of automation under the governing body’s oversight, to be implemented by authorized and skilled people, with the consequences examined and understood before implementation (38507 5.5).

The second act is the conditions on the people who use the agent, the ones the standard sets on any human using AI and implementations.md lists (38507 6.2). The standard also wants decision-makers skilled and trained for the decisions they are responsible for (6.3). When a developer hands the agent a whole feature, the agent decides things, the fraud rule among them, that nobody authorized the developer to decide. The standard for the governance of IT asks that a tool not give its user more power than the user’s authority (38500 7.2.5). The size of the step is therefore a delegation limit the body can set: what may be handed to the agent in one piece, and what people have to decide.

The third act is the appetite. The risk of losing the knowledge gets its entry in Part 6 of the statement like every other kind of risk: how much of it the company accepts, why, and the size of problem above which management escalates (value.md).

The fourth act is the money. ISO 37000 lists the conditions under which delegation works, and one of them is that the required resources are available; it adds that nobody should be held accountable for matters over which they have no authority or for which expectations were never stated (4.2.2). A body that delegates building software with agents to management and does not pay for the people who keep understanding the software has delegated without meeting that condition, and management cannot be held to account for knowledge it was given no means to keep. The standard for the governance of IT asks the body for assurance that its intentions are translated by personnel with the appropriate competence (38500 5.6.2). It also asks for assurance that the organization has the knowledge and skills to implement change in time (5.8.2). Under agents both assurances have a price, because the competence stops accumulating on its own. The body pays it as the bet implementations.md describes, and at the review date it puts the indicators above next to the estimates.

The management level has its own standard for AI, ISO/IEC 42001, written for top management and the organization; the governing body is not its addressee. It requires the organization to determine the competence needed by the people whose work affects its AI performance, to ensure they have it, and to keep evidence of it (7.2). It requires that those people know the AI policy and their part in it (7.3). It requires the organization to document the people and competences used at each stage of an AI system’s life, verification included (A.4.6). Its guidance names roles for human oversight among the necessary human resources (B.4.6). It asks that the people who do the overseeing be informed and trained, and have the authority to override the system (B.9.3). Those are the places at the management level where the body’s conditions are written down and audited. What is written there is the body’s direction: which competence, over which software, held by whom, and paid for from where. Nothing in the management standard tells the body to require comprehension of the company’s own software as a competence, and nothing in it will keep that competence if the body does not.

What not paying costs

The debts’ first bill fits the economics research’s four categories of quality cost: inspection bought after the fact, rework, and failures that reach customers because nobody could tell right from wrong. The second bill fits none of them. value.md counts a return that comes from the system itself: when the system stays quick and safe to change, every future plan is cheaper to pursue, the next feature ships sooner and the response to a competitor takes weeks. That return is the first thing the company loses when nobody understands the code. The research’s cost table puts finding what to rewrite among the most expensive steps once nobody understands the code (with agents), so every future change costs more, and every future delay with it. The decision rule on interface.md has product with engineering work out what postponing a feature costs, case by case. Under comprehension debt that number rises for every feature the company will ever ship, and no incident report will show it. It shows in the numbers engineering already answers for in Part 2 of the statement, how long changes take, rework and regressions (value.md), and the top sees it only when it reads those numbers as a trend.

Maintain

The research promises in its purpose statement that organizations gain and maintain clarity on the economics of their quality decisions. The interface keeps the numbers current: bets checked, estimates corrected, the statement refreshed. Once the people who run it can no longer say what a failure would cost or whether a check would catch it, the numbers they correct rest on as little as the evidence behind them. Keeping those people able is the governing body’s act, in four parts: the practice named as governed, the conditions on the people, the appetite for the risk to knowledge, and the money, checked at review dates against the indicators above. That is the fourth capability the problem statement promises, adoption at a pace the organization can sustain. What each role does with this, the body, the executives, and QA and engineering leadership, is on the practice page.