The Amnesia Cycle and Why AI Is Turning Developers Back Into Testers

Leave a reply

I started working in technology around 1993. One of my first jobs was in quality assurance, partly because there was no security profession to join yet.

There were people doing the work, but few companies were hiring for it. That changed within a decade. Until it did, people with the instincts that would later define security engineering landed in adjacent disciplines. Test was one of them.

That was true for me. I became a test manager fairly quickly, later worked as a test architect, then went on to software development, security, and a bunch of other things.

More than thirty years later, I find myself watching something funny happen.

AI is turning software developers back into testers.

Not the kind of testers we were in 1993. What it means to test software has changed several times since then. But at a more abstract level, the work is surprisingly familiar.

The person is no longer primarily producing the thing. They are increasingly trying to determine whether the thing that was produced is any good.

And we have been around this loop before.

We develop a specialty because a problem is hard. We get good enough at it to encode pieces of the expertise into process, tools, and automation. Eventually the machinery works well enough that the underlying expertise starts to look unnecessary. We distribute the responsibility, automate more of it, and convince ourselves that the problem has largely been solved.

Then the system grows, the environment changes, or a new technology arrives and exposes all of the judgment that never made it into the machinery.

The responsibility never left. It just changed costume.

When testing was a profession

In the 1990s, software development and software testing were much more clearly separated. Developers wrote software. Test organizations tried to figure out where it broke.

A lot of the work was manual. People installed builds, exercised features, constructed strange states, tested boundaries, wrote bug reports, and tried to reproduce failures. By modern standards, much of it would look labor-intensive. Some of it really was scut work.

But the separation had an important organizational property. The person who built the thing and the person whose job was to find out why it was wrong were different people.

Their incentives were different too. Developers were trying to make the product work and get it shipped. Testers were rewarded for finding the circumstances under which it did not work.

Those are complementary responsibilities, but they are not the same responsibility.

Over time, testing acquired a status problem. It was increasingly treated as work that did not require the same level of engineering skill as implementation. One response was to give more of it to junior developers.

Then we moved further. Instead of a separate organization owning quality, developers would test their own software.

Much of that change was good.

Unit testing was good. Test-driven development was good. Continuous integration was good. Automated regression testing was good. Testing closer to the point where software was written eliminated entire classes of expensive downstream failures.

The nature of testing changed too. It moved away from a model dominated by manually exercising a finished product and toward one where tests could become part of the way software itself was specified and constructed.

So the story is not that we eliminated QA and that was simply a mistake.

The mistake was gradually convincing ourselves that because we could automate more of the mechanics, we needed less of the judgment.

A test that passes is not evidence that the system is good

As automation improved, we became extraordinarily good at running tests.

A modern software project can execute tens of thousands of tests on every change. We can measure coverage, reject regressions, test configurations, fuzz interfaces, spin up entire environments, and tear them down again without anybody touching them.

The machinery works.

But there is a distinction that became easier to overlook.

A test can work perfectly and still tell you almost nothing useful about the quality of the system.

The hard question is not always whether the test passed. It is whether passing that test is evidence of the property you actually care about.

That gets harder as systems become more complicated. A system can have an enormous green test suite while failing in a way nobody thought to represent in the suite.

There is an important difference between mechanical verification and judgment about what deserves to be verified. One asks whether the checks we wrote passed. The other asks whether those were the right checks.

Automation became exceptionally good at answering the first question. It did much less to eliminate the difficulty of the second.

A test suite is an encoding of somebody’s model of how the system can fail. It captures the failures we anticipated, the properties we chose to represent, and the assumptions we knew enough to challenge.

It says much less about the failures nobody imagined.

Automation did not eliminate the test problem. It moved the test problem up a level.

The scarce skill became figuring out what to test, what failure looks like, which assumptions need to be challenged, and what evidence should actually make us confident in the result.

That is why what is happening with AI feels so familiar.

We thought AI would do the testing

One of the obvious expectations around generative AI was that it would automate still more testing.

If the AI can write the implementation, it can certainly write unit tests too. And it can.

But that misses the more important change.

AI is making implementation cheap.

A developer can already cause far more code to be produced than they could reasonably have written themselves. As agents improve, that multiplier gets larger. It is not difficult to imagine one engineer directing dozens, hundreds, or eventually thousands of concurrent software-producing processes.

At that point, traditional code review is not merely inefficient. It becomes physically impossible.

Nobody is going to carefully read every line produced by a thousand coding agents.

The generated output is also not quite like the output of an old deterministic compiler or code generator. These systems are probabilistic. Run them again and you may get a different implementation, a different decomposition, or a different mistake.

That makes the old assumption that we can inspect the artifact into correctness even less plausible.

So the human moves up a level.

Instead of spending most of the time constructing the implementation, we increasingly construct the conditions under which an implementation will be accepted. We write tests, evals, invariants, acceptance criteria, adversarial cases, and constraints. We decide what evidence is sufficient to tell us that the machine-produced result is actually good.

Software developers are becoming testers again.

It is not 1993-style manual QA. It is closer to specification, TDD, evaluation design, systems validation, and the construction of executable evidence.

But abstractly, it is the same work happening in a different way.

The form changes. The responsibility does not.

What the new testing actually looks like

The easy question is whether generated code passes the existing test suite.

The harder question is whether the suite represents the properties we actually care about.

Does the generated component behave correctly under inputs we did not anticipate? What happens when independently generated components interact and two locally correct decisions compose into a globally bad result? What happens after the system has been operating for days, accumulating state and acting on the consequences of its own earlier decisions?

What happens when the environment differs from the conditions represented in our evals? And what happens when an adversary deliberately searches for the assumptions we failed to encode?

Those questions are not answered by producing more tests mechanically.

They require somebody to form a theory of failure.

The scarce skill is increasingly understanding the mechanics of failure. That means knowing where the system boundaries are, which properties must remain true across those boundaries, what assumptions are hidden inside the architecture, and how reasonable local behavior can produce unreasonable global outcomes.

It also changes what a useful test looks like.

When implementations are relatively stable, testing specific examples can tell you a lot. When an AI can regenerate the implementation tomorrow, durable properties become more important. The question shifts from whether this implementation produces the expected result for this input toward which properties must remain true across whatever implementations the system produces.

Those properties might concern correctness, authority, state transitions, isolation, safety, or the relationship between components. The implementation can change while the invariant remains.

That is a different kind of leverage.

Verification also stops neatly ending at release. Some failures only emerge through interaction with real environments, long-running state, changing inputs, or behavior that was not represented during development. That pushes part of the evidence gathering into operation through telemetry, observability, runtime checks, and the behavior of the deployed system itself.

The new tester is therefore not just checking whether an implementation conforms to a specification somebody else already wrote.

Increasingly, they are responsible for deciding what the specification must say, which properties must survive implementation changes, which failures matter enough to detect, and what evidence is sufficient for the resulting system to deserve trust.

That is a substantially higher-order form of the same old responsibility.

High-quality scut work

There is a funny status inversion hiding in this.

Test used to be treated as scut work. Then we gave more of it to junior developers. Then we told every developer they were responsible for doing it themselves. Then we automated as much of it as we could.

Now we are automating the part we used to think was the prestige work, writing the software, and leaving the developer sitting above the machinery trying to determine whether any of what it produced is good.

We may have turned one of the industry’s prestige jobs into very high-quality scut work.

The funny part is that the scut work may now be where much of the value lives.

When implementation is expensive, the person who knows how to implement something is scarce. When implementation becomes cheap, the scarce person is the one who knows what should be built, what properties it needs to have, how it is likely to fail, what assumptions are hidden inside it, and what evidence would convince us that it works.

Generation gets cheaper. Judgment does not.

Security already did this

I have seen almost the same cycle happen with security.

In retrospect, it is probably not an accident that so many early security people came through test. Both disciplines train you to look at a system somebody else believes works and ask what they have failed to consider.

Testing asks how the behavior can violate what was intended. Security asks how trust, authority, or assumptions can be violated even when the system appears to be functioning normally.

Both reward a certain kind of skepticism.

When I started, security engineering barely existed as a normal software profession. By the late 1990s it was becoming one. By the early 2000s, it was clearly a distinct industry with dedicated teams and career paths.

That specialization happened because security was hard and ordinary development organizations were not consistently good at it.

Then we started saying something that was also fundamentally correct. Security should not be something another group bolts onto the product afterward. Developers should build secure systems themselves.

“Security is everyone’s responsibility.”

There is nothing wrong with that principle.

The problem is what happens when we confuse responsibility with expertise.

A product developer is trying to ship a product. They have schedules, features, performance requirements, compatibility issues, reliability problems, customer demands, and dozens of other things competing for attention.

Security becomes one of many things they are supposed to get right.

When that proves insufficient, organizations build machinery around the problem. We add secure development lifecycle processes, scanners, policy gates, paved-road platforms, and controls designed to make the safe thing easier than the unsafe thing.

All of those things can help.

But notice what we are doing.

After deciding the specialist function should be distributed into the rest of engineering, we are encoding pieces of that specialist judgment back into systems and processes.

A scanner encodes somebody’s knowledge of what a vulnerability looks like. A secure-by-default platform encodes somebody’s judgment about which choices should be permitted. A policy gate encodes somebody’s model of what conditions need to hold before software should be released.

The specialist may become less visible as that expertise gets embedded into the platform, but the expertise did not cease to exist. It became infrastructure.

Eventually there is enough machinery that it again becomes tempting to ask whether we really need the specialists.

Then the abstraction leaks.

A threat appears outside the model encoded in the scanner. A platform assumption no longer holds. A new system does something the existing governance framework was never designed to reason about.

Then we discover that we automated the known answers, not the ability to recognize new questions.

AI security and the latest rediscovery

AI makes this pattern almost comical because we are currently rediscovering old classes of security problems with new names.

Prompt injection is obviously not literally SQL injection. The implementation is different, the interpreter is different, and the failure modes are different.

But someone who spent the 1990s and 2000s dealing with SQL injection, command injection, script injection, confused deputies, trust boundaries, privilege separation, and the consequences of letting untrusted input become control should find the family resemblance hard to miss.

We have spent decades learning that you should be very careful when information from an untrusted party can influence what a privileged system interprets as instructions.

Now we have built enormously capable interpreters whose primary interface is natural language. We mix instructions and data in the same context, connect them to tools, and act surprised when hostile input changes what they do.

The technology is new. The institutional failure mode is not.

The same thing is happening in conversations about containment and sandboxing. We are once again discovering that powerful systems need boundaries, that those boundaries need to be enforced rather than merely described, and that capabilities should be constrained by something stronger than an instruction asking the system to behave.

None of that makes AI security trivial. The new systems create genuinely new problems. But novelty at one layer does not erase the accumulated lessons at another.

We knew versions of these things thirty years ago. What keeps recurring is not the exact vulnerability. It is the belief that a new abstraction has somehow relieved us of the old responsibility.

That is where the amnesia comes in.

The amnesia cycle

You could describe the pattern as specialization, codification, automation, perceived redundancy, loss of judgment, scaling failure, and rediscovery.

That is the amnesia.

We rarely forget the artifacts of the previous generation. We keep the test frameworks, scanners, development processes, controls, and automation.

What we forget is why the people who created those things thought the problem was hard in the first place.

Eventually we confuse the existence of the machinery with possession of the expertise that created it.

A passing test suite becomes evidence of quality. A security scanner becomes evidence of security. An AI eval becomes evidence that the AI is doing the right thing.

Until the system moves outside the assumptions those mechanisms encode.

Then the old problem appears again, wearing different clothes.

The responsibility never leaves

That is the through line I see between testing, security, and what is now happening with AI-assisted development.

We are very good at abstracting away mechanics. That is what engineering does.

But when we successfully automate the mechanics, it is easy to convince ourselves that we automated the underlying responsibility too.

We did not.

Testing did not disappear when the dedicated QA organization disappeared. Security did not disappear when we made it everyone’s responsibility. Verification will not disappear because an AI can generate both an implementation and a test suite that says the implementation is fine.

Somebody still has to decide what “fine” means. Somebody has to recognize the assumptions the automation does not know it is making. Somebody has to decide which failures matter. Somebody has to determine what evidence would falsify the claim that the system is working.

Somebody has to distinguish a system that successfully passes its tests from one that deserves to be trusted.

The responsibility never leaves. It changes costume.

I started my career in a world where developers wrote the software and people like me tested it. We spent the next thirty years treating test first as work developers should not have to do, then as work developers should do themselves, and finally as work machines should increasingly do for them.

Now the machines are starting to write the software.

And the developers are increasingly responsible for figuring out whether any of it is good.

Apparently the tester won.

Hurst University

Leave a reply

For as long as my children can remember, I have told them that they are students at Hurst University.

It has no campus, no accreditation, and no admissions process. Joining the family is enough to get you enrolled. Graduation is another matter.

There is only one requirement. By the time you leave the house, you should be capable of building a future for yourself.

I don’t mean that you should know what you are going to do for the rest of your life. That is one of the stranger questions we ask young people. Most adults I know have changed direction enough times that pretending an eighteen-year-old is making a permanent occupational choice is unserious.

What I mean is more fundamental. You should know how to learn, and you should know how to work. When you encounter something you don’t understand, you should have some idea how to get from ignorance to competence. You should know how to take a hit without deciding the hit defines you, how to recognize when your assumptions were wrong, and how to change direction without treating everything behind you as wasted.

Most importantly, you should increasingly understand that the responsibility for what happens next belongs to you.

That is Hurst University.

There is nothing new about the idea. If anything, it is an old model of education with a family name attached to it.

For most of human history, becoming educated and becoming useful were close to the same thing. Children watched adults do real work. Then they helped. Then they were trusted with some small part of it. They made mistakes where mistakes were survivable, were corrected by somebody who knew more than they did, and tried again. As competence increased, the work got harder and supervision got lighter. Eventually the person who had been taught became somebody who could be trusted to act without waiting to be told what to do next.

That transfer is the part I care about most. The student eventually has to become responsible for the education.

I learned that the hard way, and then I had to watch one of my children learn it too.

He was not slow. He was seeking out serious material for pleasure at an age when adults found it surprising. But the way he took in information did not match the way the school delivered it, and the instruments the school trusted returned the wrong answer about him. The system then acted on the wrong answer. He began to notice he was being handled as a different kind of student, and he could not work out why.

The part I remember is not the meetings. It is that he started to wonder whether the school knew something about him that he didn’t.

I eventually moved him somewhere more willing to respond to an individual child, which helped. The more important intervention was a conversation.

I told him what I believed to be true. No institution was ever going to be able to take full responsibility for his education. Some things were going to be harder for him than for other people, and that was not going to change.

Then I told him that his challenges were his superpower.

I meant it literally, and I explained why. Most people are never forced to learn how they learn. They get through on the method they were handed, and they only discover its limits much later, if ever. He was going to have to build his own method starting now, at nine, because the handed one did not work for him. That is a brutal assignment and it is also an enormous head start. The world keeps rewarding people who can acquire what they need without being given it, and he was going to be practicing that while everyone else was still being taught.

Then I told him he was a student at Hurst University, and that when he graduated from this house he was going to be fine.

He is grown now. In his twenties he built a business and sold it to one of the largest financial services companies in the world, which is a thing you cannot do without becoming a fast and relentless student of whatever is in front of you. Entrepreneurs are not people who already know how. They are people who find out in time.

He is one of the sharpest people I know and one of the most prepared, and the second of those is the one he built. He worked out early that being the smartest person in the room and being ready for the room are different things, and that only one of them was under his control. He stopped waiting to find out which he would get. That habit came out of the friction, and I am not sure anything else would have produced it.

I was a student at Hurst University long before I had a name for it.

I was dyslexic and dysgraphic and did not present well to the educational system. At one point my parents were told they should prepare themselves for the possibility that I would never be capable of supporting myself.

That prediction did not age well.

I went to college young and moved out at sixteen. Some of that was rebellion, but mostly I wanted independence in the literal sense. I wanted control over my own life, including the economic responsibility that came with it.

A lot of my education after that happened without anybody designing it. Computers gave me problems I cared enough about to solve. Programming led into systems. Systems led into networking. Networking led into security. Security eventually required understanding companies, incentives, law, economics, organizations, and people. I kept encountering things I did not know and learning enough to get through the next door.

One capability created a reason to acquire another.

None of it started with me.

My father grew up on a subsistence farm, where you fix what breaks with what is in the barn because the alternative is doing without. He taught himself rocket chemistry as a kid off that same principle and was working on satellite hardware by his early twenties. I have written about him before, so I will not tell it twice.

What matters here is that he never called any of it an education. It was just what you did when you needed to know something.

So I did not invent this. I inherited it, gave it a name, and made the handoff deliberate. That last part is the part that matters, because none of it moves on its own. It has to be handed over on purpose, by somebody who decides to bother.

Years ago, when I wrote about apprenticeship, I described four things that had mattered enormously in my own development: access, direction, challenges, and support. I still think that framework is right, but I now see something underneath it. Those are not merely the ingredients of a good apprenticeship. They are the ingredients of an environment that gradually teaches someone to direct themselves.

Give somebody access to things worth learning. Put people around them who know more than they do. Give them problems slightly beyond their current ability. Support them enough that failure remains recoverable. Then, slowly, stop telling them what to do next.

That last part is the actual transfer.

Having three children made this clearer, because the same philosophy looks completely different depending on the student.

My second child has always worked. At four he was doing long division and by 6 or 7 he could recite the major bone and muscle groups. As a teenager he became a nationally ranked fencer, which is not something that happens to a person for being quick. It happens through years of drilling the same movement badly until it is good, losing in front of people, and going back the next day.

So the story I used to tell myself about him, that things came easily and he had therefore never built the habit, does not survive the evidence. He built it early. He built it in a domain nobody assigned him.

What was true is that school rarely asked him for it. He was admitted to several of the best aerospace programs in the world and ultimately chose computer engineering, and he is now somewhere the standard is set by people who are also very good and where thinking fast is the baseline rather than the edge.

So what I am watching is not a young man learning to work. It is someone moving a work ethic he already owns out of the place he first built it and into the place he intends to live. That transfer is the entire point of this essay, and he is doing it in front of me.

My third is different again. Nearly all of her identity is currently organized around a single competitive sport, roughly twenty-six hours a week of it. That is not a complaint. Twenty-six hours a week of anything difficult teaches repetition, correction, pain tolerance, delayed gratification, and performing while people watch. The work ethic already exists.

The problem is that she believes the work ethic belongs to the sport. Her brother has already shown that it doesn’t, which is the most useful thing an older sibling can do.

Activities end. The machinery that produced the excellence should travel.

Aptitude changes the educational problem. It does not remove the educational problem.

A child who struggles may need to learn that difficulty is not the same as inability. A child who rarely struggles may need enough friction to discover the value of preparation. A child who becomes excellent in one domain may need to discover that excellence is partly a process that can be carried elsewhere.

The curriculum changes because the student changes. The goal does not.

The goal is agency. By agency I mean the ability to encounter something unfamiliar and say, with some credibility, “I don’t know how to do this yet, but I know how to begin.”

That may be the most durable thing an education can produce.

My father was not the only one. My mother was doing the same thing in a different register, and I was even slower to see it.

She started as a hair stylist. Later she became a tool-and-die worker at Boeing. Then she moved into knowledge work and eventually retired as a business analyst.

Described by job title, those look like unrelated careers, the sort of résumé that gets read as drift. Described by behavior, they are the same story repeated several times. She kept becoming qualified to do things she had not previously known how to do.

Nothing about cutting hair prepares you to hold tolerances on a machined part. Nothing about machining prepares you to take apart how a business actually works and put it back together as a requirement. What carried across was not the content. It was the practice of arriving somewhere without the necessary knowledge and acquiring it in public, in front of people who already had it, while the work still had to get done.

She did that at least three times, each time later in life than the last, each time with more to lose. Watching it happen taught me more than any explanation of it would have.

I think we put far too much weight on occupational identity. We ask people what they “are” when what we mean is what they are being paid to do right now. Those are not the same thing.

A mechanic who becomes an engineer does not arrive empty-handed. A construction worker who moves into software already understands sequencing, dependencies, tolerances, physical constraints, customers, mistakes, and what happens when plans encounter reality. A salesperson who becomes a product leader knows things about incentives and people that do not appear in a product-management textbook.

Learning one serious domain teaches you more than the facts of that domain. It teaches you something about systems, and, if you are paying attention, something about how you yourself become competent.

That is why the question “What are you going to do with your life?” is not useful. Ask instead what you are going to do next.

You are not choosing forever. You are choosing next.

That does not make the decision unimportant. It changes what makes the decision good. A good next step should leave you with more than you had before. More competence, more judgment, more context, more relationships, more credibility, more capital, or simply more options.

Then you choose again.

This is also why I have never been comfortable with “follow your dreams” as career advice. Dreams are useful. They give us energy and they make us try things. But desire and strategy are not the same thing.

People have aptitudes whether we like that fact or not. The world has needs whether we like that fact or not. Some capabilities are scarce, some are common, and some interests map more naturally than others onto work that can support a life. If you expect something to support your life, understanding how it creates value is part of taking responsibility for the decision.

Years ago I was riding the gondola between the peaks at Whistler with three young women. One of them had recently graduated and was telling the others about an argument she had had with her boss. She had a degree now, and she thought she should be paid more. Her boss told her he could not pay her more simply because she had acquired the degree.

She was furious. Why had she bothered getting it?

Then, somewhere in the conversation, she mentioned that she worked at a sandwich shop.

I never learned what the degree was in. That is probably why the story stuck with me. The degree may have been enormously valuable, and may have opened an entirely different career a month later. But the credential itself had not changed the economic value of the work she was performing that afternoon.

The point is the distinction between learning something, possessing evidence that you learned something, and becoming capable of doing something the world values. Those things overlap, but they are not identical.

Eventually reality gets a vote.

That phrase matters to me because it applies well beyond credentials. You can believe you understand a system until you have to build one. You can believe you understand customers until you have to sell something to them. You can believe you understand leadership until somebody else’s livelihood depends on your judgment. You can believe you understand risk until the decision is yours and the consequences arrive with it.

Capability develops when knowledge begins colliding with consequence.

That is why work, projects, apprenticeship, competition, and responsibility matter so much. They are not simply places to apply learning. They are part of how learning becomes judgment.

And judgment is difficult to acquire without being wrong.

My father used to ask, “Do you know why God made young men so stupid?”

“So they could do the impossible.”

He usually said it about the early space program, and that was not a coincidence. The young men in the joke were him and the people he worked with. He was not describing a category of person. He was describing a room he had been in.

There is a limit to the principle. Ignorance can get you killed. But experience accumulates reasons a thing will not work, and there is a danger in becoming so sophisticated about risk that the sophistication becomes a reason never to take one.

If you are going to attempt difficult things, some of them will not work. What matters is what happens next.

Failure itself is not automatically useful. Failing repeatedly without changing anything is repetition. The useful part is the loop afterward. What happened, what assumption was wrong, what did I misunderstand, what was inside my control, and what should change next time?

The attempt failed. That is information.

“I am a failure” is something else.

Rejection works much the same way. Many worthwhile things require volunteering for outcomes somebody else partly controls. Apply for the job. Ask for the opportunity. Pitch the customer. Publish the idea. Start the company. Compete.

Somebody gets to say no.

If no is psychologically intolerable, you eventually begin designing your life so nobody ever gets the opportunity to say it. That feels safer, but it also shrinks the range of possible futures.

This is why resilience is less something you explain to a child than something you progressively load. An athlete does not begin with the maximum weight. A gymnast does not begin with the hardest skill. You give somebody difficulty at a scale they can survive, let reality push back, help them understand what happened, and then load a little more.

Over time they accumulate evidence about themselves. Discomfort ends. Embarrassment is survivable. Criticism can contain useful information. Preparation changes outcomes. Being wrong does not destroy you. Failure can be followed by another attempt.

This is what I mean when I tell my children that the hardest thing in life is managing your own psychology. Intelligence does not save you from fear. Talent does not save you from insecurity. Being right does not save you from ego. Knowing what you ought to do does not guarantee that you will do it.

A surprising amount of adulthood is remaining capable of acting while your own psychology is trying to convince you not to.

Which brings me to the message I worry about most, because it arrives sounding like sophistication.

Somewhere between the start of high school and the end of it, one of my children began telling me a story about his own future. Previous generations had taken the housing. Previous generations had taken the wages. The arithmetic of an ordinary adult life no longer worked, and there was not much point pretending otherwise.

I pushed back, and I want to be careful about what I was pushing back on. I was not arguing that housing is affordable or that the numbers are fine. Constraints are real, and a young person who cannot see them is not being educated, he is being flattered.

What I objected to was the shape of the conclusion. A structural fact had quietly become a personal verdict. He was not describing a difficult environment he would have to navigate. He was describing an outcome that had already been decided, which meant navigation was beside the point.

That is the mirror image of “follow your dreams,” and it fails for the same reason. One says the world will accommodate you. The other says the world will not permit you. Both remove the part where what you actually do makes a difference.

Narratives shape agency. Give young people tools, not verdicts.

The same idea applies to the people and environments we choose.

I have told my children for years that you cannot aspire to what you have not seen or experienced. We like to think our imagination is independent. It isn’t.

If you have never met anyone who built a company, building a company feels like something done by a different category of human being. Then you spend time around someone who has done it and the thing moves from abstract possibility into the set of things ordinary humans apparently do.

Spend time around engineers and engineering becomes concrete. Spend time around excellent tradespeople and craftsmanship becomes visible. See someone change careers later in life and reinvention becomes less frightening. Watch someone take a serious risk, fail, recover, and try again, and failure becomes less terminal.

This is part of what apprenticeship does extraordinarily well. The apprentice is not merely receiving instruction. The apprentice is watching what competent people consider normal.

And normal is contagious. So are standards. So are ambitions. So are fears.

Whether people like the implication or not, we become, in meaningful ways, like the people with whom we spend our time. Which is why I have come to think of the whole environment as three inputs worth choosing deliberately. People shape your standards. Problems shape your capabilities. Feedback shapes your calibration.

Those are decisions, and like any decision they are worth revisiting. I am not disciplined about this and I have stayed in lanes longer than I should have. But without some mechanism for reconsideration, inertia starts impersonating intention.

You do not need to predict your life. You do need to keep steering it.

All of this was already true before AI. AI makes one part of it harder to ignore.

Instruction is getting cheap and abundant. The judgment that used to accumulate as a byproduct of doing unimportant work is not. I have written about both of those elsewhere, the vanishing on-ramp and what turns scarce once reasoning is cheap, and will not argue them again here.

The piece that belongs in this essay is smaller and harder. Every one of those questions is a decision about what to do next, and there is no longer anyone obvious to hand it to. A model will tell you what is true. It will not decide what you are willing to own.

That does not require a new educational philosophy. It makes an old one newly relevant.

You need access. You need direction. You need problems that matter. You need people who know more than you do. You need enough consequence for reality to get a vote. And eventually you need responsibility for deciding what happens next.

Which brings me back to the thing Hurst University was actually built around.

Family.

When children are young, the family carries almost everything. Food, shelter, transportation, money, protection, opportunity, judgment, and most of the consequences of decisions all sit primarily with the parents.

Childhood is, in part, the gradual transfer of that weight. Not all at once. That would be abandonment. A little at a time, while mistakes are still cheap and there is somebody nearby who can help make sense of them.

At first we choose for them. Then we let them choose between things we have selected. Then they make decisions we would not have made. Then they live with some of the consequences. We advise more and decide less.

If the process works, responsibility moves almost imperceptibly from one side of the relationship to the other.

Eventually they leave the house. They do not leave the family.

That distinction is more important than I understood when I first started making the Hurst University joke. The goal was never independence in the sense of needing nobody. Families do not work that way, and neither does the rest of life. The goal was to change your position inside the family.

When you are small, the family carries you. For a long time that is its job. Then you begin carrying more of yourself. Your decisions become yours. Your mistakes become yours. Your work becomes yours. Your education becomes yours. Eventually the roof over your head becomes yours too.

And somewhere along the way, if things have gone well, another transition begins. You become capable of carrying some weight for the people who once carried all of yours.

That is much closer to what I mean by graduation.

Not that you know what you are going to do for the next forty years. Not that you have accumulated the correct credentials. Not that you have stopped needing your parents, your siblings, or anyone else.

You leave the house able to participate in the family as an adult. You can build enough of a life to carry yourself, and enough capability that when the people you love need something from you, you have something to give.

The graduate of Hurst University is not the person who has all the answers. It is the person who has become difficult to make helpless.

Thirty years ago I did not have all of this worked out. I had a joke about Hurst University and a conviction that my children needed to leave the house able to make a future for themselves.

I understand the joke a little better now.

The future was never the thing I could give them. The best I could do was help them become people capable of building one.

From Periodic Audit to Continuous Assurance

Leave a reply

I have been writing about the limitations of audits and compliance systems for several years.

In Accountability and Transparency in Modern Systems, I wrote about systems producing evidence continuously rather than assembling it periodically for an auditor.

In First Principles for Root Store Management, I looked back at the decision to require WebTrust for publicly trusted CAs and argued that, if we were designing the system today, much more of the trust decision should be based on continuously verifiable behavior.

That led to The Limitations of Audits, Rethinking Compliance, and Compliance at the Speed of Code.

The common thread was that the systems we are trying to assure change much faster than the mechanisms we use to understand them.

Over the last year, I have spent considerably more time on this problem, both thinking about it and building systems intended to work differently. That work convinced me that the problem is deeper than periodicity alone.

I have pulled that thinking together into two new long-form pieces.

They are intended to be read together.

The first explains how we got here.

The second explores what comes next.

From Chains to Trees

Leave a reply

The WebPKI has two structures that are not the same shape.

One is a cryptographic graph of signed bindings. Public keys, names, entitlements, and the keys that authorized them. The other is a governance hierarchy of accountability. It explains why a relying party accepts that authority at all, and when it stops accepting it.

Nearly every interesting failure in the history of the system lives in the gap between them. Misissuance, compromise, distrust events, and the long struggle with revocation are all stories about that mismatch.

I wrote two long-form pieces that try to make the distinction legible.

The first walks through the classical system as it actually exists. What a certificate is, how trust is delegated, how root programs and policy actually work, and why the governance layer has always mattered more than the certificate chain itself.

A Deep Dive on the Classical WebPKI

The second examines the redesign now underway. Post-quantum signatures are simply too large for the classical model at public scale. The response is Merkle Tree Certificates. A CA logs certificates into its own tree, signs the tree head, and each certificate carries a short inclusion proof. One signature covers the batch. The proof is the path.

This is not merely a cryptographic migration. It is the ecosystem cashing in a forced upgrade to close a decade-old compromise in Certificate Transparency. Transparency stops being a post-issuance promise and becomes the issuance mechanism itself. The wire format changes. Most of the governance carries forward.

The Post-Quantum WebPKI

The Status Quo Outlived Its Status

Leave a reply

In security we like to say that the problems live in the gaps between systems. Each system, on its own, is usually coherent. It has a threat model, invariants, and someone who owns it. The seam between two systems is owned by nobody, and each side quietly assumes the other is handling the thing that neither is. The load balancer assumes the backend validates. The parser assumes the canonicalizer normalized. The audit covers the software but not the network it runs on. Attackers don’t have to beat either component. They just have to find the assumption neither side wrote down. The attacker gets to pick the threat model, and they pick the one that lives in the seam.

Once you see this pattern, you see it everywhere, and not just in security.

Quality Lives in the Gaps of Ownership

In software quality, the same topology produces a different failure class. Security gaps produce exploits. Ownership gaps produce jank.

A user’s journey through a product is inherently horizontal. They sign up, configure, use, get billed, get help. But ownership is vertical, carved along team boundaries. So the experience degrades precisely at the handoffs. The onboarding flow owned by one team dumps you into a product owned by another, with terminology that doesn’t match, settings that don’t carry over, and an error message that references a concept from a third team’s domain model. Every screen passed its own review. The journey never got one, because the journey has no owner.

This is the same root cause as the security version. Contracts between components are written in terms of what each side provides, not what the whole must feel like or withstand. Functionality composes. Quality attributes don’t. Not security, not usability, not performance, not consistency. Those are emergent properties of the composition, and emergent properties are exactly what per-team accountability structures can’t see. A dashboard that takes eight seconds to load is usually five services each meeting their SLO.

You Ship the Org Chart in N Dimensions

Conway’s law is usually quoted as a statement about architecture, that organizations design systems which mirror their communication structures. But architecture is just the most legible projection. The org chart also manifests in the security posture, where trust boundaries land wherever the reporting lines do. It shows up in the latency profile, where every org boundary becomes a network hop plus a queue plus a retry policy. It shows up in the data model, where the same “customer” is defined four ways because four VPs own four systems. It shows up in the compliance scope, where audits map to cost centers rather than to where the risk actually lives. It even shows up in the documentation, where each team documents its interior and nobody documents the crossings.

You don’t ship your org chart once. You ship it in every dimension at the same time.

And it’s stickier than the chart on the wall, because the formal org chart is only the visible part. The real partitioning is cultural. It is who trusts whom, which teams have history, where the scar tissue from the last reorg sits, and who won the last budget fight. Systems calcify around those boundaries. That’s why reorgs so rarely fix seam problems. You can redraw the chart in a day, but the shipped artifact embodies the org chart as it existed at every point in the system’s history. Legacy code is really legacy org structure. You’re maintaining the fossil record of decade-old turf wars, and the team that could explain a given seam disbanded three reorgs ago.

Bureaucracy Is the Fixative

Here’s where it hardens. Process is how organizations serialize distrust between units. Every approval gate, every ticket queue, every review board is a treaty boundary between fiefdoms, and treaties optimize for non-aggression, not for the emergent properties of the whole.

The bureaucratic instinct when a seam fails is to add process at the seam. A checklist, a sign-off, a form. This papers over the gap without giving it an owner, and now the seam has a compliance artifact defending its existence.

Which brings us to the uncomfortable part. Bureaucracy defends the status quo long past the point where the status quo lost its status. Not out of malice, and usually not even out of preference. The mechanism is simpler and more forgivable than that. Process is memory without comprehension.

Every rule is a compressed lesson. Some incident happened, someone got burned, a control was born. But the compression is lossy. The rule survives while the context doesn’t. The organization keeps executing the answer long after everyone who understood the question is gone. It’s Chesterton’s fence, except nobody can find the fence’s author, the field it enclosed is now a parking lot, and there’s a Fence Compliance team whose headcount depends on the fence.

That’s the inversion point. Controls that began as instruments become constituencies. A process accretes staff, tooling, budget, an annual review cycle. It stops being a means and becomes a stakeholder. And stakeholders defend themselves.

Asymmetric Bookkeeping

The genius of bureaucratic self-defense is that it never has to argue the status quo is good. It only has to make change expensive.

Every proposal to remove a control gets evaluated by asking what risk removal creates. Nobody asks what risk retention creates. The cost of the existing process is denominated in currencies the review process can’t count, things like velocity, morale, and opportunities that quietly went elsewhere, while the cost of change is denominated in the one currency it’s built to count. With bookkeeping that asymmetric, the ratchet only turns one way.

There’s a reliable tell for when status is lost but the defense continues. The justifications go circular. Ask why we do this and the answer stops referencing a threat or an outcome and starts referencing the process itself. It’s required for the audit. It’s policy. That’s the template. When a control’s referent is another control, you are no longer managing risk. You are maintaining a liturgy. And liturgies are stable. That’s what they’re for.

The people defending the liturgy usually aren’t cynics, either. Institutions promote the people who thrived under the current rules, which means the people with the authority to change the system are precisely the ones whose careers validate it. They don’t defend the status quo because they’ve weighed it and found it good. They defend it because it’s the ladder they climbed, and it’s genuinely hard to see your own ladder as arbitrary. The system doesn’t need guards. It manufactures believers. That’s why correction so often comes from outside, from a competitor, a collapse, or a technology that routes around the institution entirely, rather than from reform. Reform requires the institution to metabolize the idea that its own selection function is the problem, which is roughly asking the liturgy to audit itself.

Deletion Has No Constituency

So what do you do about it? Two things, and both are harder than they sound.

First, admit that the seams are the system, and staff them accordingly. You can’t fix an n-dimensional Conway problem with a one-dimensional intervention. A design system fixes the UX projection. A service mesh fixes the network projection. A GRC tool fixes the audit projection. But the generator is the accountability topology itself, so the pathology just re-expresses in whatever dimension you didn’t treat. The only durable moves are changing the human topology, which is rare, painful, and temporary, or forcing the interfaces to be explicit, adversarially specified, and owned as products. That’s the real lesson of the Amazon API mandate. It wasn’t about services. It was about giving the gaps owners.

Second, build a decay function. Controls get created by incidents, which are vivid and have advocates. Control removal has no incident, no advocate, no ceremony. The beneficiary of deletion is diffuse while the loser is a specific person in the room. So organizations accumulate process the way arteries accumulate plaque, one reasonable deposit at a time. Sunset clauses, zero-based process reviews, and deletion treated as a first-class ritual with the same ceremony as launch are ideas everyone nods at and almost nobody funds. The organizations that stay fast are the ones that treat removing a rule as an achievement, not an admission.

Because the status quo isn’t defended because it won an argument. It’s defended because it’s the null hypothesis, and the burden of proof only ever runs one direction. Every so often, you have to flip the burden and make the process re-justify itself in terms of an outcome rather than another process. If it can’t, it isn’t protecting you anymore.

It’s just protecting itself.

Why FIPS 140 Means Running Old Code

2 Replies

You need to use FIPS 140 because of compliance, but have you ever asked what that requirement is actually for? What security properties are the authors of these policies trying to achieve?

In high-assurance deployments, the practical goal is usually to establish a meaningful security boundary around cryptographic keys. Organizations want explicit controls over who and what can use a key, and they do not want the answer to be every application or administrator with access to the host. They are also worried about key theft and abuse. For important signing and decryption keys, keeping the key out of the hands of the application and host OS is often the simplest way to force reasonable key-protection practices.

These are real problems. When the threat group Storm-0558 acquired a highly sensitive Microsoft MSA signing key, operational failures allowed key material to escape the isolated signing environment and become accessible from a compromised engineering environment. That single extraction let the attackers forge tokens and compromise customer email accounts at scale. It is much harder to see that happening when a key is non-exportable and managed inside a hardware boundary. HSMs are not the only way to get these properties, but they are the one tool that forces you to think hard about how you operationalize a key, and that discipline has value.

The trade-off is that you end up running old software you cannot patch for upstream security vulnerabilities. In the best case, you are years behind.

Worse, this is usually non-memory-safe code that is entirely opaque to you. The firmware, the middleware, and the technical documentation are kept strictly behind lock and key by the vendor. You cannot inspect the code to see if it is vulnerable, and independent review is virtually impossible. The HSM vendor may have proactively patched those vulnerabilities, but more likely they have not.

The cryptography in these modules almost never breaks. What breaks is the plumbing. For example, when an HSM takes in a payload like an administrative command or an authentication token of some sort, it has to parse it. This may mean relying on complex parsers or business logic, often written in aging C code. Because that plumbing is not memory safe, a single malformed input can lead to a buffer overflow or remote code execution right past the validated boundary, or even worse, a long-forgotten feature combined with new code could bypass policy controls around accessing the module altogether.

Take the U-Boot forks as another example. When you look at how these embedded systems are actually built, they lean heavily on bootloaders, embedded operating systems, and vendor firmware that sit outside the cryptographic boundary but inside your trust story. A vendor might fork U-Boot, validate their module, and then that code is essentially frozen.

Think about what boot verification actually involves. Something has to parse the firmware image, figure out which bytes are covered by the signature, compute the digest, and check the signature. The math for the signature might be FIPS validated, but the parsing, the offset arithmetic, and the handling of attacker-controlled structures is all just non-memory-safe C code.

When researchers find vulnerabilities in X.509 parsing or U-Boot image processing, the flaws are not in the cryptography. They are in the plumbing. And because vendor version strings in this opaque, locked-down firmware often have no meaningful relationship to upstream release numbers, that fork may be a decade old.

The real danger is that two curves are moving in opposite directions. The attack surface is being examined continuously. AI and automated variant analysis are getting exponentially better at finding previously unknown exploitable memory-safety bugs in old C code. Meanwhile, the validated base firmware remains essentially fixed.

We do not have to guess about this. I recently published a data project analyzing recent FIPS 140-3 validations. The data shows how frozen these systems actually are. Out of 415 validated modules we looked at, 324 of them, 78 percent, showed no recorded public update after their initial validation. The knowledge of security issues compounding every week is colliding with code that barely moves.

FIPS 140 compliance buys you necessary domain separation and forces good operational habits. But the way the market achieves that separation often leaves you depending on a static, aging, and opaque codebase. The certificate tells you that a particular version met the requirements when it was evaluated. It does not tell you that the same code remains secure years later. At some point, the certificate becomes evidence not only of what was validated, but of how long the underlying system has stood still.

The Certification Ends Where the Code Begins

Leave a reply

Disclosure: I am an advisor to Binarly.

I recently built the FIPS 140-3 Corpus, a dataset that pulls together the public record of FIPS validations. It combines CMVP certificate records, Security Policies, implementation details, operational environments, firmware versions, algorithm claims, and lifecycle data into something you can actually query and analyze rather than read one certificate at a time.

I built it because I have spent enough years around certification programs to know that the interesting information is rarely in any single document. It emerges when you look at the record as a system. Once you do, a pattern shows up that I think deserves more attention than it gets. The public evidence tells you a great deal about what was evaluated and almost nothing about whether the code that shipped actually behaves the way the evaluation assumed.

What the paper trail shows

When you read Security Policies in bulk, you start seeing the same dependencies over and over. Validated modules lean on bootloaders, embedded operating systems, update agents, and vendor firmware that sit outside the cryptographic boundary but inside the trust story. The certificate covers the module. The security property depends on everything around it.

U-Boot is a good example. Several modules in the corpus disclose it as part of their firmware or boot environment, and in some of those cases it participates directly in verifying firmware integrity before execution. Think about what that verification actually involves. Something has to parse the firmware image, figure out which bytes are covered by the signature, compute the digest, check the signature, and then decide what to run. RSA and SHA-256 handle two steps in that sequence. The parsing, the offset arithmetic, the decision about which fields are authenticated and which are attacker controlled, all of that is ordinary C code, and it is exactly where things tend to go wrong.

Binarly’s researchers recently published Unfit to Boot, which found previously unknown vulnerabilities in U-Boot’s FIT image processing and signature verification path. The flaws were not in the cryptography. They were in the handling of attacker controlled structures before and around the verification operation. The math was fine. The plumbing was not.

This is the oldest lesson in applied cryptography and we keep relearning it. The primitive is almost never the weakest link. The code that feeds the primitive is.

Where the evidence runs out

Here is where it gets uncomfortable. A Security Policy might identify its bootloader with a string like CNN35XX-UBOOT-4.03-03. That tells you a vendor U-Boot derivative is present. It tells you almost nothing else. Which upstream revision was it forked from? What did the vendor change? Which FIT features were compiled in? Were the fixes for known parsing flaws ever backported? Can externally supplied firmware even reach those code paths in this product?

None of that is answerable from the certification record. Vendor version strings in embedded firmware often have no meaningful relationship to upstream release numbers. The fork may be a decade old. The fixes may have been applied selectively, or renamed, or lost in a rebase nobody documented.

Conventional software composition tools do not close this gap either. They work by matching. Filenames, manifests, version strings, hashes, YARA rules, CVE mappings. That approach answers a useful question, namely whether a binary appears to contain a component already known to be vulnerable. Firmware defeats it routinely. Dependencies get statically linked into larger executables, symbols get stripped, and vendor forks drift far enough from upstream that the signatures stop matching anything.

And matching cannot help with flaws nobody has found yet. Before the Unfit to Boot research existed, there was no CVE to map, no affected version range, no signature to match. Someone had to go look at the implementation first. Credit to the Binarly team for doing that work, and disclosure noted, but the point stands independent of any vendor. Until somebody examines what actually shipped, every downstream tool, database, and compliance process is working from an empty record.

Why I care about this for HSMs and BMCs

The corpus is full of devices that sit in unusually trusted positions. HSMs hold the keys for certificate authorities, payment systems, and governments. BMCs sit beneath the host operating system with control over firmware updates, recovery, and remote administration. I have spent much of my career depending on the first category and being quietly worried about both.

These devices are exactly where the paper trail is weakest. They accumulate long lived vendor forks, inherited open source components, proprietary parsers, and hardware specific code written over many years, most of it statically linked and distributed only as compiled firmware. HSM firmware makes the visibility problem even worse. It is almost never publicly accessible, and on the rare occasion you do get an image, it is often encrypted or obfuscated, frequently with a key shared across the product line. Whatever that design accomplishes, it means customers and independent researchers see less of the code than a motivated attacker willing to recover the key. So we end up trusting these devices on the strength of certifications that, as the U-Boot example shows, stop well short of the code paths where real failures happen.

That does not make the certifications worthless. It makes them a starting point. A validation record that discloses a U-Boot derivative in the boot chain has handed you a concrete question to ask your vendor. What evidence supports the claim that your product is unaffected by this class of flaw? Has anyone analyzed the released binary, or is the answer derived from a spreadsheet of version strings? Which fields of an incoming update are actually authenticated before any code touches them? Vendors who can answer those questions with evidence are telling you something important. So are vendors who cannot.

Documentation, inference, evidence

The way I think about it, assurance comes in layers and each layer answers a different question. The certificate tells you what was evaluated and under what assumptions. The corpus connects those artifacts across the whole ecosystem and exposes the shared dependencies and recurring architectures the individual documents obscure. The final layer is evidence about what actually shipped, and it can come from several places. Vendors tracking their forks against upstream and documenting backports. Independent analysis of released binaries. Researchers publishing the failure modes of mechanisms everyone assumed were sound.

Most of the industry stops at the first layer. Procurement checks for the certificate, the checkbox gets ticked, and the boot chain full of forked bootloader code goes unexamined until someone publishes research like Unfit to Boot and everyone scrambles to figure out whether they are affected.

I built the corpus to make the second layer easier, so that the public record can generate the right questions instead of just decorating RFP responses. The questions are the point. A validation record that names a bootloader fork should end in a conversation with the vendor about what is in it, not in a filed PDF. Trust in labels got us the last twenty years of firmware security. Trust supported by evidence is going to have to get us the next twenty.

Steve Jobs, AI, and the Problem of Analysis Without Ownership

Leave a reply

There is an old Steve Jobs clip from a 1992 MIT Sloan talk that feels newly relevant in the age of AI. In the talk, available here as Steve Jobs MIT 1992 Lecture, Jobs is asked about consultants. His answer is not that consultants are unintelligent or useless. His criticism is more subtle. He says consultants often get to see a lot, analyze a lot, and recommend a lot, but they do not stay with the work long enough to own the consequences.

They do not spend years living with the product, the team, the tradeoffs, the mistakes, the customers, the budgets, the bugs, or the recovery. They may see the fruit, as Jobs put it, but they “never really taste it.”

That distinction matters.

There is a kind of knowledge that comes from observation, and there is a different kind of knowledge that comes from ownership. Observation can make you articulate. Ownership makes you careful. Observation helps you describe what should happen. Ownership teaches you what actually happens when a recommendation meets constraints, incentives, politics, timelines, systems, and human behavior.

There is also a kind of knowledge that only comes from time.

Some problems cannot be understood in a single sitting. You need to carry them around for a while. You read, step away, come back, notice what still bothers you, test a different framing, sleep on it, and then see the thing that was hiding in plain sight. That kind of soaking is not inefficiency. It is often how judgment forms.

That is the parallel to AI.

AI is making analysis abundant. It can read more than we can read, summarize faster than we can summarize, find patterns across larger datasets, generate plausible options, and produce recommendations that sound polished and confident. That is useful. But it is not the same as judgment.

Used poorly, AI becomes consulting at machine scale. It is fast, articulate, and superficially impressive, but disconnected from whether its recommendations actually survive contact with reality.

It can say what should be done without knowing what happened after someone tried to do it. It can identify risks without understanding which ones mattered. It can produce a roadmap without living through the missed dependency, the customer objection, the policy constraint, the budget cut, the migration failure, or the second-order effect six months later.

It can also make us confuse speed of response with depth of understanding. That may be the deeper risk. AI can collapse the slow work of thinking into the first plausible answer. It can make a problem feel resolved before we have really spent time with it. It can produce fluency before we have earned conviction.

That does not make AI useless. It makes the design problem clearer.

The lazy version of the AI story is the self-driving car analogy, namely once the machine becomes safer, faster, or more consistent, the human gets pushed out of the loop. There will be domains where that is true. But much of knowledge work is different. The goal is not only to execute a task correctly. The goal is to understand the problem well enough to make better decisions the next time.

Execution tools can displace. Reasoning tools should compound.

That is why the most interesting promise of AI is not simply that it becomes a better consultant or even a better operator. It is that AI can help humans become better operators.

Used well, AI becomes a way to think with the material. It helps us understand datasets that are too large to hold in our heads. It lets us explore problem spaces from more angles. It helps test assumptions, compare interpretations, surface edge cases, and ask better questions. It can show us patterns we would have missed, but the value is not just the pattern. The value is that, through the process of interrogation, we understand the problem more deeply ourselves.

AI should not shorten our attention so much as deepen what our attention can hold.

A good AI system should help us return to a problem with more context than we had the last time. It should preserve the questions we asked, the assumptions we tested, the contradictions we found, the evidence that mattered, and the places where our understanding changed. It should make it easier to spend real time with the problem, not merely produce an answer faster.

In that sense, the best use of AI is not instant certainty. It is structured patience.

It lets us soak in a problem more effectively. By that I mean it enables us to hold more evidence in view, revisiting prior interpretations, comparing today’s answer to yesterday’s uncertainty, and gradually turning analysis into understanding.

The goal should not be to outsource judgment to AI. The goal should be to use AI to improve the conditions under which judgment is formed.

A good AI system should not merely say, “Here is the answer.” It should help us see why the answer might be true, where it might be fragile, what evidence supports it, what alternatives exist, and what would change our mind. It should help us move from diagnosis to action, from action to feedback, and from feedback to learning.

That is the line between AI as consultant and AI as learning partner.

This is also where many AI products will disappoint. Dashboards full of findings, risks, summaries, and recommendations can look impressive while still leaving the actual burden on the human team. They create the appearance of progress without necessarily improving understanding. The human still has to decide what matters, translate the finding into action, make the change, verify the outcome, and remember the lesson later.

The point is not that findings and recommendations are useless; they are necessary. Findings are the beginning of the loop, not the end of it. Systems that stop there are not doing judgment automation. They are doing analysis transfer.

The more interesting systems will close the loop. They will connect analysis to execution, execution to verification, and verification to institutional memory. Not because humans should be removed from the process, but because humans should be able to reason from a better substrate.

This is where Jobs’ point lands today. The scarce thing is not access to analysis. AI will make analysis abundant. The scarce thing is accumulated judgment, something that only comes from acting, observing, correcting, and learning over time.

Observation gives you language. Ownership gives you consequence. Time gives you depth. Feedback gives you judgment.

Jobs’ critique of consulting was not just a warning about consultants. It was a warning about any tool, process, or person that gets rewarded for sounding right without having to live with whether they were right.

AI will be most valuable not when it becomes the smartest consultant in the room, but when it helps teams build judgment faster, seeing more, acting sooner, sitting with the problem longer, verifying outcomes, and remembering what reality taught them.

The future of AI in knowledge work should not be analysis without ownership. It should be ownership made smarter.

The Breaker, the Priest, and the Philosopher

Leave a reply

Spend enough years in security and you notice that the people whose judgment you actually trust are rarely the ones with the cleanest credentials.

They are the ones who have been wrong in public often enough to develop taste. Their authority is earned backward, from scars rather than definitions. When they look at a scheme and say, no, that is wrong, and here is the deeper reason, they are not deriving it from first principles. They are recognizing a shape they have been cut by before.

That is worth taking seriously. In security you get little standing to philosophize until you have shipped something, broken something, defended something, or watched something fail. The person who starts from what is identity, or what is trust, but has never lived with the consequences of an answer, barely exists as a respected type. The people who carry real philosophical weight almost always came up through contact with failure first.

So the security philosopher is not the pure theorist. The philosopher is what a survivor of failure becomes once the failures start to rhyme and a reflective habit sets in. Most often that survivor is a breaker, because breaking is the most direct contact you can have with the gap between how a system should work and how it does. But breaking is not the only contact that leaves marks, and that turns out to matter later.

Everyone who matures this way is answering one question, whether or not they say it out loud.

Why does this keep happening?

You earn the right to answer by watching it happen enough times that fixing the bug stops feeling like an answer. Whatever conclusion you settle on is what you turn into. Some decide people do not understand the systems deeply enough. Some, that bad claims go unchallenged too long. Some, that security is downstream of engineering. Some, that the bytes are downstream of institutions and power. Some, that the abstractions themselves are broken. Some, that the failure is intrinsic and the only honest response is to keep hunting.

Those answers create the taxonomy. The genus is philosopher. The species are sorted by the answer each one gives, and by the way each one tries to make that answer true for other people.

The Sage

The Sage believes it keeps happening because nobody understands the systems deeply enough.

The Sage transmits by instantiation rather than argument. The worldview gets built into a tool, written into a book, embedded in a way of working, and you absorb it by use. A fuzzer can teach an entire philosophy of bug finding. Coverage becomes the thing worth chasing. The tool makes the argument every time it runs, so nobody has to persuade you in a thread. The position is already standing in the room, made of working code.

The Sage is contemplative, the monk of the genus. The work is not quiet because it is timid. It is quiet because it expects reality to do the teaching.

The Gadfly

The Gadfly believes it keeps happening because people are wrong in public and nobody corrects them.

The Gadfly’s philosophy exists only in motion. It lives in the thread, the argument, the review comment, the refusal to let an incorrect claim stand. This is Socratic in the original and irritating sense. You learn what is true by watching the argument refuse to die.

The Gadfly may share the Sage’s diagnosis exactly. People do not understand the system. But the method is the opposite. The Sage builds, the Gadfly fights. Both believe misunderstanding is the enemy. They differ on whether the cure is a tool or a wound.

The Builder Evangelist

The Builder Evangelist believes it keeps happening because security is downstream of bad engineering.

This is the breaker who concludes the dramatic failure is only the visible symptom. The real incident happened earlier, in how software was designed, reviewed, deployed, owned, or forgotten. The answer is not more heroics. It is to change how teams build, with security folded into normal engineering life rather than bolted on as a gate at the end or a priesthood that arrives with findings after everyone has moved on. The insight only counts if it propagates into how people actually work.

The missionary energy is the tell. Converts always have it, and this is the convert’s slot. Philosophy that has to spread to count as true.

The Statesman

The Statesman believes it keeps happening because the bytes are downstream of institutions, incentives, and power.

This one did not go deeper into the stack. They went up, out of the assessment shop and into platforms, governments, trust programs, procurement, incident governance, the places where the adversary is sometimes the org chart and sometimes a nation. The wisdom gets cashed out in policy and in who sits at which table.

The risk of the type is floating clear of the actual bytes. The best Statesmen never do. They remember the policy is only real if it changes what happens at the machine, the credential, the incident bridge. The worst become fluent in altitude and lose contact with the ground.

The Theorist

The Theorist believes it keeps happening because the abstractions themselves are unsound.

This is the opposite vector from the Statesman. Where the Statesman goes up toward power, the Theorist goes down toward formalism. Weird machines. Exploitation as programming a machine nobody meant to build. Security as a subset of reliability. Trust as an operational claim, not a noun.

This is the species that comes closest to the academic register, but the ticket was still bought through breaking first. The formalism is trusted because the person doing it has felt the abstraction fail in their hands. That is the difference between earned theory and decorative theory.

The Refusenik

The Refusenik is the apex breaker who declines to metamorphose at all. Not from inability. From principle.

The refusal is itself an answer. It keeps happening because failure is intrinsic to systems of any real complexity, and pretending a framework or a doctrine can end it is the deeper error. There is no theory waiting at the top of the climb, only the next finding. So the Refusenik stays the predator and calls the philosophizing a comfortable retreat from the only thing that is real, the work in front of them.

This is the lower bound of the taxonomy. It proves that becoming a philosopher of the discursive kind is a choice, not an inevitability, and it does so by holding a real position rather than an empty one. Some of the best who ever lived remain here permanently, and they are right to.

The Priest

The Priest is the auditor, the compliance keeper, the custodian of the framework. The Priest is the scandal of the taxonomy, but not for the reason the field assumes.

The reflexive complaint is that the Priest arrived without breaking. No public exploit, no system torn open, no credential earned the old way. By the breaker’s accounting, no scars at all. That accounting is the actual error.

Much compliance really is theater. Much audit work mistakes evidence for reality. Much framework worship trains people to pass inspections while risk keeps moving underneath them. None of that is in dispute. But at scale the Priest is often the only force keeping an organization’s vital signs visible. Compliance read generously is not theater. It is a pulse, one of the few observable ways to ask whether the organization is doing the things it claims to.

And the Priest does have scars, just not the kind the breaker recognizes. The Priest learns by watching organizations lie to themselves in patterns, watching the same control fail the same way across a dozen audits, watching risk reappear in process and ownership long after the technical finding was closed. That is sustained contact with failure. It leaves marks. The breaker simply does not read them as marks, because they did not draw blood the familiar way.

So the scandal is not that the Priest skipped the initiation. It is that the breaker cannot see the Priest’s scars as scars. Seeing them requires fusing two diagnoses that rarely live in the same person. The Builder Evangelist says automate or drown. The Statesman says the real system is organizational health. The generous read of the Priest requires both at once, and most people who came up breaking cannot hold both, so they read the Priest as the enemy.

Sometimes they are right. Sometimes they are only defending their own credentialing system.

The seam in the taxonomy

Two forces are doing the work here. One is diagnosis, the answer to why this keeps happening. The other is temperament, how you try to transmit that answer. The clean version of the theory says these collapse into one, that the way you transmit is downstream of what you concluded and who you are. The Sage builds because he decided depth is the problem and because he is contemplative. The Gadfly fights because he decided public error is the problem and because he cannot leave it alone.

But the Sage and the Gadfly may share the same diagnosis. If they do, they are one species wearing two faces, and the real joint is diagnosis, with transmission as a surface effect. The alternative is that transmission is itself fundamental, that how you choose to make a thing true for other people is a deeper fact about you than the proposition you are trying to make true.

I do not think this is settled, and I am not sure it should be. A taxonomy that resolved it cleanly would claim to know which of two people who believe the same thing is the more serious, on the sole basis of whether they build or fight. That is a claim worth resisting.

The point

Security does not really trust credentials. It trusts scars. That instinct is mostly healthy. It keeps empty abstraction out of the room and gives weight to people who can smell a failure coming rather than just describe one.

But every credentialing system has a blind spot, and the breaker’s is believing that only breaking confers standing. The whole taxonomy is one argument against that belief, because every species on it earned its authority through a different kind of contact with failure.

The breaker learns by cutting into systems.

The builder learns by watching teams repeat the same mistakes.

The statesman learns by watching incentives defeat correctness.

The theorist learns by watching abstractions collapse.

The priest learns by watching organizations lie to themselves in patterns.

The refusenik learns by never looking away from the hunt long enough to be comforted by a story about it.

Five are reflective and one refuses reflection, but all six are forms of earned contact, and none is the only one that counts. What you become is the answer you settle on for why this keeps happening, and the way of making it true for others that you cannot help but reach for.

The breakers who refuse the question stay hunters. The keepers who never broke anything are mistrusted for it, sometimes fairly and sometimes not. Everyone else becomes a philosopher of one species or another, whether or not they would accept the word.

The only real question is which one you became.

The Prompt Is an Argument

Leave a reply

The prior piece made a narrow claim. The prompt is the record, because a system can only act on what reaches it. Intent that stays in your head does not govern anything. Context that never reaches the model does not constrain anything. Purpose that is not in the prompt, the retrieved material, the tools, the policies, or the examples is not part of the decision environment.

If you accept that, the next question is unavoidable. What kind of record should a good prompt be?

For goal-oriented work, it should be an argument. More precisely, it should be enthymematic, an argument that leaves its obvious premise unstated for the audience to supply.

The form is Aristotle’s. He called the enthymeme the body of proof, the strongest of rhetorical proofs, a kind of syllogism with a premise left out. Its strength comes from the omission. Stating what the audience already accepts is tedious, while assuming the right premise makes the conclusion feel almost self-evident. That power holds on one condition. The missing premise has to be one the audience can already supply.

A capable model supplies a premise readily. It does not reliably supply yours, and from the output alone you cannot tell which one it used. Where your premise and its default mostly agree, the gap is cheap. Where the wrong one is costly, or where you have to reconstruct the reasoning later, it is not.

Look at a bad prompt with that in mind.

“Write about DevOps automation.”

That is a topic, not a goal. The model can go anywhere. CI/CD, infrastructure as code, job displacement, Kubernetes, a bland survey that explains nothing. It did not disobey. The prompt simply did not contain an argument for it to advance, so it advanced none.

Now the same subject with the premise restored.

“Explain why DevOps automation is becoming more viable now that production systems can be modeled, tested, and validated in synthetic environments.”

This carries a claim the first version left in your head. Automation gets safe where results can be checked. Coding agents became useful because code has feedback loops. It compiles or it does not, tests pass or fail, the type system objects. As production systems become modelable and testable, operations starts to acquire the same machinery, so it starts to look more automatable for the same reason coding did. You could not spell all of that out if you tried. The context behind any goal is unbounded, so every prompt is already a compression, and the only real choice is what to keep. Keep the load-bearing premise and drop what the model already handles. Here that premise is the causal claim, present enough that the model knows which answers would count as success, the one it would otherwise have to invent.

That is the difference between a task prompt and a goal prompt. A task prompt says do X. A goal prompt says do X in service of Y, because Z is the relationship that matters. The “because Z” is where most prompt failures live.

People leave it out because they assume the model already holds the frame. They write as if it knows the business context, the adversary, the institutional scar tissue, the product strategy, the audience, the risk appetite, the standard of correctness. Then a plausible, fluent, useless answer comes back and surprises them. The model was not confused. It was unconstrained. It got a task without the premise that made the task mean anything.

Humans skip premises constantly and get away with it. A lawyer says “that creates a reliance problem” and the room knows the shape of the issue. An engineer says “that breaks rollback” and the team feels the operational cost. A security person says “that moves the trust boundary” and the people who have lived with the system know why it matters. The shorthand works because the history is shared. A model does not inherit that history unless you hand it over. It does not know which premise is obvious inside your company, which analogy is carrying the real weight, which part of the request is decorative and which part is load-bearing. It does not know that “make this clearer” meant keep the technical claim but cut the wording that lets a reader hear monocausality.

This is why writing a good prompt feels more like drafting than asking. A good lawyer does not write words that merely sound like the client’s intent. A good lawyer writes words that survive interpretation by someone else, later, under pressure, with incentives to read them the wrong way. The document has to carry its operative meaning forward once the author has left the room. A prompt carries the same burden into a system that completes patterns from whatever materials it has. Leave the wrong premise implicit and the model may complete the argument in the wrong direction. Omit it and the model substitutes a generic one. Supply several competing premises and the model optimizes for the one easiest to write about rather than the one that matters. That is how a prompt produces fluent nonsense. Not that the model cannot write, but that the prompt did not preserve the reasoning.

So before writing a goal-oriented prompt, ask one question. What has to be true for this output to be useful? Not what topic it should cover, what format it should take, how long it should run. What has to be true. The answer is usually the missing premise. For a product strategy it is often the market wedge. For a security review it is the attacker’s actual path. For an executive summary it is the decision the executive has to make. For a critique it is the standard the work should be judged against. For a rewrite it is the misunderstanding you are trying to prevent.

That premise does not have to appear as a formal sentence. It can live in the framing, in an example, in the acceptance criteria, in the source material, or in an instruction about what not to optimize for. It only has to be somewhere in the record. Otherwise the system is not helping you pursue a goal. It is producing text near a topic.

In a single prompt you can hold the whole record in view. A production system spreads it out. The effective prompt is no longer the user’s sentence. It is that sentence plus the system instruction, the developer instruction, the retrieved documents, the available tools, the policies, the examples, the memory, the model version, and the configuration around all of it. The argument is distributed across those layers, which makes enthymematic design more important rather than less. The question stops being what the user asked and becomes what argument the system received. Did retrieval supply the missing premise or omit it? Did the tool definitions encode the right standard for action? Did the policy layer quietly override the user’s goal? Did the examples teach the wrong pattern?

Those are governance questions. A system that cannot reconstruct its effective prompt cannot reconstruct the argument it acted on. And a system that cannot reconstruct that argument cannot explain why its output was reasonable or unreasonable, compliant or not, safe or merely plausible.

The enthymeme works when the missing premise is shared enough to be supplied safely. That is the one condition these systems do not meet on their own.

So the work is not to make prompts longer. It is to make them carry the right inference.

The missing premise will be supplied either way. The only question is by whom. You provide it, or you let the system invent it.