You need to use FIPS 140 because of compliance, but have you ever asked what that requirement is actually for? What security properties are the authors of these policies trying to achieve?
In high-assurance deployments, the practical goal is usually to establish a meaningful security boundary around cryptographic keys. Organizations want explicit controls over who and what can use a key, and they do not want the answer to be every application or administrator with access to the host. They are also worried about key theft and abuse. For important signing and decryption keys, keeping the key out of the hands of the application and host OS is often the simplest way to force reasonable key-protection practices.
These are real problems. When the threat group Storm-0558 acquired a highly sensitive Microsoft MSA signing key, operational failures allowed key material to escape the isolated signing environment and become accessible from a compromised engineering environment. That single extraction let the attackers forge tokens and compromise customer email accounts at scale. It is much harder to see that happening when a key is non-exportable and managed inside a hardware boundary. HSMs are not the only way to get these properties, but they are the one tool that forces you to think hard about how you operationalize a key, and that discipline has value.
The trade-off is that you end up running old software you cannot patch for upstream security vulnerabilities. In the best case, you are years behind.
Worse, this is usually non-memory-safe code that is entirely opaque to you. The firmware, the middleware, and the technical documentation are kept strictly behind lock and key by the vendor. You cannot inspect the code to see if it is vulnerable, and independent review is virtually impossible. The HSM vendor may have proactively patched those vulnerabilities, but more likely they have not.
The cryptography in these modules almost never breaks. What breaks is the plumbing. For example, when an HSM takes in a payload like an administrative command or an authentication token of some sort, it has to parse it. This may mean relying on complex parsers or business logic, often written in aging C code. Because that plumbing is not memory safe, a single malformed input can lead to a buffer overflow or remote code execution right past the validated boundary, or even worse, a long-forgotten feature combined with new code could bypass policy controls around accessing the module altogether.
Take the U-Boot forks as another example. When you look at how these embedded systems are actually built, they lean heavily on bootloaders, embedded operating systems, and vendor firmware that sit outside the cryptographic boundary but inside your trust story. A vendor might fork U-Boot, validate their module, and then that code is essentially frozen.
Think about what boot verification actually involves. Something has to parse the firmware image, figure out which bytes are covered by the signature, compute the digest, and check the signature. The math for the signature might be FIPS validated, but the parsing, the offset arithmetic, and the handling of attacker-controlled structures is all just non-memory-safe C code.
When researchers find vulnerabilities in X.509 parsing or U-Boot image processing, the flaws are not in the cryptography. They are in the plumbing. And because vendor version strings in this opaque, locked-down firmware often have no meaningful relationship to upstream release numbers, that fork may be a decade old.
The real danger is that two curves are moving in opposite directions. The attack surface is being examined continuously. AI and automated variant analysis are getting exponentially better at finding previously unknown exploitable memory-safety bugs in old C code. Meanwhile, the validated base firmware remains essentially fixed.
We do not have to guess about this. I recently published a data project analyzing recent FIPS 140-3 validations. The data shows how frozen these systems actually are. Out of 415 validated modules we looked at, 324 of them, 78 percent, showed no recorded public update after their initial validation. The knowledge of security issues compounding every week is colliding with code that barely moves.
FIPS 140 compliance buys you necessary domain separation and forces good operational habits. But the way the market achieves that separation often leaves you depending on a static, aging, and opaque codebase. The certificate tells you that a particular version met the requirements when it was evaluated. It does not tell you that the same code remains secure years later. At some point, the certificate becomes evidence not only of what was validated, but of how long the underlying system has stood still.
I recently built the FIPS 140-3 Corpus, a dataset that pulls together the public record of FIPS validations. It combines CMVP certificate records, Security Policies, implementation details, operational environments, firmware versions, algorithm claims, and lifecycle data into something you can actually query and analyze rather than read one certificate at a time.
I built it because I have spent enough years around certification programs to know that the interesting information is rarely in any single document. It emerges when you look at the record as a system. Once you do, a pattern shows up that I think deserves more attention than it gets. The public evidence tells you a great deal about what was evaluated and almost nothing about whether the code that shipped actually behaves the way the evaluation assumed.
What the paper trail shows
When you read Security Policies in bulk, you start seeing the same dependencies over and over. Validated modules lean on bootloaders, embedded operating systems, update agents, and vendor firmware that sit outside the cryptographic boundary but inside the trust story. The certificate covers the module. The security property depends on everything around it.
U-Boot is a good example. Several modules in the corpus disclose it as part of their firmware or boot environment, and in some of those cases it participates directly in verifying firmware integrity before execution. Think about what that verification actually involves. Something has to parse the firmware image, figure out which bytes are covered by the signature, compute the digest, check the signature, and then decide what to run. RSA and SHA-256 handle two steps in that sequence. The parsing, the offset arithmetic, the decision about which fields are authenticated and which are attacker controlled, all of that is ordinary C code, and it is exactly where things tend to go wrong.
Binarly’s researchers recently published Unfit to Boot, which found previously unknown vulnerabilities in U-Boot’s FIT image processing and signature verification path. The flaws were not in the cryptography. They were in the handling of attacker controlled structures before and around the verification operation. The math was fine. The plumbing was not.
This is the oldest lesson in applied cryptography and we keep relearning it. The primitive is almost never the weakest link. The code that feeds the primitive is.
Where the evidence runs out
Here is where it gets uncomfortable. A Security Policy might identify its bootloader with a string like CNN35XX-UBOOT-4.03-03. That tells you a vendor U-Boot derivative is present. It tells you almost nothing else. Which upstream revision was it forked from? What did the vendor change? Which FIT features were compiled in? Were the fixes for known parsing flaws ever backported? Can externally supplied firmware even reach those code paths in this product?
None of that is answerable from the certification record. Vendor version strings in embedded firmware often have no meaningful relationship to upstream release numbers. The fork may be a decade old. The fixes may have been applied selectively, or renamed, or lost in a rebase nobody documented.
Conventional software composition tools do not close this gap either. They work by matching. Filenames, manifests, version strings, hashes, YARA rules, CVE mappings. That approach answers a useful question, namely whether a binary appears to contain a component already known to be vulnerable. Firmware defeats it routinely. Dependencies get statically linked into larger executables, symbols get stripped, and vendor forks drift far enough from upstream that the signatures stop matching anything.
And matching cannot help with flaws nobody has found yet. Before the Unfit to Boot research existed, there was no CVE to map, no affected version range, no signature to match. Someone had to go look at the implementation first. Credit to the Binarly team for doing that work, and disclosure noted, but the point stands independent of any vendor. Until somebody examines what actually shipped, every downstream tool, database, and compliance process is working from an empty record.
Why I care about this for HSMs and BMCs
The corpus is full of devices that sit in unusually trusted positions. HSMs hold the keys for certificate authorities, payment systems, and governments. BMCs sit beneath the host operating system with control over firmware updates, recovery, and remote administration. I have spent much of my career depending on the first category and being quietly worried about both.
These devices are exactly where the paper trail is weakest. They accumulate long lived vendor forks, inherited open source components, proprietary parsers, and hardware specific code written over many years, most of it statically linked and distributed only as compiled firmware. HSM firmware makes the visibility problem even worse. It is almost never publicly accessible, and on the rare occasion you do get an image, it is often encrypted or obfuscated, frequently with a key shared across the product line. Whatever that design accomplishes, it means customers and independent researchers see less of the code than a motivated attacker willing to recover the key. So we end up trusting these devices on the strength of certifications that, as the U-Boot example shows, stop well short of the code paths where real failures happen.
That does not make the certifications worthless. It makes them a starting point. A validation record that discloses a U-Boot derivative in the boot chain has handed you a concrete question to ask your vendor. What evidence supports the claim that your product is unaffected by this class of flaw? Has anyone analyzed the released binary, or is the answer derived from a spreadsheet of version strings? Which fields of an incoming update are actually authenticated before any code touches them? Vendors who can answer those questions with evidence are telling you something important. So are vendors who cannot.
Documentation, inference, evidence
The way I think about it, assurance comes in layers and each layer answers a different question. The certificate tells you what was evaluated and under what assumptions. The corpus connects those artifacts across the whole ecosystem and exposes the shared dependencies and recurring architectures the individual documents obscure. The final layer is evidence about what actually shipped, and it can come from several places. Vendors tracking their forks against upstream and documenting backports. Independent analysis of released binaries. Researchers publishing the failure modes of mechanisms everyone assumed were sound.
Most of the industry stops at the first layer. Procurement checks for the certificate, the checkbox gets ticked, and the boot chain full of forked bootloader code goes unexamined until someone publishes research like Unfit to Boot and everyone scrambles to figure out whether they are affected.
I built the corpus to make the second layer easier, so that the public record can generate the right questions instead of just decorating RFP responses. The questions are the point. A validation record that names a bootloader fork should end in a conversation with the vendor about what is in it, not in a filed PDF. Trust in labels got us the last twenty years of firmware security. Trust supported by evidence is going to have to get us the next twenty.
There is an old Steve Jobs clip from a 1992 MIT Sloan talk that feels newly relevant in the age of AI. In the talk, available here as Steve Jobs MIT 1992 Lecture, Jobs is asked about consultants. His answer is not that consultants are unintelligent or useless. His criticism is more subtle. He says consultants often get to see a lot, analyze a lot, and recommend a lot, but they do not stay with the work long enough to own the consequences.
They do not spend years living with the product, the team, the tradeoffs, the mistakes, the customers, the budgets, the bugs, or the recovery. They may see the fruit, as Jobs put it, but they “never really taste it.”
That distinction matters.
There is a kind of knowledge that comes from observation, and there is a different kind of knowledge that comes from ownership. Observation can make you articulate. Ownership makes you careful. Observation helps you describe what should happen. Ownership teaches you what actually happens when a recommendation meets constraints, incentives, politics, timelines, systems, and human behavior.
There is also a kind of knowledge that only comes from time.
Some problems cannot be understood in a single sitting. You need to carry them around for a while. You read, step away, come back, notice what still bothers you, test a different framing, sleep on it, and then see the thing that was hiding in plain sight. That kind of soaking is not inefficiency. It is often how judgment forms.
That is the parallel to AI.
AI is making analysis abundant. It can read more than we can read, summarize faster than we can summarize, find patterns across larger datasets, generate plausible options, and produce recommendations that sound polished and confident. That is useful. But it is not the same as judgment.
Used poorly, AI becomes consulting at machine scale. It is fast, articulate, and superficially impressive, but disconnected from whether its recommendations actually survive contact with reality.
It can say what should be done without knowing what happened after someone tried to do it. It can identify risks without understanding which ones mattered. It can produce a roadmap without living through the missed dependency, the customer objection, the policy constraint, the budget cut, the migration failure, or the second-order effect six months later.
It can also make us confuse speed of response with depth of understanding. That may be the deeper risk. AI can collapse the slow work of thinking into the first plausible answer. It can make a problem feel resolved before we have really spent time with it. It can produce fluency before we have earned conviction.
That does not make AI useless. It makes the design problem clearer.
The lazy version of the AI story is the self-driving car analogy, namely once the machine becomes safer, faster, or more consistent, the human gets pushed out of the loop. There will be domains where that is true. But much of knowledge work is different. The goal is not only to execute a task correctly. The goal is to understand the problem well enough to make better decisions the next time.
Execution tools can displace. Reasoning tools should compound.
That is why the most interesting promise of AI is not simply that it becomes a better consultant or even a better operator. It is that AI can help humans become better operators.
Used well, AI becomes a way to think with the material. It helps us understand datasets that are too large to hold in our heads. It lets us explore problem spaces from more angles. It helps test assumptions, compare interpretations, surface edge cases, and ask better questions. It can show us patterns we would have missed, but the value is not just the pattern. The value is that, through the process of interrogation, we understand the problem more deeply ourselves.
AI should not shorten our attention so much as deepen what our attention can hold.
A good AI system should help us return to a problem with more context than we had the last time. It should preserve the questions we asked, the assumptions we tested, the contradictions we found, the evidence that mattered, and the places where our understanding changed. It should make it easier to spend real time with the problem, not merely produce an answer faster.
In that sense, the best use of AI is not instant certainty. It is structured patience.
It lets us soak in a problem more effectively. By that I mean it enables us to hold more evidence in view, revisiting prior interpretations, comparing today’s answer to yesterday’s uncertainty, and gradually turning analysis into understanding.
The goal should not be to outsource judgment to AI. The goal should be to use AI to improve the conditions under which judgment is formed.
A good AI system should not merely say, “Here is the answer.” It should help us see why the answer might be true, where it might be fragile, what evidence supports it, what alternatives exist, and what would change our mind. It should help us move from diagnosis to action, from action to feedback, and from feedback to learning.
That is the line between AI as consultant and AI as learning partner.
This is also where many AI products will disappoint. Dashboards full of findings, risks, summaries, and recommendations can look impressive while still leaving the actual burden on the human team. They create the appearance of progress without necessarily improving understanding. The human still has to decide what matters, translate the finding into action, make the change, verify the outcome, and remember the lesson later.
The point is not that findings and recommendations are useless; they are necessary. Findings are the beginning of the loop, not the end of it. Systems that stop there are not doing judgment automation. They are doing analysis transfer.
The more interesting systems will close the loop. They will connect analysis to execution, execution to verification, and verification to institutional memory. Not because humans should be removed from the process, but because humans should be able to reason from a better substrate.
This is where Jobs’ point lands today. The scarce thing is not access to analysis. AI will make analysis abundant. The scarce thing is accumulated judgment, something that only comes from acting, observing, correcting, and learning over time.
Observation gives you language. Ownership gives you consequence. Time gives you depth. Feedback gives you judgment.
Jobs’ critique of consulting was not just a warning about consultants. It was a warning about any tool, process, or person that gets rewarded for sounding right without having to live with whether they were right.
AI will be most valuable not when it becomes the smartest consultant in the room, but when it helps teams build judgment faster, seeing more, acting sooner, sitting with the problem longer, verifying outcomes, and remembering what reality taught them.
The future of AI in knowledge work should not be analysis without ownership. It should be ownership made smarter.
Spend enough years in security and you notice that the people whose judgment you actually trust are rarely the ones with the cleanest credentials.
They are the ones who have been wrong in public often enough to develop taste. Their authority is earned backward, from scars rather than definitions. When they look at a scheme and say, no, that is wrong, and here is the deeper reason, they are not deriving it from first principles. They are recognizing a shape they have been cut by before.
That is worth taking seriously. In security you get little standing to philosophize until you have shipped something, broken something, defended something, or watched something fail. The person who starts from what is identity, or what is trust, but has never lived with the consequences of an answer, barely exists as a respected type. The people who carry real philosophical weight almost always came up through contact with failure first.
So the security philosopher is not the pure theorist. The philosopher is what a survivor of failure becomes once the failures start to rhyme and a reflective habit sets in. Most often that survivor is a breaker, because breaking is the most direct contact you can have with the gap between how a system should work and how it does. But breaking is not the only contact that leaves marks, and that turns out to matter later.
Everyone who matures this way is answering one question, whether or not they say it out loud.
Why does this keep happening?
You earn the right to answer by watching it happen enough times that fixing the bug stops feeling like an answer. Whatever conclusion you settle on is what you turn into. Some decide people do not understand the systems deeply enough. Some, that bad claims go unchallenged too long. Some, that security is downstream of engineering. Some, that the bytes are downstream of institutions and power. Some, that the abstractions themselves are broken. Some, that the failure is intrinsic and the only honest response is to keep hunting.
Those answers create the taxonomy. The genus is philosopher. The species are sorted by the answer each one gives, and by the way each one tries to make that answer true for other people.
The Sage
The Sage believes it keeps happening because nobody understands the systems deeply enough.
The Sage transmits by instantiation rather than argument. The worldview gets built into a tool, written into a book, embedded in a way of working, and you absorb it by use. A fuzzer can teach an entire philosophy of bug finding. Coverage becomes the thing worth chasing. The tool makes the argument every time it runs, so nobody has to persuade you in a thread. The position is already standing in the room, made of working code.
The Sage is contemplative, the monk of the genus. The work is not quiet because it is timid. It is quiet because it expects reality to do the teaching.
The Gadfly
The Gadfly believes it keeps happening because people are wrong in public and nobody corrects them.
The Gadfly’s philosophy exists only in motion. It lives in the thread, the argument, the review comment, the refusal to let an incorrect claim stand. This is Socratic in the original and irritating sense. You learn what is true by watching the argument refuse to die.
The Gadfly may share the Sage’s diagnosis exactly. People do not understand the system. But the method is the opposite. The Sage builds, the Gadfly fights. Both believe misunderstanding is the enemy. They differ on whether the cure is a tool or a wound.
The Builder Evangelist
The Builder Evangelist believes it keeps happening because security is downstream of bad engineering.
This is the breaker who concludes the dramatic failure is only the visible symptom. The real incident happened earlier, in how software was designed, reviewed, deployed, owned, or forgotten. The answer is not more heroics. It is to change how teams build, with security folded into normal engineering life rather than bolted on as a gate at the end or a priesthood that arrives with findings after everyone has moved on. The insight only counts if it propagates into how people actually work.
The missionary energy is the tell. Converts always have it, and this is the convert’s slot. Philosophy that has to spread to count as true.
The Statesman
The Statesman believes it keeps happening because the bytes are downstream of institutions, incentives, and power.
This one did not go deeper into the stack. They went up, out of the assessment shop and into platforms, governments, trust programs, procurement, incident governance, the places where the adversary is sometimes the org chart and sometimes a nation. The wisdom gets cashed out in policy and in who sits at which table.
The risk of the type is floating clear of the actual bytes. The best Statesmen never do. They remember the policy is only real if it changes what happens at the machine, the credential, the incident bridge. The worst become fluent in altitude and lose contact with the ground.
The Theorist
The Theorist believes it keeps happening because the abstractions themselves are unsound.
This is the opposite vector from the Statesman. Where the Statesman goes up toward power, the Theorist goes down toward formalism. Weird machines. Exploitation as programming a machine nobody meant to build. Security as a subset of reliability. Trust as an operational claim, not a noun.
This is the species that comes closest to the academic register, but the ticket was still bought through breaking first. The formalism is trusted because the person doing it has felt the abstraction fail in their hands. That is the difference between earned theory and decorative theory.
The Refusenik
The Refusenik is the apex breaker who declines to metamorphose at all. Not from inability. From principle.
The refusal is itself an answer. It keeps happening because failure is intrinsic to systems of any real complexity, and pretending a framework or a doctrine can end it is the deeper error. There is no theory waiting at the top of the climb, only the next finding. So the Refusenik stays the predator and calls the philosophizing a comfortable retreat from the only thing that is real, the work in front of them.
This is the lower bound of the taxonomy. It proves that becoming a philosopher of the discursive kind is a choice, not an inevitability, and it does so by holding a real position rather than an empty one. Some of the best who ever lived remain here permanently, and they are right to.
The Priest
The Priest is the auditor, the compliance keeper, the custodian of the framework. The Priest is the scandal of the taxonomy, but not for the reason the field assumes.
The reflexive complaint is that the Priest arrived without breaking. No public exploit, no system torn open, no credential earned the old way. By the breaker’s accounting, no scars at all. That accounting is the actual error.
Much compliance really is theater. Much audit work mistakes evidence for reality. Much framework worship trains people to pass inspections while risk keeps moving underneath them. None of that is in dispute. But at scale the Priest is often the only force keeping an organization’s vital signs visible. Compliance read generously is not theater. It is a pulse, one of the few observable ways to ask whether the organization is doing the things it claims to.
And the Priest does have scars, just not the kind the breaker recognizes. The Priest learns by watching organizations lie to themselves in patterns, watching the same control fail the same way across a dozen audits, watching risk reappear in process and ownership long after the technical finding was closed. That is sustained contact with failure. It leaves marks. The breaker simply does not read them as marks, because they did not draw blood the familiar way.
So the scandal is not that the Priest skipped the initiation. It is that the breaker cannot see the Priest’s scars as scars. Seeing them requires fusing two diagnoses that rarely live in the same person. The Builder Evangelist says automate or drown. The Statesman says the real system is organizational health. The generous read of the Priest requires both at once, and most people who came up breaking cannot hold both, so they read the Priest as the enemy.
Sometimes they are right. Sometimes they are only defending their own credentialing system.
The seam in the taxonomy
Two forces are doing the work here. One is diagnosis, the answer to why this keeps happening. The other is temperament, how you try to transmit that answer. The clean version of the theory says these collapse into one, that the way you transmit is downstream of what you concluded and who you are. The Sage builds because he decided depth is the problem and because he is contemplative. The Gadfly fights because he decided public error is the problem and because he cannot leave it alone.
But the Sage and the Gadfly may share the same diagnosis. If they do, they are one species wearing two faces, and the real joint is diagnosis, with transmission as a surface effect. The alternative is that transmission is itself fundamental, that how you choose to make a thing true for other people is a deeper fact about you than the proposition you are trying to make true.
I do not think this is settled, and I am not sure it should be. A taxonomy that resolved it cleanly would claim to know which of two people who believe the same thing is the more serious, on the sole basis of whether they build or fight. That is a claim worth resisting.
The point
Security does not really trust credentials. It trusts scars. That instinct is mostly healthy. It keeps empty abstraction out of the room and gives weight to people who can smell a failure coming rather than just describe one.
But every credentialing system has a blind spot, and the breaker’s is believing that only breaking confers standing. The whole taxonomy is one argument against that belief, because every species on it earned its authority through a different kind of contact with failure.
The breaker learns by cutting into systems.
The builder learns by watching teams repeat the same mistakes.
The statesman learns by watching incentives defeat correctness.
The theorist learns by watching abstractions collapse.
The priest learns by watching organizations lie to themselves in patterns.
The refusenik learns by never looking away from the hunt long enough to be comforted by a story about it.
Five are reflective and one refuses reflection, but all six are forms of earned contact, and none is the only one that counts. What you become is the answer you settle on for why this keeps happening, and the way of making it true for others that you cannot help but reach for.
The breakers who refuse the question stay hunters. The keepers who never broke anything are mistrusted for it, sometimes fairly and sometimes not. Everyone else becomes a philosopher of one species or another, whether or not they would accept the word.
Why textualism, original public meaning, and AI governance all turn on the same uncomfortable fact: intent does not travel unless it becomes part of the record.
There is an old fight in legal interpretation about where meaning lives.
Intentionalists look for purpose. They ask what Congress meant to do, what the drafters were trying to accomplish, and what problem the law was meant to solve. Legislative history matters in this view because floor speeches, committee reports, drafter notes, and surrounding debate can reveal the intent behind the enacted words.
Textualists are skeptical of that move. They argue that the law is the text that was enacted, not the private intentions of the people who helped write it. The words are the law. The meaning is what the text would have conveyed to a reasonable reader, not what a motivated advocate can later reconstruct from a convenient committee report.
Originalists make a related move in constitutional interpretation. Original public meaning says the Constitution means what its words would have been understood to mean by the public at the time of ratification. Not what a drafter secretly intended. Not what a later judge wishes it said. The meaning is anchored in the text, the historical context, and the interpretive record available at the relevant time.
You do not have to be a textualist or an originalist to see the infrastructure point.
Once a system has to act on language, intent is not enough. Intent has to travel through something. It has to travel through text, context, rules of interpretation, and a record of what was available to the interpreter when the decision was made. Otherwise, “what I meant” becomes an after-the-fact story.
Anyone who has spent time debugging prompts has run into the same problem.
You thought you were telling the model to be careful. You were actually telling it to hedge every answer into uselessness. You thought you were asking it to be concise. You were actually removing the context it needed to be right. You thought you were giving it freedom to reason. You were actually giving it permission to invent.
The words on the page were doing work you did not realize they were doing.
That is the uncomfortable part of working with language models. A prompt is not what you meant. It is not the conversation you wish you had with the model. It is not the background assumptions in your head. It is not the thing you would have clarified if another person had looked confused.
A prompt is the artifact the system received, interpreted, and acted on.
That is why the analogy to textualism matters. In human organizations, we constantly rely on unwritten context. We rely on shared history, institutional memory, tone, relationships, and the ability to stop and ask, “wait, what did you mean by that?” Human communication survives ambiguity because humans have recovery mechanisms.
AI systems do not get those recovery mechanisms for free.
There is no hallway conversation where you explain that when you said X you obviously meant Y. No shared institutional memory unless it is supplied. No unstated assumptions unless they are embedded somewhere in the context. No ability to rely on “what everyone knew” unless what everyone knew made it into the materials the system was given.
The system only has the record.
This does not mean the model is a textualist judge. It is not. The originalist’s reasonable reader is a legal construct. A model is a versioned, probabilistic system operating inside a specific runtime. The same words can produce different behavior under a different model, a different instruction hierarchy, a different retrieval result, a different tool definition, a different policy layer, or a different configuration.
So the lesson is not that the model found the one correct meaning of your prompt. The lesson is that your intent did not travel.
Intent does not become operational just because it existed in your head. Context does not exist just because your team would have understood it. Purpose does not govern the system unless it is encoded in the materials the system actually sees.
This is the prompt engineering lesson people often miss. Prompt engineering is not merely a collection of magic phrases, though some prompt patterns work for model-specific reasons that are not obvious from the surface text. At its core, prompt engineering is drafting. It is the work of turning intention into operative language.
That is why it feels so much like legal drafting. You are not writing what you wish the system understood. You are writing the thing the system will act on.
In production AI systems, that “thing” is larger than the sentence the user typed.
The operative prompt includes the system instruction, the developer instruction, the retrieved documents, the tool definitions, the prior turns, the examples, the policies, the memory, the model version, the ranking logic that decided which facts were included and which were left out, and the configuration that shaped how deterministic or creative the output could be.
That is the effective prompt.
This is where context engineering starts to absorb prompt engineering. The hard problem is no longer merely finding better words to ask the model. The hard problem is constructing the interpretive environment in which the model can behave reliably enough, predictably enough, and accountably enough for the job it is being asked to do.
Legal interpretation has always depended on more than raw text. Textualists still need grammar, usage, canons of construction, dictionaries, historical context, and an account of the reader. Original public meaning still needs evidence of how words were used at the relevant time. Even the most text-centered theories need a record.
AI systems make that dependency operational.
What was the user prompt? What system instruction controlled it? What policy applied? What documents were retrieved? What facts were omitted? Which tool schemas were available? Which model ran? Which version of the surrounding system produced the answer?
These are not merely implementation details. They are the interpretive record of the system.
That is why the prompt is the meaning. Not because the model is a perfect reader. Not because the words have one stable meaning across all systems. Not because intent is irrelevant. But because the system can only act on what made it into the operative prompt. If something was not in that record, it was not available to govern the system’s behavior.
At that point the issue changes. It is no longer just an interpretation problem. It becomes an evidence problem.
For a toy prompt, failure looks like annoyance. The answer was too verbose. The model misunderstood. The prompt needed tuning.
For a production system, failure looks like a governance gap. The system made a decision, but nobody can reconstruct what it was asked, what it knew, what it was allowed to do, what policy constrained it, what model produced it, or which version of the surrounding context shaped the result.
If you cannot reconstruct the effective prompt, you cannot explain the output. If you cannot explain the output, you cannot evaluate whether the system behaved correctly. If you cannot evaluate whether it behaved correctly, you cannot govern it.
This is why prompts need version control. Retrieved documents need provenance. Tool definitions need change history. Policy layers need to be preserved. Model versions need to be tied to outputs. Evaluations need to capture not just the answer, but the context that made the answer plausible.
But preservation is only the first step. A record without evaluation is archaeology. A record without monitoring is trivia. A record without enforcement is a diary. Governance requires the record, but it also requires machinery that acts on it: tests, policy gates, drift detection, human review, incident analysis, and change control.
Otherwise, the organization has outputs but not governance.
A log tells you what happened. A record tells you what the system was asked to do, what it was allowed to know, what constraints it was operating under, and why the resulting behavior was plausible from the materials available to it.
Without that, accountability collapses into storytelling.
The team says the model was supposed to be careful. The prompt says “avoid unsupported claims,” but the retrieved material was stale. The policy says “follow the customer’s procedure,” but the procedure was missing from context. The audit says the system made a decision, but nobody can reconstruct the versioned bundle of instructions, documents, tools, model, policies, and configuration that shaped it.
At that point, you are not governing the system. You are narrating around it.
That is the deeper lesson from textualism, originalism, and prompt debugging.
Text is never just text. It is text plus an interpretive frame, text plus context, text plus a record of what counted as meaning at the time. In law, we fight about that because rights, obligations, and institutional power depend on it. In AI systems, we are turning that fight into software.
The prompt is not your intent.
The output is not self-explaining.
The effective prompt is the record.
You may not fully own the model. You may not own the provider’s policy stack. You may not own the training process, the safety layers, or the hidden machinery that shapes the output.
But you can own your side of the interpretive record. You can preserve what you supplied, what the system saw, what it was allowed to do, what it returned, how it was evaluated, and what changed afterward.
That is the infrastructure of accountable AI.
The prompt is the meaning because the prompt, properly understood, is the record the system acted on.
You either govern that record, or you inherit someone else’s explanation.
Back in the 1990s, I was doing some consulting for DigiNotar, yes, that DigiNotar. They had CA facilities in a data center whose perimeter still had WWII-era anti-tank obstacles, large concrete barriers sometimes called “dragon’s teeth.” Of course, this was an artifact of the facility’s history, but data centers are designed from a security perspective with layers of physical protection, including barriers, mantraps, biometrics, individual vaults with cages, individual racks with their own locks and biometrics, cameras, and more. The threat of physical theft, destruction, or manipulation is exactly what these facilities are designed to mitigate.
When building a CA inside one of these facilities, we design yet another layer of protection. Administration networks are segmented from transaction networks, interconnects from supporting infrastructure, the issuance environment from the systems holding root keys. We add our own physical segmentation on top of that so we can build controls around multiple parties being necessary for the more sensitive operations, while still letting routine hardware maintenance happen on the schedule the SLA needs.
These are all useful and important things, but the reality is that CA key material is not likely to be physically stolen. It is more likely to be compromised from the outside. We solve these problems through design, not by writing more code.
Meaningfully measured code forces design upstream of the code itself.
A measurement of a monolithic blob proves almost nothing useful. A measurement that names a specific role, a specific security domain, and a specific assertion the verifier is supposed to act on, proves something. The roles, the domain boundaries, and the questions the verification has to answer have to exist before any code is written, or the attestation is a signature on nothing in particular.
In operating system design, we have similar problems. In naive systems, we load cryptographic keys into memory on a running, network-connected system, accidentally exposing ourselves to memory-disclosure bugs where a network attacker can steal keys. Heartbleed is the canonical example, but the class is what matters. We do this because it is simpler and faster, but it is also less secure. As systems designers, we address this by moving those keys out of the process of the network-connected application and into a different user context. That way, an attacker cannot simply get the network-connected service to dump memory. They have to get persistence and cross a kernel-enforced user boundary.
This is old wisdom. Least privilege and privilege separation exist because network-facing code should not also be the thing that controls the keys.
A parallel showed up in early cryptocurrency exchanges. Hot wallets were used as signing oracles because the design and deployment work needed to prevent that had not been done, and many of the high-profile compromises of that era trace back to that gap. The exchanges that survived learned to put boundaries between the wallet and the network. That boundary did most of the work. The dragon’s teeth around the building did the rest.
When third parties need to rely on external services operating in these environments, they often rely on auditors to attest that management assertions about operational practices are actually being followed. These assessments are usually performed by CPAs, not security specialists, which can limit their value. They also often rely on sampling a small portion of transactions to confirm that the controls being evaluated are being followed. That sample is drawn from evidence provided by the entity being audited, which is also the party paying for the audit.
All of the things discussed above help bring some minimal level of transparency and verifiability, but it is turtles all the way down, layer on layer, none of them reaching the runtime where the actual compromise happens. This is where confidential computing, and solutions like Private Cloud Compute, start to matter.
Policy changes meaning in this model. In the traditional assurance world, policy is a written promise. The CA publishes a CP or CPS, the operator commits to following it, and the auditor samples evidence to decide whether that promise was kept. In a runtime-evidence model, policy becomes part of the mechanism. A measured binary evaluates a specific policy, produces a decision, and the digest of that policy travels with the evidence. The shift is from policy as promise to policy as enforcement, from “trust us, this is what we do” to “this is the policy the measured system actually applied.”
Apple’s Private Cloud Compute is the worked example. PCC nodes attest to the binary they are running, refuse to do work for clients that cannot verify that attestation, and publish every production build for public inspection. The user’s device, not Apple, decides whether a given node is acceptable. That inversion, the relying party verifying the service rather than the service asserting to the relying party, is the part of the pattern that matters. The pieces are not new individually. The combination, at the scale Apple shipped it, proves the pattern is real. The third-party security reviews prove the architecture is serious. Attacks on confidential computing do not refute that point. They prove there is now a boundary worth attacking, measuring, and improving.
Concretely, a CA built on the Private Cloud Compute pattern looks like this.
Issuance is split into two attested components. The first, the registration authority, takes the certificate request, resolves identity from authoritative sources, evaluates the issuance policy, and produces a signed authorization context. The second, the signing oracle, holds the CA private key and produces the signature. Each runs in a separate attested enclave, and each is measured separately. This means policy can evolve without re-measuring the key-custody component, and keys can rotate without re-measuring the policy component.
The policy layer matters here too. Each component is not just running code, it is making a verifiable policy decision before it acts. The RA decides whether the request is authorized and the identity evidence is sufficient. The signing oracle decides whether the RA, the request, and the authorization context are acceptable before it signs. The evidence does not just say which binary ran. It also says which policy that binary evaluated.
The two components do not trust each other because they are on the same network. They trust each other through attestation, mutually verified at every connection, and the signing oracle does not merely accept the RA’s conclusion. Before it signs, it independently verifies the RA’s attestation, checks that the authorization context is fresh, confirms that the request is bound to an allowed profile, and verifies that the policy facts asserted by the RA match the evidence presented to the oracle. A compromised RA, even one with its own signing key, does not get to mint an out-of-profile certificate, bypass attestation, or turn the CA key into a general-purpose signing oracle.
The CA private key is not loaded by the operator. It is held by a custodian, a hardware security module, a cloud KMS, or another enclave, and it is wrapped so that the custodian will release it only to a signing oracle whose attestation matches a published image. The list of acceptable images is small, public, and updated through a documented process. An operator who runs a different binary, however benign the reason, does not get the key. The dragon’s teeth around the data center are still there. They no longer have to do the whole job.
Both the RA binary and the oracle binary are built from public source and are reproducibly buildable. Anyone can rebuild from the published sources, compare their measurement to the one in the attestation, and confirm that the two match. This is the part of the model that makes trust mean something specific. Not the operator’s word, not the auditor’s snapshot, not the CA’s policy statement, but the build process and the published source. To verify what a particular issuance was actually done by, you would not need to be admitted to the data center. You would need a compiler.
Each issued certificate is accompanied by a portable evidence bundle, signed by the attested issuance system. The bundle names the binary that produced the signature, the attestation root that vouched for the binary, the RA policy decision, the oracle policy decision, the identity assertion the RA accepted, and the inputs the oracle independently verified before signing. A relying party who trusts the chip vendor’s attestation root can determine for themselves whether the issuance was performed by code on the published list, against the policy on the published list, by an RA that accepted the identity claim it claimed to accept. The CA is not asked to be trusted. The CA is asked to produce evidence.
None of this removes the HSM, the auditor, or the operator. The HSM is still excellent at the threat it was built for, and a custodian holding a key wrapped to an attestation policy is still doing HSM work under the hood. The auditor is still needed to attest that the published policy is sensible, that the source matches the binary, that the threat model is honest, and that the runbook is followed in the moments where attestation cannot help. The operator is still needed to run the infrastructure and respond when things break.
What changes is what they are asked to prove.
Today, a relying party mostly gets institutional assurances. The CA says it followed its policy. The auditor samples evidence and says the controls were operating. The operator says the production system was the one described. Those are useful assurances, but they are indirect. They do not let the relying party inspect the actual path between a request, a policy decision, and a signature.
A Private Cloud Compute style CA changes that. It turns the issuance path itself into evidence. The question is no longer only whether the CA says it followed the rules. The question becomes which measured binary evaluated this request, which measured binary signed it, which policy digest was used, which identity evidence was accepted, what validation methods were used during issuance, and whether all of that matches the public commitment the CA made.
When the source is open and reproducibly buildable, that evidence includes a hash of the code that made the decision and signed attestations about the runtime elements that went into that decision. When the code is not open source, third parties can come in and validate the source, the build process, and the correctness of the claims, as Apple did with Private Cloud Compute. The public hashes then let others verify that the code claiming to provide these guarantees is, in fact, the code that ran.
Open source is not magic, and the point is not faith in “many eyes.” The point is that this shifts the emphasis from betting on physical security and operational practice audits to secure system design and cryptographic evidence about what code actually ran and what it actually did.
That is the threat model mismatch, and it is not only a CA problem. We built the WebPKI around buildings, cages, ceremonies, HSMs, and audits because those were the tools we had. We did the same thing cryptographically. We built systems around the assumption that factoring large composites and solving discrete logs on elliptic curves were out of reach. Q-day changes that assumption. Runtime compromise changes the operational assumption just as fundamentally.
We apply the same instincts in any environment we want to call high-assurance. They still matter, but most of the failures we care about are not physical failures. They are logical, remote, operational failures in the runtime path. The rate of change makes that gap wider every year. Annual audits are retrospective, and between them systems change thousands of times, so what the auditor described is rarely what is actually running when a relying party sees a certificate.
Cryptography turns security problems into key-management problems. AI turns assurance problems into runtime-evidence problems. Once agents are making decisions, calling tools, and changing state, the question is no longer what policy you wrote or what control an auditor sampled. The question is what actually ran, what it saw, what boundary contained it, what policy constrained it, and what evidence survived execution.
A Private Cloud Compute style CA gives us a way to make that path visible, attestable, and independently verifiable. The same pattern applies wherever the gap between what we say a system does and what it actually does at runtime matters.
In the mid 90s I worked at a company called Cybersafe. Today it would get labeled an IAM/SSO vendor. What we actually built was a first-generation security platform: Kerberos, password management, PKI-based MFA, key management, host intrusion detection, and what would now be called zero trust access. The company failed for the usual startup reasons. People. Corporate Politics. Timing. The technology was a decade ahead of its market.
One debate from that period has stayed with me. As we expanded into host intrusion detection, the question of automated response kept surfacing. Could a system safely act on its own to contain an intrusion in progress? Drop a connection. Kill a process. Isolate a host. Nobody on the team could imagine a credible answer. The false positive risk was unbounded. The response itself could be weaponized. The rule sets were not trustworthy enough to delegate authority. We shipped detection and let humans make the call.
That debate has an answer now, and it is not the one we expected. Automation on the offensive side is not new. Worms, exploit kits, credential stuffing, and phishing infrastructure have been automated for decades. What is new is broad delegated judgment at machine speed, in the hands of people who do not have to worry about false positives because the blast radius is somebody else’s network.
What the report actually shows
The interesting question is not whether AI helped produce a zero-day. That was inevitable. The interesting questions are operational. What kinds of systems make bad machine judgment cheap enough to deploy at scale. What kinds of defensive systems are still pretending human review is the control boundary.
Google Threat Intelligence Group’s latest AI Threat Tracker report documents the first zero-day exploit that GTIG says it has high confidence was developed with AI assistance. The headline framing is technically correct. The specifics tell a more interesting story.
The exploit was a Python script that bypassed 2FA on an open-source web-based system administration tool. It required valid user credentials in the first place. The criminal group planned a mass exploitation campaign, and Google disrupted it through responsible disclosure to the vendor. GTIG identified the artifact as AI-developed because the code carried obvious tells. A hallucinated CVSS score. Textbook Python formatting. Detailed help menus. Educational docstrings characteristic of training data. The artifact still carried the seams of its production.
This is not the LLM failing at the hard part. The vulnerability itself is a real find. GTIG specifically notes that the 2FA flaw stems from a hardcoded trust assumption, a high-level semantic logic flaw of the kind that fuzzers and static analyzers tend to miss but that frontier LLMs can reason about by reading developer intent. The model did discovery work that previously required a competent human auditor. Where the operation broke down was in weaponization. The attacker shipped an artifact that still looked like a tutorial.
This is a familiar failure pattern showing up on the offensive side for the first time. Fluency reads as competence. The attacker trusted an artifact with hallucinated metadata and educational comments still attached because it looked like a real exploit, in the same way over-eager engineering teams hand agents production credentials because the agent sounded like it knew what it was doing. The criminals here got bitten by the same dynamic that has been producing outages and data loss in vibe-coded production systems for the last eighteen months. The substrate is doing some of the work of inviting the misconfiguration.
Hultquist’s thread on the report is hedged correctly. The importance is the trajectory, not this specific specimen. Pull the camera back and the rest of the report is more interesting than the lede.
Three things worth surfacing
APT45 sending thousands of repetitive prompts. The North Korean group has been observed using recursive prompting to analyze CVEs and validate proof-of-concept exploits at scale. That is the industrial-scale answer to LLM variance. Solve the quality problem by amortizing across volume, then have humans cherry-pick the outputs that survived validation. The same statistical strategy that makes modern fuzzing work, applied one layer up the stack. The model does not have to be reliable. The pipeline has to be cheap enough that unreliability does not matter.
CANFAIL and LONGSTREAM using LLM-generated decoy code. A Russia-nexus intrusion cluster has been deploying malware that uses LLM-generated code to conceal malicious functionality. GTIG documented LONGSTREAM containing 32 instances of code querying the system’s daylight saving status, repetitive benign-looking activity used to camouflage the malicious core. CANFAIL carries similar filler logic with LLM-generated comments self-describing the decoy blocks. The stylistic noise of LLM output is becoming the obfuscation layer. The verbose docstrings. The textbook structure. The over-explained variable names. These used to be tells. They are now camouflage. Any heuristic built on the AI-tell will start producing false negatives.
The wooyun-legacy skill plugin. A specialized GitHub repository is being distributed as a Claude code skill plugin that integrates a distilled knowledge base of over 85,000 real-world vulnerability cases from the Chinese bug bounty platform WooYun (2010 to 2016). This is the supply side of the same market. Skill packs are tooling. Tooling gets distributed. The economic logic for adversarial skill packs is identical to the economic logic for legitimate ones. Any platform hosting them inherits a familiar problem. App stores and package registries have been working through it for two decades. Making trust decisions at distribution scale about code from parties you cannot directly inspect.
Both sides are running on the same substrate
On the defensive side, Google is using Big Sleep to find vulnerabilities and CodeMender (Gemini-driven) to fix them automatically. The criminals are pulling from a model class indistinguishable from the one Google is running its defensive tooling on. Both sides have access to the same substrate. The differential collapses to data quality, harness sophistication, and discipline around permissions.
That last one is the part the 90s HIDS conversation did not anticipate. It is also the part that should be the least surprising. The controls discipline did not get easier because the platform got more capable. If anything the gradient got worse. A confused regex IDS in 1999 had a bounded action space. The rule set was enumerable. You could write down what it would do wrong. A confused agent in 2026 has whatever action space its credentials grant it, which in most deployments is more than it should. The fluency that made it easy to give the agent broad permissions in the first place is exactly the property that makes its failures look reasonable in the moment.
The race Hultquist refers to is real, and it has started. The race is not about model capability. Both sides are running models from the same vendors, often the same model. The race is about who has better-curated data feeding their harnesses. Who has stricter discipline around what their automation can touch. Who has the institutional memory of what happens when you delegate authority to a system whose judgment you cannot audit in advance.
The HIDS debate from the mid-90s got an answer. It came from the other side of the wire. Not because defenders learned how to trust autonomous judgment, but because attackers learned they did not need to. They could delegate broadly, externalize the blast radius, and let volume compensate for judgment. The defensive answer cannot be more vibes, broader credentials, and better prompts. It has to be the inverse. Narrower authority. Better harnesses. Replayable decisions. And institutional memory about what happens when fluent systems get mistaken for trustworthy ones.
Back in 2000, the rule of thumb at Microsoft was that each employee needed to average roughly $600K in top-line revenue. Inflation adjusted, that is about $1.1M to $1.2M today. Microsoft was a high-margin software monopoly at peak, so it is not a universal benchmark, but it gives a sense of what disciplined operating leverage looked like even at a company printing money.
Over the last decade, and especially during the COVID-era zero-rate and QE environment, many companies responded to dysfunction by hiring around it instead of fixing it. Cheap capital reduced the pressure to make hard operating decisions. Necessity is the mother of invention, but cheap money suppressed that necessity for a long time.
Then two things changed at roughly the same time. Rates went from zero to five, and Section 174 of the tax code stopped letting companies expense software developer salaries in the year incurred. The R&D amortization rule from TCJA kicked in for the 2022 tax year, forcing five-year amortization domestically and fifteen years for work done offshore. At the exact moment capital got expensive, a major software-company cost center became less friendly from a cash-tax and after-tax economics perspective.
Now AI has added a new pressure. Companies are adopting AI quickly, but we are still early. Much of what is happening inside enterprises is still R&D, experimentation, platform buildout, workflow redesign, and internal tooling. That work is not free. It comes with token costs, infrastructure commitments, GPU capacity, vendor contracts, and a lot of expensive trial and error.
Jensen Huang has made the point, in characteristically aggressive form, that if he pays someone $500K, he expects them to use a meaningful amount of compute to become more productive. Whether or not you take the specific numbers literally, and you probably should not since Nvidia sells the machines that consume those tokens, the economic point matters. AI spend has to come from somewhere.
That is the part many layoff narratives miss. Companies are not simply replacing workers with AI. They are also reallocating budget toward AI. Token budgets, model access, inference costs, internal AI platforms, data infrastructure, and R&D commitments are becoming real line items. To fund them, companies are looking at the headcount they accumulated under different interest-rate assumptions, different tax assumptions, and a different view of software demand.
There is also a demand-side story. COVID pulled years of enterprise software adoption into eighteen months, and a lot of what gets reported as growth now is ARR rotating through M&A rather than new logos landing. In parts of the market, revenue is moving around as much as it is expanding.
That is the real backdrop for the wave of layoffs. AI is the story being told on earnings calls. The reality is accumulated management debt finally meeting a cost of capital that punishes it. Layers of process. Unclear ownership. Duplicated work. Headcount that grew faster than execution improved. And now, on top of that, companies need to make room for a new class of AI-related spend.
The pressure also lands hard on old farts like me. We are expensive. And to be honest, some of us (not all) do not want to change how we work or keep up with how the technology is evolving. That makes us easy targets when finance needs to hit a cost number. AI gives the story a forward-looking sheen, but the underlying move is simpler: reduce expensive headcount, flatten layers, correct years of operational laziness, and redirect budget toward the new thing everyone believes they must fund.
AI is real. The layoff narrative around it usually is not. When you read a layoff announcement blaming AI, you are mostly reading a press release about cost of capital, tax policy, demand pull-forward, AI infrastructure spend, and an org chart that finally got too expensive to defend.
“If you agree that AI will help attackers discover and exploit vulnerabilities 10-100x more easily, then your excess attack surface has also just become 10-100x more of a liability. The right defensive strategy is to prioritize reducing attack surface and trusted computing bases.”
The argument is right. It is also not new.
We have been working on this problem for fifty years
Operating system designers gave this set of principles a name in 1975. Saltzer and Schroeder published The Protection of Information in Computer Systems and laid out economy of mechanism, least privilege, separation of privilege, complete mediation, fail-safe defaults, and open design. The Orange Book formalized “trusted computing base” a few years later, with the central observation that the security of a system depends on what is inside the TCB, and that smaller TCBs are easier to make trustworthy than stronger ones. The microkernel debate that ran from Mach through L4 was an argument about how aggressively to apply these principles to commodity systems. seL4 went further and produced a formally verified microkernel in 2009, demonstrating that the principles could be pushed all the way to mathematical proof.
The same ideas show up everywhere once you look. Chrome’s site isolation is privilege separation applied to the browser. OpenBSD pledge and unveil are least privilege applied to userland. Linux namespaces, capabilities, and seccomp are mediation primitives. CHERI takes the same intuitions down into the instruction set. GlobalPlatform Security Domains are the smart-card-world version of compartmentalized trust, with separate keysets, separate trust roots, and isolation between issuers, verifiers, and applications on the same chip.
None of this is new vocabulary. Security domains. Privilege separation. Attack surface reduction. Trusted computing bases. We have known the names of these things for decades, and we have known what to do about them.
What AI changes is the math, not the principles. Excess privilege has always been a liability. The probability of it mattering on any given day was low enough, and the timescale on which it mattered was long enough, that organizations could carry oversized TCBs and broad blast radii in the backlog as “things we should clean up someday.” AI compresses the timescale and raises the probability. The slack that was tolerable on a five-year cleanup horizon is not tolerable on a six-month one. Dai Zovi’s 10-100x is a multiplier on the cost of carrying slack, not a discovery about whether slack should be carried.
The OS tradition assumed you owned the layer below the boundary
There is one place where the classical OS framework needs an extension before it covers the world we are actually deploying into.
The kernel could enforce process isolation because the kernel was below the processes. The hypervisor could enforce VM isolation because the hypervisor was below the VMs. The trust property was “I control the layer below the boundary, so the boundary is meaningful to me.” Every classical OS-level guarantee depends on that.
Cloud broke that assumption. AI workloads, which run on cloud GPUs and orchestration infrastructure that almost nobody owns, intensify the break. The layer below your workload is operated by someone else. Their hypervisor, their firmware, their physical facility, their scheduling. The classical principles still apply, but their enforcement mechanism is gone.
Reduction is necessary. Reduction is not sufficient. Once you have shrunk the attack surface and the TCB to something defensible, you still have to prove that the small thing you reduced to is the small thing actually running, and that what it just did is what you said it would do. Without that proof, the small thing is functionally indistinguishable from the large thing. An attacker who replaces your tiny attested signing service with a tiny lookalike has bought themselves all the same access at a lower cost.
The defensive posture in an AI-leverage world is not just smaller. It is smaller and provable.
Law #3 did not go away
There is also one law older than the OS-design principles that the cloud security pitch of the last decade has spent a lot of energy pretending to repeal.
Microsoft’s Ten Immutable Laws of Security were published by Scott Culp in 2000. Law #3 is the relevant one here. If a bad actor has unrestricted physical access to your computer, it’s not your computer anymore. The marketing for confidential computing has, in effect, been an extended argument that hardware-encrypted memory and remote attestation make Law #3 obsolete on cloud infrastructure. They do not, and the research record is clear that they will not.
Cloud TEEs share microarchitectural resources with the hypervisor and with co-tenants. That is what produces the side-channel catalog. Cloud providers have physical access to every server they operate. That is what produced TEE.Fail. Hardware roots of trust have a shelf life because they live on the same silicon as everything else, and that silicon is in the operator’s possession. None of these properties are bugs. They are what “running on hardware somebody else owns” means.
The criticism in those pieces is specific. It is about the gap between what cloud TEEs are sold as doing (defeating the operator) and what they actually do (making narrow verifiable claims to relying parties about specific operations). The criticism is not that the underlying assurance technology is useless. The technology delivers exactly what it was originally designed to deliver, in the contexts where the original threat model holds. The marketing has been run over those contexts.
Where the assurance property actually delivers
The assurance property does deliver, where the model fits. The model fits when the hardware is in the user’s possession, when the device is discrete and tamper-resistant, and when attestation is used to prove “the key in this request lives on this specific device and has never left it” rather than to prove “the operator of the rack cannot read your memory.” That is the threat model the technology was designed for, and it has been working in production for a long time.
A few examples of the pattern done honestly.
YubiKey PIV attestation. The YubiKey can produce an attestation certificate, signed by Yubico’s manufacturer key, asserting that a private key was generated on this YubiKey, has the slot and policy attributes you expect, and is non-exportable. Yubico documents the protocol clearly. The trust property is sharp because the device is sharp. Discrete silicon, tamper-resistant package, manufacturer chain you can pin against. Law #3 still applies, and it cuts the right way: the user has unrestricted physical access to the YubiKey, and the YubiKey is the user’s computer.
Apple Secure Enclave for SSH agents.Paprika and Secretive are SSH agents that store the private key in the Mac’s Secure Enclave Processor. The application processor never sees the key, and even root on the Mac cannot extract the key material. Root can still cause the key to be used through the legitimate signing API, modulo whatever consent prompts apply, but extraction itself is what the SEP boundary is built to defeat. The user owns the laptop, the key is on a physically separated processor on the same SoC, and the threat model (other applications on the same device, or malware that compromises the application processor) matches what the SEP was built for.
Smart cards and HSMs. GlobalPlatform Security Domains, the Yubico PIV applet, hardware-backed PKCS#11 tokens, FIPS 140-3 Level 3 modules. Discrete silicon, tamper-resistant packaging, attestation chains rooted in manufacturer keys. The model that worked in the late 1990s and that still works today, because the threat model has not drifted.
PeculiarVentures/attestation is the verification side of all of this. Parsing, validating, and reasoning about attestation evidence from these various sources. Attestation without a verifier is a claim. Attestation with a verifier is something the relying party can act on.
The common shape across all of these is that the user owns the hardware, the boundary is physical, and the attestation chain anchors in a manufacturer key whose threat model the user can actually evaluate. Law #3 is honored rather than denied.
Transparency is the other cross-machine extension
There is a second extension of the classical OS-design tradition that matters for the AI-leverage world, and that composes with attestation in important ways.
Saltzer and Schroeder’s open design principle says the security of a system should not depend on the secrecy of its mechanism. The cryptography community has applied this rule to algorithms for decades. The systems community has been slower to apply it to operations. What is the rack actually doing right now? and what has it done in the past? are operational questions, and historically the answer was “trust the operator’s audit logs.”
Transparency logs are the operational extension of open design. The idea is to publish what a system is doing to an append-only public log, with cryptographic proofs that the log cannot be retroactively modified, and to design the relying party to require evidence from the log before trusting any operation. Multiple independent witnesses cosign the log so that no single party can serve different views of reality to different relying parties.
The pattern is in production at scale. Certificate Transparency requires every WebPKI certificate to be logged publicly before browsers will trust it, which converts CA misissuance from “discovered by accident, sometimes” into “discovered by anyone watching the log.” Sigstore applies the same model to software signing, with every signature published to Rekor and consumers able to require log inclusion before accepting a binary. Google DeepMind’s Verifiable Data Audit was an early attempt to apply the same model to data access in healthcare. The infrastructure is consolidating at transparency.dev, and C2SP standardizes the interoperability primitives: tlog-tiles, the witness and cosignature protocols, signed-note, and static-ct-api.
Attestation tells a relying party “this code is running right now.” Transparency tells a relying party “this code has been published, reproduced, and witnessed by parties whose collusion would be visible.” The two compose. Apple’s Private Cloud Compute is the most prominent recent example. Every production build is published to a transparency log, user devices will only communicate with nodes whose attested measurement matches the log, and Apple released a virtual research environment so anyone can verify the build claims independently. Google’s Project Oak was an earlier expression of the same combination, building remote attestation against publicly-published binaries as the foundation of trust. The Merkle Tree Certificates draft, now a working group document in the IETF’s new PLANTS working group, extends the same logic to TLS at scale, replacing traditional X.509 issuance with batched, transparency-native cert formats designed for the shorter lifetimes the WebPKI is moving toward.
The relevant property for the AI conversation is that transparency reduces the number of parties you have to trust to one less than would otherwise be required. With attestation alone, you trust the manufacturer of the silicon. With transparency, you trust any of the witnesses to be honest, plus the manufacturer of the silicon. That asymmetry is what makes transparency the right tool for environments where the operator might be the adversary.
What this leaves for server-side TEEs
Bounded usefulness, designed honestly.
Server-side cloud TEEs do not defeat the operator. They produce narrow verifiable claims that a relying party can check against their own trust anchors. This signing service ran this image at this measurement. This certificate was produced by this enclave for this RA. This policy was applied. This key was attested as non-exportable by the HSM that signed. Each of those is a useful property. None of them is “the operator cannot see your data.” Building an architecture that pretends otherwise is how organizations end up with a single point of failure they did not know they had.
I have been building GoodKey CA as a worked example of the bounded-usefulness pattern. A certificate authority is a useful test case for this kind of architecture, because the trust property is sharp and the threat model is well understood. The shape of the answer is mostly classical OS design pulled across machine boundaries, with hardware-anchored trust at the endpoints and a deliberately bounded intermediary in the middle.
Each enclave is a security domain. RA, CA, and HSM are independent compartments. Each has its own measured image, its own keys, and its own attested boundary. Compromising one does not compromise the others. Privilege is separated by design rather than by policy.
The TCB inside each domain is small enough to characterize. Each enclave runs a single-purpose deterministic image. The measurement is one number. The image is reproducible from source. There is no general-purpose runtime to subvert and no orchestration sidecar to gain a foothold from. AWS Nitro Enclaves were the deliberate choice over SGX or TDX. The architecture uses VM-level isolation with dedicated CPU and memory rather than carving enclaves out of shared-cache, shared-core silicon, which reduces a large class of the microarchitectural side-channel exposure that the SGX and TDX families have to grapple with. Dedicated resources, minimal hypervisor, deterministic measurement.
Mediation is complete and inside the boundary. Every signing operation goes through the policy evaluator (Cedar) inside the enclave. Authorization is part of what is attested, not external to it. A compromised RA cannot lie about what policy was applied, because the policy evaluation was inside the measurement.
Trust is not transitive. When the RA tells the CA that a client attestation passed, the CA does not believe it. The CA re-runs the verification itself, against its own registered verifier, before signing anything. This is the cross-machine version of “the kernel does not trust userland’s claim that a syscall is authorized.” The CA does the check itself, every time.
Per-operation attestation, not per-boot attestation. The CA produces a fresh Nitro attestation for every certificate it signs, with user_data set to SHA-256(certDER || raKeyFingerprint). That binds this specific certificate to this specific enclave with this specific RA on the other end of the conversation. A boot-time attestation tells you the box looked right when it started. A per-operation attestation tells you the box looked right when it did the thing you actually care about.
Hardware-anchored trust at the endpoints. The signing keys themselves live in a hardware HSM with discrete-silicon attestation rooted in the Marvell manufacturer chain. The clients prove they hold hardware-protected keys via TPM or device attestation. The Nitro layer in the middle does not have to defeat AWS to be useful, because the actual key material is protected by a different boundary that AWS does not own, and the evidence on the wire is anchored in trust roots the relying party already trusts.
Operations published to a transparency log. The CA’s attested measurements, policy versions, and issuance records get logged to an append-only structure with multi-witness cosigning. The operator still chooses what to submit. What the operator does not get is the ability to retract entries after the fact, modify history, or serve a different version of the log to a different relying party without those parties detecting the divergence. A relying party’s confidence that the system has been running honestly over time stops being a function of trust in the operator’s audit logs and starts being a function of properties that hold against the operator. This is the same shape Certificate Transparency gives the WebPKI, applied to the CA’s own operational claims about itself.
Failure modes are bounded by design. Certificates are seven days. ACME Renewal Information lets the CA shorten renewal windows targeted at specific machines or specific profiles, and goodenroll polls for those signals on its own schedule. The fleet rotates without an emergency window and without anyone touching a machine. The exposure window becomes a configuration choice rather than a function of certificate lifetime, and revocation infrastructure stays out of the critical path of the threat model.
Post-quantum where it counts.ML-DSA-65 (FIPS 204) for certificate signing, ML-KEM-768 (FIPS 203) as the subject key for TLS key-exchange certificates. ARI is what makes the migration tractable on the deployed fleet, because you do not have to wait for natural expiry to do the work.
Nitro is a bounded-trust intermediary. AWS still owns the silicon it runs on. What the architecture buys you is that the property the relying party has to verify is narrow and concrete, and that the actual long-lived secrets are protected by hardware that AWS does not own. Against an AWS-internal threat with full physical access and unbounded effort, Law #3 still applies. Against the attacks the architecture is actually defending against (software compromise of the CA pipeline, a rogue admin pulling secrets through the management plane, a tampered build reaching production), the bounded property is exactly the property you need.
The substrate
An architecture like this only works if the underlying primitives are right. Three pieces of infrastructure I have been spending time on are upstream of GoodKey CA.
PeculiarVentures/scp is GlobalPlatform Security Domain key management in Go. The name is not a coincidence. Smart cards and HSMs have been doing security domains in hardware for two decades, with separate keysets, separate trust roots, and isolation between issuer, verifier, and application code on the same chip. The library implements SCP03 and SCP11 and a typed Security Domain management layer for key lifecycle, certificate provisioning, and trust validation, against verified profiles with byte-exact validation against independent reference implementations. This is the unglamorous work of “make sure the keys you are putting on hardware are actually being put on hardware in the way you think they are.” If the key on the device is not where you think it is, every downstream signature is asserting something false.
draft-ietf-acme-device-attest, which I am a co-author on, is the cross-machine extension on the client side. It standardizes how a device proves to an ACME server that the key in a certificate request lives in attested hardware on a specific device. The recent revisions resolved several interoperability gaps that had blocked broad implementation, including the Apple-specific attToBeSigned semantics around sha256(token) versus sha256(keyAuth), an explicit identifier-verification step, the badAttestationStatement error type, and a hardware-module identifier type. The point of the work is to make the client side of the trust chain as verifiable as the CA side. An attested signing service that issues credentials to anyone who asks is not solving the problem, it is moving it.
PeculiarVentures/attestation closes the loop. It is the verifier side that consumes attestation evidence from these various sources (TPMs, YubiKeys, Apple devices, Nitro Enclaves) and reduces it to claims a relying party can act on. Without a verifier, attestation is marketing. With a verifier, it is engineering.
These are not separate efforts. They are what makes hardware-anchored cross-machine trust mean anything in the wild. The transparency-log side of the same problem is being standardized in parallel through transparency.dev, C2SP, and the Merkle Tree Certificates draft, which together extend the same model to issuance auditability at WebPKI scale.
What this asks builders to do
The Dai Zovi prescription is operating-systems hygiene applied to the whole stack. The verifiability corollary is the same hygiene extended across machines you do not own. Both are old. AI is what is making them mandatory.
Pick small. Compartmentalize. Strip privilege to what each component genuinely needs. Make each component’s TCB small enough that one person can characterize it in a sitting. Single-purpose services, deterministic builds, dedicated resources rather than shared microarchitectural state, single-image enclaves rather than orchestrated runtimes.
Make it provable across machines. Per-operation attestation rather than per-boot. Independent re-verification at every hop, not transitive trust. Authorization decisions inside the attested boundary. Evidence bundles the relying party can run a verifier against, with their own trust anchors. Short lifetimes with active rotation rather than long-lived credentials backstopped by revocation. And publish the operations themselves to a transparency log with independent witnesses, so the proofs survive disagreement about who saw what when, and so a single dishonest operator cannot serve different versions of reality to different relying parties.
Anchor trust in hardware whose threat model you can actually evaluate. Where you can put the long-lived secret on hardware the user owns, do that. YubiKey, Apple Secure Enclave, TPM in the laptop on the engineer’s desk, smart card in the operator’s pocket. Where you cannot, use a cloud TEE as a bounded-trust intermediary that produces narrow verifiable claims, and design the architecture so the long-lived material lives in a different boundary that the cloud operator does not own.
And know what your assurance is buying you. Cloud TEEs are not how you defeat the operator. They are how you make narrow operations verifiable to relying parties while accepting that absolute properties against the operator are not on offer. The places where attestation delivers what it advertises are the places where the user owns the silicon. Law #3 has not been repealed, and AI has only raised the cost of pretending otherwise.
Smaller is the easy half. Provable is most of the engineering. On hardware you own is where the property actually holds.
Spend enough time around AI right now and you start to get the feeling that everything is speeding up, all the time.
Every week there is a new model, a new capability, a new claim that some industry is about to be remade. It starts to feel like rapid change is just the new baseline. Like history has bent into a permanently steeper slope.
I do not think that is right.
What I think is closer to the truth is that we have gotten used to confusing motion with progress, and delay with inevitability. Some things are moving very quickly. Others are barely moving at all. We treat the former as inevitable and the latter as unavoidable.
Neither is true.
My father was born in 1942. That is not ancient history. When he was born, there was still a lot of basic infrastructure left to build.
Within a little more than a decade, nonstop transcontinental passenger air service became viable. Less than eight years after that, a human entered space. Eight years later, people were walking on the Moon.
That is a staggering amount of change in a very short period of time.
In one person’s early life, we went from making coast-to-coast air travel practical to landing human beings on another celestial body. Not as a thought experiment. Not as a roadmap. We just did it.
And it was not only aerospace. The Golden Gate Bridge was built in about four years. The first transcontinental railroad was completed in about six. These were massive physical undertakings that reshaped how people moved and how economies functioned, delivered on timelines that would feel almost implausible now.
The easy way to dismiss this is to say that software is fast and physical infrastructure is slow. That if AI looks fast and transit looks slow, that is just how the world works.
But that does not really hold up.
Ukraine did not build its drone ecosystem on leisurely timelines. Tesla compressed what many assumed would be a slow industrial transition into something the rest of the auto industry had to react to. When something actually matters, physical systems move. Supply chains get reorganized. Tradeoffs get made. Bureaucracies get bent. Talent concentrates. People stop explaining why something is hard and start figuring out how to get it done.
That is part of what makes Artemis interesting.
This is not a criticism of Artemis. It is an ambitious and serious effort. But it is also a reminder that progress is not self-sustaining. Apollo is often remembered as a triumph of technology, but it was just as much a triumph of focus, alignment, and urgency. Artemis reminds us that those things matter just as much as the rockets do.
There is another force that shows up in systems like this.
At Google, there was a name for it: slime mold.
It is what happens when layers of process, approvals, coordination costs, and local incentives build up over time until forward motion gets harder even when nobody involved is being unreasonable. Everything makes sense on its own. The system just moves more slowly.
Technology policy has its own versions of slime mold.
We saw it in the crypto wars, when policymakers convinced themselves that math could be slowed down with policy, as if cryptographic reality were open to negotiation. It was not. What that produced was not real control. It produced friction, workarounds, and the illusion of governance.
You can see the same instinct showing up again in parts of the conversation around AI. When institutions feel outpaced, they respond with process. That instinct is understandable, but it rarely solves the problem. You do not make systems safer by pretending inevitabilities are optional. You make them safer by building the infrastructure, incentives, and accountability needed to deal with what is actually happening.
But that is not how we tend to think about progress.
We talk about technological achievement as if it were mostly about invention, as if once something has been demonstrated it remains latent in society, ready to be called back into service whenever we need it.
That is not how any of this works.
The ability to do ambitious things quickly depends on organizational memory, industrial capacity, political alignment, tolerance for risk, and a culture that still expects big things to happen on human timescales.
Lose enough of that, and even getting back to where you once were becomes hard.
You can see it in infrastructure. Projects that once would have been treated as urgent now take decades, often in fragments so small that earlier generations would have treated them as preliminary milestones. Over time, that changes expectations. Slowness starts to look like responsibility. Ambition starts to sound naive.
That is the trap.
The problem is not just that progress slows. It is that people get used to it. What would once have looked like drift starts to look like process. What would once have sounded like an excuse starts to sound like maturity.
Meanwhile, in domains where urgency and incentives line up, things still move very quickly. ChatGPT was released publicly in late 2022. In a few years, AI went from something most people associated with research labs to something embedded in everyday workflows, products, and policy debates.
AI did not prove that everything is accelerating.
It proved that when enough capability, capital, and attention line up, rapid change is still possible.
That is the point.
The world is not uniformly speeding up. Some parts of it are. Others are not. And the difference has less to do with atoms versus bits than with whether we have decided something actually matters.
That ought to make us a little less complacent.
People like to tell themselves that once a technology is important enough, the rest somehow sorts itself out. The problems get solved. The risks get managed. The surrounding systems catch up.
History does not really support that.
Things were only all right in the past because people worked very hard to make them all right. The systems that made aviation safe, that made infrastructure dependable, that made computing usable in high-trust environments, none of that appeared on its own.
The same will be true here.
If we want AI to be safe, trustworthy, and broadly useful, that will not happen as a side effect of capability gains. Security will not emerge on its own. Governance will not emerge on its own. The infrastructure needed to make these systems worthy of dependence will not emerge on its own.
Those things only happen when people decide they matter.
That is the real problem with the idea that everything is accelerating. It makes it easy to believe that progress takes care of itself.
It does not.
Progress happens when people decide it needs to, and then do the work.
