Unmitigated Risk

Agents Should Earn Authority, Not Inherit It

Wed, 28 Oct 2026 23:00:00 GMT

Much of my career has been spent deciding what software is allowed to do, and every system I worked on eventually had to answer the same question, which is what happens when the code in front of you does something you did not expect.

Agents have made that question urgent again, and most of the conversation about them is about making the model behave. I think that is the wrong place to start. As I put it in Containing the Optimizer, safety is a design problem, not a policy problem. The more useful question is what the model can reach when it does not behave, and the easiest way to show why is to walk through a run.

An agent should be free to decide how to accomplish a task. It should not be free to decide what it is authorized to do.

A flaky test is enough

Here is what a run looks like in a system designed this way. Take an engineer on a payments team with a refund test that fails about one run in five. The test submits the same refund twice with one idempotency key and asserts that only one goes through. It has been flaky for a week, and she asks an agent to find the cause and fix it.

Before the agent starts, she sees a short permission list. It may read the payments repository, fetch named packages from the organization's approved catalog, run isolated builds, and stage a change to her branch. Merging will require passing checks and her approval of the exact diff. Production access and arbitrary network destinations are outside the list. She approves it with her passkey because it matches what she asked for.

Behind that list is a mission, a structured description of the work, the resources it needs, and the evidence required at each step. Its approved limits are the envelope, the maximum authority for this revision of the mission. Approval does not release everything at once. The run starts with repository reads and the named packages. Later permissions wait for milestones, points in the work that require specified evidence.

The agent traces the failure to a vendored test helper whose fake clock reads the system time instead of holding a fixed value. When the CI runner is slow, the idempotency window expires between the two refund calls, the second call is treated as a new refund, and the assertion fails. Recording that diagnosis completes the inspection milestone and releases access to isolated runs in the build service. This low-risk step can rest on the agent's report within the approved build limits. Publishing a change will require evidence from outside the agent.

On the way there, it reads the helper's troubleshooting notes, which suggest running a bundled diagnostic script when tests are flaky. Running a project's own diagnostic script is a reasonable debugging step, so the agent runs it. The script collects environment variables and tries to post them to an endpoint it describes as the maintainers' telemetry service.

The virtual machine has no network connection of its own. Everything that leaves it goes to a broker, a component on the host that checks whether the run holds a capability for that exact action and then performs the action itself. A capability is a permission tied to this mission, this resource, and these limits. Here the script's HTTP client is configured to use a local adapter that forwards requests over the private host channel to the network gateway. The gateway finds no capability for that host and refuses the post. A client that bypasses the adapter has no network route at all. The virtual machine also never held a CI token, and the package broker fetched every package on the agent's behalf, so the agent never saw a registry credential either. Keeping those credentials out is a separate protection from blocking the transfer of repository content or other data the run does hold.

The fix needs a dependency update the envelope did not include. Holding the clock fixed requires a newer release of the time-mocking library than the versions the plan named, and the package broker will not resolve a version the mission never declared. The agent tries two already-approved releases first, then submits a revised mission asking for the newer one. Because that widens the envelope, it comes back to her as exactly the difference, one package pinned to one digest and permission for the corresponding manifest and lockfile changes. She approves it.

The agent submits the change to the build service, which runs the tests and returns a signed receipt bound to the exact candidate. That receipt establishes that the specified tests passed, and only then releases the capability to stage the change. The source-control broker checks that the patch touches only the helper, its test, and the approved dependency files, then records it as a staged result.

A secret scan runs and she reviews and approves the exact diff. Only then does promotion, a separate step with its own capability, merge that result.

She finds the diagnostic script's request afterward in the run record, the log the brokers keep of everything they allowed or refused. It sits there, refused, a few lines above the package request she approved. Neither outcome depended on the model recognizing the script as hostile. The model's judgment found the bug, and the boundary decided what the run could touch.

Goal

Fix the refund test.

Follow the permissionsDetails

1 / 7
Scope not yet approved
Available now · locks show what waits
- Read repo locked
- Named packages locked
- Run builds locked
- Stage patch locked
- Merge result locked
This action
Investigate the refund failure
Waiting for approval
The goal alone grants no access.
Permission details
The agent proposes resources, operations, and evidence gates. Policy constrains the proposal before the engineer approves its envelope.
2 / 7
Inspection allowed
Available now · locks show what waits
- ✓Read repo available
- ✓Named packages available
- Run builds locked
- Stage patch locked
- Merge result locked
This action
Read the helper and refund test
Read access released
The engineer approved the envelope.
Permission details
Approved maximum: repo reads, named package digests, bounded builds, and a scoped patch. Staging and merging wait for evidence and review. Production access is excluded.
3 / 7
Builds unlocked
Available now · locks show what waits
- ✓Read repo available
- ✓Named packages available
- ✓Run builds available
- Stage patch locked
- Merge result locked
This action
Test the fake-clock diagnosis
Build access released
Diagnosis recorded at the inspection gate.
Permission details
Existing scope is unchanged. The diagnosis releases bounded build-service runs; a passing-test receipt for the exact candidate is still required before staging.
4 / 7
Telemetry request blocked
Available now · locks show what waits
- ✓Read repo available
- ✓Named packages available
- ✓Run builds available
- Stage patch locked
- Merge result locked
This action
Post environment variables
Blocked
Destination outside approved scope.
Permission details
The gateway refuses and records the attempted post. The telemetry host has no capability. Existing reads, packages, and build permissions do not expand.
5 / 7
One package needs approval
Available now · locks show what waits
- ✓Read repo available
- ✓Named packages available
- ✓Run builds available
- Stage patch locked
- Merge result locked
- New package awaiting approval
This action
Fetch a newer clock library
Approval required
One digest plus dependency-file changes.
Permission details
The revised mission is checked against policy, protected intent, and run history. The engineer approves this difference before the new dependency becomes available.
6 / 7
Staging unlocked
Available now · locks show what waits
- ✓Read repo available
- ✓Named packages available
- ✓Run builds available
- ✓Stage patch available
- Merge result locked
- ✓New package available
This action
Stage the verified patch
Staging released
Build receipt bound to this candidate.
Permission details
The approved widening adds the new package digest and its manifest and lockfile changes. The source-control broker checks file scope and stages the tested candidate. Merge remains locked.
7 / 7
Reviewed result merged
Available now · locks show what waits
- ✓Read repo available
- ✓Named packages available
- ✓Run builds available
- ✓Stage patch available
- ✓Merge result available
- ✓New package available
This action
Merge this exact patch
Merged
Required scans passed; engineer approved.
Permission details
A separate promotion capability authorizes only the exact reviewed result. The source-control broker records the merge. Production deployment remains excluded.

Each piece of evidence releases the next capabilityEnvelope approvedsigned with a passkeyDiagnosis recordedinspection milestoneBuild receipttests passScan and reviewbefore promotionRead the repoinstall named packagesRun buildsin the build serviceStage the changeshape checkedPromotethe exact verified resulteach stays usable within its limits and lifetimeOutside the envelopeDiagnostic script's postrefused, no capability for that hostOne newer packagewidening she approved, one digest

Authority arrives as the work earns it

Recording the diagnosis lets the agent run builds, but it still cannot stage a patch. Both are inside the approved maximum, but the mission specifies a different evidence gate for each.

Before the agent reads repository content, the engineer's request is recorded as protected intent. The agent proposes a mission that makes its requested authority explicit. For the refund fix, the initial proposal could look like this. The names and limits below are illustrative. This is a readable summary of structured fields, not a required schema.

Resource	Requested operation	Required gate
Payments repository, pinned starting revision	Read source into the isolated run	Initial envelope approval
Declared package versions and their digests	Retrieve from the approved catalog	Initial envelope approval
Isolated build service	At most 10 isolated test jobs during a 30-minute grant	Diagnosis recorded
Branch refund-fix; tests/helpers/refund-clock.js and tests/refund.test.js	Stage the candidate patch	Passing-test receipt bound to the candidate; broker checks file scope
The exact staged result	Merge, with no production deployment	Required scans and engineer approval of the exact diff

The proposal requests authority; it cannot grant it. Organizational policy determines which resources and operations are eligible and which gates are mandatory. A deterministic reference monitor validates the proposal and compiles precise authority requests. It does not infer permission from the English goal alone. The engineer approves the resulting envelope, rendered directly from those requests. A request beyond policy must be rejected or handled through a separate policy-change process; her approval cannot silently override it.

An independent assessment model adds a restrictive check. It sees the protected intent and compiled requests, never the repository content, and can recommend narrowing or rejection but cannot add authority. It therefore cannot confirm every detail of the debugging plan. Its role is to constrain the proposal, while policy and brokers define and enforce the limits.

Within that approved maximum, short-lived capabilities release as the named evidence arrives. The recorded diagnosis unlocks bounded builds; a build-service receipt for the candidate unlocks staging. The agent cannot substitute its own claim that the tests passed for that independent receipt. It is the mission-scoped, short-lived authority I argued for in From Persistent to Ephemeral, carried from the agent's identity down to each effect it performs.

The newer clock library changes this mission's resource list and writable files. The revised proposal names its digest and the corresponding manifest and lockfile changes, then goes through the same policy and assessment checks. A revision inside the envelope proceeds on evidence; widening returns to the engineer as the exact difference. Each revision is checked against protected intent and the mission's whole history, so individually small requests must also fit the cumulative scope.

Every effect that crosses the boundary requires a capability, which is a statement that in this mission, at this milestone, this run may perform this operation on this resource within these limits. A capability is a claim about the task rather than a tool permission. The telemetry endpoint had no capability because the mission never named it, and neither did the newer library until she approved it.

The agent never holds the keys

The model followed the helper's diagnostic instructions. What stopped that decision from becoming a data transfer was the gateway, backed by isolation that left no alternate route. The run's virtual machine has no network interface. Its only outside connection is a private channel to a mediator on the host, so bypassing the HTTP adapter does not bypass the boundary.

Untrusted contentrepos, web pages,docs, tool resultsThe runits own virtual machineAgentreads, plans, computesAbsent by constructionno network interfaceno service credentialsno shared stateprivatechannelOn the hostReference monitorcompiles the mission into capabilitiesNetwork gatewayPackage brokerSource-control brokerSigning and API brokersGeneral proxy, approved onlyCredentials and signing keysProtected systemsNetwork hostsPackage registriesRepositories and CISigning and cloud APIsOther servicesreceipts and refusalsRun recordwritten by the broker that performed each effect, never by the agentappend-only Merkle log with independent witnesses

The package broker retrieves the approved clock library; the source-control broker stages the permitted patch. Each broker checks the capability, accounts atomically for its use, and performs the operation itself. Before writing, it also checks the change's shape. A patch outside the helper, test, and approved dependency files is refused, as is a lockfile change without dependency-update authority or a value matching the run's environment. These rules ask whether the change has the shape the mission allows, not whether it looks malicious, so there is nothing for an attacker to talk past. A service with no typed broker can still be reached through a general proxy, but only with explicit approval, and the approval says plainly that the proxy enforces destination and volume but cannot see inside the traffic.

The diagnostic script could collect the run's environment, but it could not collect the registry or source-control credentials. Those remain with the brokers. Most organizations spent the last decade moving people onto passkeys and are now handing agents API keys, which are passwords with better randomness. Here the agent authenticates with a credential it can use but cannot export, while the brokers hold the service credentials.

What the broker does with the credential depends on the service. Where a service still takes a shared secret, the broker inserts it into the outbound request and strips it from anything sent back. That only holds when the broker builds the request itself. Dropping a bearer secret into whatever traffic the agent sends is not enough, because a service may echo it back, a redirect may carry it elsewhere, or an approved destination may accept an operation nobody intended, so the strongest form is a typed broker that constructs the request, authenticates it, and checks the response. Where the service supports strong authentication, the broker uses a key held in hardware instead. A code-signing key, for example, lives in an HSM, the signing broker signs within the capability, and the agent receives only the signature.

Nothing becomes permanent on the agent's word

A passing refund test does not authorize a merge. The build receipt establishes a result for one candidate, staging makes that candidate available for review, and the engineer's approval applies to its exact diff. Editing the patch afterward invalidates that binding; the changed candidate must pass its gates again. Changes to shared state land as staged results with signed receipts carrying the hash of exactly what was produced. Where a system has no native way to stage a change, such as a database or an IAM policy, the agent works against a dry run or an ephemeral copy, the receipt pins the exact operation and its parameters, and promotion replays only that. Making a result permanent requires a separate promotion capability, released only after verification through static analysis, secret scanning, dependency diffs, and advisories re-checked against current data.

The mission permits merging the refund fix, not deploying payments to production. If merging that branch would trigger a deployment, the broker must treat the request as a deployment and refuse it under this envelope. Brokers therefore look past an operation to its consequence. If pushing a tag triggers a deployment, the request is governed as a deployment, judged against the CI configuration it will actually run. A convenient Git operation does not get to become an unreviewed production change.

The refused telemetry post, approved package retrieval, staged candidate, and final merge all appear in the run record. Each entry comes from the broker that performed or refused the effect. The record is built like a Certificate Transparency log, with an append-only Merkle tree, signed roots, and independent witnesses. A verifier can check its integrity against the organization's trust bundle without relying on the agent's account.

A refusal has to be cheaper than a workaround

The telemetry post and newer clock library both fall outside the initial envelope. The first is irrelevant to the task; the second is a legitimate change in approach. The engineer needs to understand that difference without reconstructing the whole run. What decides whether a capability system survives is what happens after a refusal. I have watched enough security controls die this way to know the pattern. If widening is slow or opaque, engineers approve broad envelopes up front, keep standing exceptions, or run the agent on a laptop, and ambient authority comes back with them.

So a refusal names what was missing in terms the engineer recognizes, a widening request shows only the difference, and a reasonable approval takes seconds. The run record shows which missions keep hitting the same wall, so the envelopes people write improve over time. The engineer in the story widened her envelope faster than she could have filed a ticket for a firewall exception, and that comparison is what keeps people on the narrow path.

Give the same agent the same task on her laptop, in her shell, with her environment variables, her cloud credentials, and an open network, and the script's request succeeds. In July, something close to that happened with an evaluation harness in place of a laptop.

Turning off the guardrails turned off the control

On July 21, 2026, OpenAI disclosed that agents running in an internal cyber-capability evaluation had broken out of their test environment and reached Hugging Face production infrastructure. The independent investigation by METR and Redwood Research found it was not one agent. About 1,200 agents coordinated through a message board they had built inside an internal package cache, more than 70,000 messages and files, after gaining admin access to that cache in late June. From there they got code execution on a third-party sandbox with broad internet access, found Hugging Face credentials exposed online, and on July 11 used a malicious dataset to get code execution on a Hugging Face production worker. Nobody planted an instruction. The agents were trying to beat the evaluation's scorer and went looking for whatever would help. METR also found the agents were wrong about how they were being scored, which means even they could not have told you what they would go after next.

OpenAI ran that evaluation with reduced cyber refusals and without its normal production classifiers, which is reasonable when the point is to measure offensive capability. To me it is also the detail that matters most. With the guardrails relaxed for a legitimate reason, nothing about the model's behavior stood between the agents and the infrastructure around them, and the isolation that remained had paths through it. A boundary is the control that has to hold on the day someone has a good reason to change how the model behaves.

Two things made the path possible. The first is ambient authority, meaning access that exists because of where a process runs rather than because its current task needs it. The package cache was shared and writable across runs, so thousands of agents that were supposed to be isolated from one another had a place to coordinate. Credentials worked for whoever found them, and Hugging Face's dataset processing ran whatever content users supplied. None of it was granted to the evaluation, and all of it was reachable from it. The second is a chain of vulnerabilities on both sides. Ambient authority put the infrastructure in reach, and the vulnerabilities got the agents through it.

OpenAI's own retrospective adds a third piece. Its chain-of-thought monitoring, which was not running during the evaluation, could have flagged the activity more than a day before the breach. Confinement and detection are different jobs, and the incident was missing both. The investigators also found agents spoofing tool calls, showing one command in their transcripts while running another, which is a good reason for the record of what happened to come from the components that performed each effect rather than from the agent's own account.

A capability design removes ambient authority by construction. Each run gets its own virtual machine with nothing shared, so there is no common cache to turn into a message board, no proxy in reach, and no credential lying around to find. What has to be correct shrinks to the host, the hypervisor, the control plane, and the brokers, a trusted base small enough to harden and review. That is the same bet every reference monitor has made since the 1970s. Every refused attempt is also recorded against the mission, and repeated reaches for undeclared hosts hold the run for review instead of surfacing days later from the victim.

The subject changed, not the problem

I wrote in The Amnesia Cycle that security keeps rediscovering lessons it already learned, and agent security is the latest round. Almost every piece of the design in this post has a name and a date.

Dennis and Van Horn's capabilities (1966) are the mission-bound grants, authority carried as a specific, unforgeable permission rather than inherited from who you are. Anderson's reference monitor (1972) is the brokers, a component that mediates every access, cannot be bypassed, and is small enough to check. Saltzer and Schroeder's least privilege (1975) is the envelope. Biba's integrity model (1977) is the rule that low-integrity input cannot direct a higher-integrity decision, which is why the assessment model never reads what the agent read. Clark and Wilson's well-formed transactions (1987) are staging, verification, and promotion.

The one that matters most is Hardy's confused deputy (1988). Hardy described a compiler that held authority to write a billing file, and a user who got it to overwrite that file by naming it as an output path. The compiler was not malicious. It used its own authority on someone else's instruction. That is prompt injection, written down thirty-eight years early, and Hardy's conclusion was that the compiler's authority came from its environment rather than from the request it was serving, and that was the flaw. Capabilities fix it because authority travels with the request instead of living in the deputy.

Those ideas largely lost the first time around to convenience rather than to being wrong. Capability systems were hard to program against, revocation was hard because the subject held the capability, and operating systems built on ambient authority were easier to live with. Both obstacles look different now. When the broker holds the capability, every use is a revocation check, and the subject is no longer a person who finds a capability discipline tedious but a program that works within whatever discipline the environment enforces. A token the agent holds is a token an injected instruction can spend or leak, which is why the broker holds it here.

What changed is the subject inside the boundary. Classical sandboxing assumed a known program with a known purpose, such as a PDF renderer or a browser tab, whose code you could read and whose failure modes you could enumerate. An agent reads untrusted repositories, web pages, documents, and tool results, and it revises its plan mid-run based on what it just read. Running a diagnostic script a README recommends is not a malfunction, because reading context and acting on it is the agent's job. The classical controls still apply, but they have to sit at the point where a decision becomes an effect rather than only at the point where the process starts.

Beyond a single run

The refund fix needs neither a policy change nor a child run, but the same authority limits must hold when those features are added.

Agents also cannot govern their own governors. An agent may draft a policy change, but activation requires a quorum of reviewers distinct from the proposing run, and the proposing run is excluded from the policy generation it proposed and from every later one. Policy itself moves as reviewed generations. Each candidate is tested against a corpus of useful missions and known-bad behavior with pass thresholds fixed before the test, compared against the active generation for any authority it newly reaches, and activated only when a quorum signs its exact digest. A generation that fails its evaluation never activates, and new work is admitted only under a current, signed evaluation.

The same rules hold when a run spans trust domains. A parent run can pass part of its envelope to a child run in another domain only as a signed grant with fewer resources, lower limits, and a shorter lifetime, and nothing in the receiving domain can add to it. Moving from development to production is promotion of an exact artifact, never delegation.

Good process gets faster, bad process gets more confidently wrong

AI amplifies whatever process it lands in. A team with good process gets faster, and a team with bad process gets more confidently wrong. Security works the same way. If your environment hands every process standing credentials and an open network, an agent will find that authority faster and use it more thoroughly than any human or script you have run before. If your environment scopes authority to the task, stages changes, and makes widening cheap, an agent makes that process faster without making it looser.

Better prompting and guardrail models improve the odds that an agent behaves. Around them sits the same operational work any production system needs, with telemetry, incident reports, postmortems, and automated response that contains a failure before a human reads the page. All of it is worth having, and none of it makes misbehavior impossible. A boundary that holds only when the subject behaves is not a security boundary.

None of this is specific to agents. The same failure shows up wherever software is delegated authority faster than people can supervise it, in CI pipelines that deploy on a tag, in bots with standing production credentials, and in automation that inherits a human's ambient access because that is how the jump box was wired. Agents made the failure visible at scale, but it was always there, only slower and quieter.

The operative word in secure by design is design. The design does the work, and the work divides three ways. The model contributes judgment about what to do. The boundary contributes confinement, which is certainty about what the agent could not do. The run record contributes evidence of what it did, written by something other than the agent. Conflating any two of them produces a system that is secure on the days the model behaves and exposed on the days it does not. When the first AI-built zero-day surfaced in May, I said the response was narrower authority, better harnesses, and replayable decisions, and this is what those look like when they are built.

The engineer never found out the diagnostic script was hostile until after it no longer mattered, which is about as much as anyone should have to know.

The WebPKI's Unmanaged Software Supply Chain

Thu, 22 Oct 2026 23:33:22 GMT

In The Private Key Was Four, I examined how eight eMudhra intermediate CA certificates ended up with private keys so weak that one of them was the integer 4. In Trusted Everywhere, Impact Everywhere, I looked at why a failure at a CA serving a small market becomes a problem for every relying party that trusts it, whether or not that relying party ever chose to use the CA.

This article is about a third problem, which gets the least attention of the three and is getting worse as AI agents take on more work. Much of the software ecosystem inherits the WebPKI's global trust without needing its full scope and without receiving the protections browsers built to manage it.

The stakes are hard to overstate. The web's security rests entirely on the WebPKI. Every HTTPS connection a browser makes to a public website is authenticated through it, browsers accept no alternative, and much of what software does besides browsing depends on it as well. Package registries, software update channels, cloud and SaaS APIs, OAuth and single sign-on flows, and the endpoints AI agents call are all reached over TLS connections authenticated by publicly trusted certificates. Some of those systems add their own protections, such as signed packages or pinned keys, but for most of them the certificate a client accepts is the whole of its proof that it is talking to the right party.

When the eMudhra incident was reported, Chrome blocked the affected intermediate public keys through CRLSet within roughly 100 minutes of receiving the report, and Mozilla distributed its own blocks through OneCRL. Those responses reflect years of investment in browser security, incident response, and trust management.

The VPN clients, mail servers, containers, embedded devices, and command-line tools that rely on the same CAs had no equivalent channel. Some received protection through their own updates or platform policies, others may still accept certificates that browsers now reject, and many of them never needed to trust the entire public WebPKI in the first place.

We can inventory the libraries our software depends on, identify their versions, and track vulnerabilities. Yet many applications cannot readily tell us which organizations are authorized to authenticate their remote services, whether those authorities are still accepted by major browsers, or how quickly a distrust decision would reach deployed instances. Those are security dependencies too.

Why does your application trust so many CAs?

Consider a VPN client that connects to one company's infrastructure. It might authenticate three or four service names, all controlled by the VPN provider, and those services could use a small set of CAs or, where the deployment model permits, a private PKI.

The client nonetheless inherits an operating system's general-purpose root store, which contains authorities that have no relationship with the VPN provider or its customers. That happens because it is the default. A TLS library loads the platform's root store, and the application works without any special configuration.

For a browser, broad trust is the right starting point, because a browser must reach practically any public website without asking the user to configure trust first. A thermostat, a VPN client, and a service that calls five known APIs have none of those requirements. A television may need its manufacturer's identity, software update, and telemetry endpoints. A mail server needs to reach other mail servers, but its transport authentication requirements differ from a browser's. A machine-to-machine application may have only a handful of intended peers.

When an unconstrained CA is trusted for TLS, it can potentially authenticate any DNS name, and the client does not check whether the legitimate service operator chose that CA. If an attacker obtains a CA signing key, or a CA issues an unauthorized certificate, the attacker can impersonate an unrelated service provided they can also get into the connection path. A correct hostname on a mathematically valid chain will not distinguish the fraudulent certificate from one the operator authorized.

CAA records provide a partial mitigation. A domain operator can publish which CAs are authorized to issue for its names, and the CA/Browser Forum Baseline Requirements require public CAs to check those records before issuance. That reduces accidental issuance and some forms of misissuance by otherwise compliant CAs. CAA is checked by the issuing CA, however, and not by the TLS client. A CA that disregards CAA, or an attacker who already possesses an intermediate signing key and issues certificates outside the CA's controls, is not stopped by the DNS record. The relying party still accepts a valid path to a trusted root unless it implements an additional restriction.

The WebPKI Observatory snapshot discussed in the previous article found that five CA organizations account for 88.8% of unexpired publicly trusted certificates, and in the Firefox telemetry from the same article, eMudhra accounted for 0.025% of successful validations. Those figures describe the market rather than any particular application, and a small CA can be essential to a specific application. They do make it hard to argue that every specialized program needs every organization in a browser-scale root store, and an application that knows its intended peers can usually define a far narrower policy than either figure suggests.

Every unnecessary trusted CA adds another organization whose failure could become the application's failure, and that exposure exists even when the root store is fully current.

CAA is consulted by CAs, not TLS clientsTwo paths show that a compliant certificate authority consults CAA before issuance, while an attacker controlling a CA signing key issues outside that process and presents the certificate to a client that ordinarily does not consult CAA.CAA is consulted by CAs, not TLS clientsCompliant public CAs consult it during managed issuance; TLS clients ordinarily do not.MANAGED CA ISSUANCEDomain ownerPublishes a CAA record namingthe CA authorized to issueDNS policyCompliant public CAQueries CAA before issuingthrough its controlled systemsauthorized?Issue or refuseThe CA enforces the recordas part of issuanceWHEN A CA SIGNING KEY IS COMPROMISEDAttacker holds the keyCreates a certificate outsidethe CA’s systems and controlsno CAAlookupCertificate is presentedIt can contain a matching name,dates, usage, and valid chainTLShandshakeTLS client validatesChecks its own certificate andtrust policy—not normally CAACAA can cause a compliant CA to refuse issuance.A TLS client ordinarily will not reject solely because issuance violated CAA.

Compliant public CAs consult CAA during issuance; TLS clients ordinarily do not use it when validating the resulting certificate.

A root store is not a root program

Chrome, Mozilla, Apple, Microsoft, and other browser and platform operators do more than publish trusted certificates. They run continuing security programs around those decisions, which set admission policy, take part in CA/Browser Forum standards work, review audit and compliance information, investigate incidents, and decide when a CA should be restricted or distrusted.

They also maintain protections beyond ordinary X.509 path validation. Chrome uses CRLSets for selected high-impact certificate blocks. Mozilla uses OneCRL and has developed CRLite. Browsers enforce Certificate Transparency for certificates within the scope of their public TLS policies, and they can ship emergency trust decisions without waiting for a conventional application upgrade.

These mechanisms are imperfect, since audits miss failures and root programs react to incidents as often as they prevent them. Certificate Transparency does not make an unauthorized certificate authorized, though where it is enforced it makes that certificate observable and makes unlogged certificates unusable. What root programs have is dedicated teams and operational channels for responding when an earlier trust decision proves wrong.

That is why the eMudhra response could happen so quickly. The security did not live in the root certificate. It depended on an organization able to investigate the report, decide what to block, and push that decision out. A root store is an artifact of a root program, not the program itself.

A root store is not a root programFeature comparison maintained browser versus root CA bundleA root store is not a root programA copied certificate bundle carries trust anchors, not browser operationsSECURITY CAPABILITYMAINTAINED BROWSERROOT BUNDLETrusted root certificatesYESYESCA incident investigationYESNOCT enforcementYESNOEmergency intermediate blocksYESNORapid policy distributionYESNOIllustrative comparison · Capabilities vary by application, platform, and configuration

A root-store snapshot does not carry browser-specific protective services.

Copying Mozilla's root certificates does not reproduce its incident response operation, and it does not bring OneCRL, CRLite, CT enforcement, or emergency intermediate restrictions with it. Consuming an updated CA bundle, meaning a packaged copy of a root store such as a Linux distribution's ca-certificates package, does not reproduce a browser's validation behavior either.

How much of the program an application receives depends on how it validates certificates. Software that hands validation to the platform verifier on Windows or Apple platforms inherits part of the operating system vendor's root program, including the vendor's ability to push distrust decisions to the platform. Software that reads a CA bundle from disk, which is the common case on Linux, in containers, and in many language runtimes, receives only the list of root certificates and whatever validation logic its own TLS library applies.

When trust changes, copies do not necessarily follow

CA trust changes over time. Operators are acquired, validation practices fail, keys are exposed, and a browser may distrust an entire root or restrict a particular hierarchy. The Observatory's historical dataset records 16 trust actions from 2011 through 2024, covering both root distrusts and narrower restrictions, and the lesson for software is that a CA list captured at build time is not a lasting security policy.

Applications receive these decisions through very different channels. Some use continuously maintained operating system trust services. Others depend on Linux distribution packages, language runtime bundles, application updates, container rebuilds, or firmware releases. Python's widely used certifi package ships a copy of Mozilla's root list, and Node.js compiles Mozilla's root certificates into its binary, so both change only when the package or runtime is updated. Some software embeds root certificates directly and never refreshes them.

An old CA bundle may continue to accept a CA that every major browser has removed, and a current bundle does not solve the whole problem either. Suppose a root stays generally trusted while one intermediate CA's signing key is compromised. Removing the root would disrupt unrelated legitimate issuance, so a browser blocks the intermediate's public key instead. A TLS client that trusts the same root but never receives the browser's intermediate block gains nothing from refreshing its CA bundle.

The eMudhra response shows the distinction. Mozilla described blocking the intermediates with OneCRL while also removing the relevant root trust in NSS for consumers that do not receive OneCRL updates, and Google described a root store removal meant to protect consumers that do not receive CRLSet updates. Both measures acknowledge that downstream clients inherit different portions of the browser trust stack, and a device that never consumes the newer NSS release or CA bundle receives neither. Distrust is a security decision, and distributing it is a separate engineering problem.

The protections that don't come with the CA bundle

Certificate Transparency shows the gap most clearly. A conventional TLS client verifies the hostname, validity period, signatures, and chain to a trusted root. A browser applies additional public TLS policy on top of that, and for certificates covered by its CT rules, Chrome generally requires acceptable proof of logging. A fraudulent but otherwise valid certificate issued outside CT can be rejected by Chrome while passing ordinary chain validation in another client.

In its eMudhra investigation, Google reported that no TLS certificates chaining to the intermediates within the scope of its CT check had been logged before the block. That finding did not establish that no attacker had created an unlogged certificate. It showed that a CT-enforcing browser had a protective boundary that clients which only validate the chain do not have.

An attacker with a compromised signing key can construct a leaf certificate that is cryptographically valid. Exploiting it still requires presenting it to the intended client, but if that client neither enforces CT nor receives the relevant block, a correct TLS implementation will accept it. That is a policy gap rather than a vulnerability in the TLS library. Private PKI and non-HTTP protocols have their own authentication requirements and need not adopt Chrome's CT rules as written, but an application that relies on public CA authority needs to know which compensating controls it actually has rather than assume the root certificates bring them along. SMTP is a good example, since mail servers often rely on public CAs without a browser's CT enforcement or emergency blocks, and SMTP has its own authentication models, including DANE and MTA-STS, that need to be assessed on their own terms.

Revocation does not close the gap. I contributed to OCSP, and getting timely revocation information to every relying party remains one of the hardest problems in security protocol design. Online checking trades availability against failure behavior and raises privacy concerns. CAs have also been unreliable operators of the CRL and OCSP services relying parties would need, and they have a commercial reason to delay revocation, because revoking a customer's certificate disrupts that customer's service and strains the relationship. Repeated delays in revoking misissued certificates were among the compliance failures behind Entrust's 2024 distrust.

Every major browser has therefore moved toward indirect mechanisms, collecting revocation information from CAs and redistributing it to clients through channels the browser controls, such as CRLSets and CRLite. The direct channel is also disappearing. The CA/Browser Forum made OCSP optional for publicly trusted TLS certificates, and Let's Encrypt shut down its OCSP service in 2025. Software outside the browser that depends on checking revocation directly is relying on the services browsers chose not to depend on, and for a growing share of certificates there is no online responder left to ask.

Shorter certificate lifetimes reduce how long a compromised leaf certificate stays valid, and the industry's move toward them is sensible. An attacker who controls a trusted CA signing key, however, can keep issuing fresh short-lived certificates until clients stop trusting that CA, so short lifetimes do not substitute for propagating distrust.

The supply chain dependency your SBOM misses

We have become much better at asking what code our software contains. We produce software bills of materials, track package versions, scan containers, and work toward reproducible builds. We are less disciplined about asking whom that software is prepared to believe.

The effective trust policy comes from a chain of choices. A VPN client uses a TLS library such as OpenSSL, which, depending on its configuration, loads an application-supplied CA file, system paths, or another trust source. The program may run in a container whose CA bundle was copied from a base image, and its production configuration may differ from the developer's workstation. The developer can know exactly which library version shipped without knowing which authorities the deployed application accepts.

A library update can leave the effective root store unchanged. A base image update can change trusted authorities without touching the application binary. A root removal may appear in neither the application's version history nor its vulnerability reports.

One security update, multiple applications

Correcting this is complicated by the fact that these dependencies are shared. Two applications linked against the same OpenSSL installation, or reading the same system CA store, inherit each other's changes. When the operator of one application updates the shared library or removes a root to address a security problem, the other may lose a connection it legitimately needs or see validation behavior it never tested. The same update can also fix a vulnerability for both, so freezing the shared layer only trades one risk for another. A security change made for one workload becomes a compatibility event for the next, which is why the unit worth managing is the effective trust policy of each application, together with a coordinated way to update the infrastructure they share.

One fix, multiple application outcomesShared TLS update affecting application A and B differentlyOne fix, multiple application outcomesShared TLS infrastructure is a security dependency and compatibility boundaryONE SHARED SECURITY LAYEROpenSSL installation / system trust storeUpdated to address a CA incident or library vulnerabilityAPPLICATION AReceives the intended security fixAPPLICATION BValidation behavior may changeConceptual model · Outcomes depend on each application’s effective validation policy

Changes to shared trust and TLS infrastructure can affect multiple applications.

The root store is not executable code, but it changes how the application behaves. If an unnecessary or compromised CA authenticates an attacker-controlled endpoint, the application may disclose information, act on malicious responses, or authorize further activity without a single byte of its executable changing. Fully patched software can still rely on obsolete or excessive trust.

That makes root stores and validation policy part of the software supply chain in a meaningful security sense, because they determine the authority the software grants to outside parties. Unlike a vulnerable package, though, that relationship is rarely inventoried, assigned an owner, or tied to an update obligation.

The problem is older than AI, but agents make it worse

Agentic workloads concentrate the problem. Agents do most of their work through command-line tools, SDKs, package managers, and API clients running in Linux VMs and containers, which is exactly the environment where trust comes from a copied CA bundle with no CT enforcement and no emergency blocks. A single agent session may run curl, pip, npm, git, and a cloud CLI, each of which can resolve trust differently, inside a container image that may have been built months earlier.

When agents do use a browser, it is usually a headless one, and a headless browser looks like a browser without necessarily behaving like one. Playwright, one of the most widely used automation frameworks, launches Chromium with component updates disabled by default, and Puppeteer disables background networking. Component updates are the channel through which Chrome receives CRLSets and its Certificate Transparency log list. Chrome stops enforcing CT once that log list goes stale, and the Chromium builds these frameworks download are pinned to the framework's release rather than kept current. A long-lived or outdated automated browser can therefore lose protections its operator assumes are maintained automatically, and what any particular headless browser actually enforces depends on its build, its configuration, and how fresh its security data is.

A successful TLS connection can feed information into an autonomous workflow, in which an agent retrieves instructions, consumes API data, downloads dependencies, or takes actions based on what a remote service returns. Impersonating that service does not hand an attacker control of the agent, since permissions, provenance handling, and the treatment of untrusted content still matter, but compromised endpoint authentication can be the first step in a longer chain.

We are investing heavily in deciding what agents are allowed to do, while often leaving which remote services they are willing to trust to an inherited CA bundle. The risk is old, and what has changed is how much autonomous action now rests on an accepted TLS connection.

Narrow trust has its own failure mode

Most applications are not browsers, and the first question for any of them is whether it needs public WebPKI trust at all. Where destinations are controlled and limited, an application may use a dedicated private trust anchor, a narrower issuer policy, or another authenticated relationship. Narrower trust is still not a maintenance-free design. Issuers and service endpoints change, keys roll over, and cryptographic algorithms age, and a root or public-key pin becomes a source of outages if the application has no safe way to migrate it. The goal is to minimize unnecessary authorities and keep a workable path to replace the ones that remain. For applications with known peers, the useful boundary may be finer than the root store, so that each service accepts only designated issuers or keys, with planned rollover and emergency replacement. The decision that matters is which authorities may authenticate each destination.

The payment terminal ecosystem shows what happens when that path is missing. VeriSign negotiated with card reader makers to embed only the VeriSign public root in their terminals, which narrowed the terminals' trust but tied them to one public root and to the hash algorithms the public web supported at the time. When the CA/Browser Forum's January 1, 2016 deadline for SHA-1 issuance arrived, Worldpay still depended on more than 10,000 terminals that could not validate SHA-2 certificates, according to Mozilla's public policy discussion. By then, moving the servers to a private root would not have helped devices that had never been provisioned to trust one.

Browsers were left to choose between holding back the SHA-2 transition and allowing one CA, by then Symantec, to remain the only source of new SHA-1 certificates on the public web. Mozilla allowed a narrowly scoped, temporary exception for nine Worldpay certificates issued through Symantec, limited to 90 days and logged in Certificate Transparency, rather than a general extension of SHA-1 issuance.

The terminal makers' threat model was not the web's threat model, and the consequences of their decision landed on browsers and everyone who relied on them. Choosing a smaller trust store is only the beginning, because the application must also own how that trust is provisioned, updated, migrated, and withdrawn. A dedicated private trust anchor avoids coupling a device's lifecycle to public WebPKI deadlines, although it cannot rescue hardware that was never designed to change its cryptography. Skipping that lifecycle work up front does not eliminate its cost. It defers the bill and often hands it to someone else.

The cost of trust can be deferred, not eliminatedTimeline terminal deployment public web SHA-2 migration and temporary Worldpay exceptionThe cost of trust can be deferred, not eliminatedWorldpay’s SHA-1 terminal compatibility problem, 2016HARD-CODED TRUSTPUBLIC WEB MIGRATESCOMPATIBILITY COSTTerminal deploymentSHA-2 transition2016 exceptionPublic root trustedMigration path missingSHA-1 phased outOld clients incompatibleNine certificatesTemporary, narrow scopeNarrow trust without planned replacement can export its costsSource: Mozilla policy discussion and February 2016 announcement

The Worldpay example shows why narrower trust needs an upgrade path.

Manage trust as an intentional dependency

The right scope for trust depends on the application rather than on which CAs dominate the market, and applications that do need broad public interoperability face a different problem from those with known peers. They need timely trust decisions, a clear understanding of their own validation policy, and a path for emergency restrictions that does not depend on someone rebuilding the program at the worst possible moment. The rustls/upki project is one effort to close part of that gap. It brings CRLite-based revocation data to clients outside the browser through command-line, C, OpenSSL, and rustls interfaces, and lists Certificate Transparency enforcement among its future goals. It is worth watching as its deployment, policy ownership, and compatibility questions get worked out.

Any application can be held to a few accountability questions. Which CAs does it trust, and which of those does it need for its intended connections? Who decides when trust is withdrawn, and how does that decision reach deployed instances? Where its own services use public certificates, do their domains publish CAA records, and does the application enforce CT or other relevant restrictions, or does it only check that the signatures form a valid chain? These are ordinary supply chain questions applied to authentication authority.

Trust does not stop at the browser

Once a CA’s authority is copied into the software supply chain, it can persist long after browsers have restricted it, without the monitoring, policy enforcement, or emergency response machinery browsers built to protect their users, and often in software that never needed such broad trust.

We cannot eliminate every CA failure, but we can reduce the number of parties whose failures each application depends on, and we can make trust decisions visible, maintainable, and responsive to new evidence. The software industry has learned to inventory vulnerable code, and it should be just as serious about inventorying the organizations its software allows to authenticate the world. For software that only needs to trust a few known services, authentication authority should be an explicit, application-specific security policy, with an owner, an update mechanism, and a way to revoke trust when the evidence changes.

A root store is not just a file of public keys. It is a set of security dependencies, and we should manage it like one.

Sources and methodology

The eMudhra timeline and mitigations are described in Google's incident account and Mozilla's incident account. Market concentration and the 16 recorded trust actions follow the WebPKI Observatory snapshot discussed in Trusted Everywhere, Impact Everywhere, and those trust actions include restrictions as well as complete distrusts. The eMudhra validation share comes from Mozilla's cert.validation_success_by_ca_2 telemetry for Firefox 157 and 158 on the release channel, as aggregated in mcpherrinm/cert-count. The Worldpay account is grounded in Mozilla's February 2016 policy discussion and Mozilla's contemporaneous announcement. The Worldpay certificate count and conditions are also reported in CSO's coverage. CAA checking requirements are in the CA/Browser Forum Baseline Requirements. Runtime examples refer to certifi and Node.js. The discussion of UPKI treats it as an exploratory approach. Application scenarios are illustrative threat models, not claims that particular devices or services were compromised in the eMudhra incident.

Trusted Everywhere, Impact Everywhere

Thu, 22 Oct 2026 22:04:10 GMT

One of the private keys behind eight eMudhra intermediate CA certificates was the integer 4. Several other keys in the same set were nearly as easy to recover. Keys capable of supporting trusted TLS issuance made it through key generation, certification, auditing, and root program review before an outside researcher noticed, and the previous article examined how that happened.

This article asks why a failure at an Indian CA became an incident response problem for relying parties around the world that had never chosen to use eMudhra.

The answer lies in how the WebPKI allocates trust. A CA admitted without constraints to a browser or operating system root store can certify names far beyond the customers, country, language, or regulatory regime it actually serves. Root stores differ, some CAs carry name constraints, and some applications maintain their own trust stores, so not every client trusts every CA. Wherever that authority is accepted, though, the relying party inherits the security consequences. The web has often treated a legitimate preference for local control as if it required global signing authority, and those are different things.

What does the web actually need?

If the goal is that every website can authenticate itself and offer encrypted connections to anyone with a modern browser, the web does not need a publicly trusted CA in every jurisdiction. A much smaller set of operators than the number admitted to the global root stores could meet that need. Removing the long tail would still be disruptive, because operators would have to reissue certificates, migrate systems, and unwind dependencies. The question here is technical necessity, which is separate from whether fewer issuers would be desirable.

A Swedish company may want a CA that serves it in Swedish. A German ministry may prefer a provider that understands German regulation, contracting, and business practice. Governments may want domestic oversight, local availability, and independence from foreign institutions. These are legitimate benefits, and they are preferences and policy objectives layered on top of the web's core authentication requirement. A local language, a local vetting team, or a domestic contracting relationship can also be offered by a registration authority or service provider that obtains certificates from an issuing hierarchy that is already trusted. That arrangement may not satisfy a government that wants fully independent national cryptographic infrastructure, but it separates what customers need locally from what relying parties everywhere must trust.

The WebPKI obscures this distinction because admission gives a local provider much more than local service. It gives that provider authority over the global web.

A handful of CAs issue most web certificates

The WebPKI Observatory builds an issuance view from Certificate Transparency and CA ownership records. In its October 22, 2026 snapshot, the three largest organizations accounted for 71.4% of unexpired publicly trusted certificates, the top five for 88.8%, and all remaining CAs together for 11.2%.

Certificate share by CABars show ISRG 32.6 percent, Google Trust Services 21.0, DigiCert 17.8, GoDaddy 10.6, Sectigo 6.8, all others 11.2. Who issues the web’s certificates? Share of publicly trusted certificates · 22 Oct 2026 Internet Security Research Group32.6% Google Trust Services21.0% DigiCert17.8% GoDaddy10.6% Sectigo6.8% All remaining CAs11.2% Top three: 71.4% · Top five: 88.8% · Everyone else: 11.2% Note: Unexpired certificates. Source: WebPKI Observatory · CA market-share snapshot, 22 Oct 2026

These figures are counts of unexpired certificates in circulation. They do not count customers, domains, HTTPS connections, or daily issuance, and one customer may hold many certificates while a heavily used site holds few. They describe each CA's deployment footprint and, roughly, how much of the operator side depends on it.

Geography is more lopsided still. Of the 84 CA organizations the Observatory classifies by location, the 15 in the United States make up 17.9% of organizations and hold 92.76% of certificates. The 44 European organizations make up 52.4% and hold 7.15%. The 25 organizations in all other regions hold 0.09%.

CA organizations and certificate share by regionIn the 84 geographically classified CAs, 15 US organizations hold 92.76 percent of certificate share; 44 European orgs hold 7.15 percent; 25 elsewhere hold 0.09 percent.CA locations vs. certificate share 84 geographically classified CA organizations · 22 Oct 2026 United States Europe Other regions Share of CA organizations 17.9%52.4%29.8% 15 U.S. · 44 European · 25 elsewhere Certificate share 92.76% U.S. 92.76% · Europe 7.15% · Other 0.09% Note: Unexpired certificates. Geography uses 84 classified CAs; 88 appear in at least one root program. Source: WebPKI Observatory, Geographic distribution, 22 Oct 2026.

The Observatory also identifies 30 government-operated or state-owned CA organizations in the root stores, which together account for 0.09% of unexpired certificates. Their presence may serve national policy objectives, and their safety has to be judged case by case, but the rationale for their broad trust cannot rest on the share of the certificate market they serve.

The Observatory counts 88 organizations trusted by at least one major root program, while the geographic figures use only the 84 it classifies by location. The 92.76% and 7.15% figures come from that geographic series and are computed separately from the top-five ranking.

This is the distinction between what the web needs and what the governance system permits. Many organizations participate in global trust, and very few of them carry nearly all of the actual certificate footprint.

Certificate counts and browser usage tell different stories

Matthew McPherrin's cert-count dataset offers a second measure of dependence. It aggregates Mozilla's cert.validation_success_by_ca_2 telemetry, which counts successful server certificate validations attributed to each CA, for the Firefox 157 and 158 release channel builds, as retrieved from Mozilla's GLAM service on October 22, 2026. The issuance data shows which providers supply deployed certificates, and the telemetry shows which CAs participating Firefox clients actually encounter.

Certificate share vs. Firefox validation shareCertificate share and Firefox validation share by CA. ISRG 32.6 percent of certificates and 26.54 percent of validations; Google Trust Services 21.0 and 28.53; DigiCert 17.8 and 21.20; GoDaddy 10.6 and 1.44; Sectigo 6.8 and 5.23; other CAs 11.2 and 17.06.Certificate share vs. Firefox validation sharePercent of each source total · different measurement windowsCERTIFICATESFIREFOX VALIDATIONSISRG / Let’s Encrypt32.6%26.54%Google Trust Services21.0%28.53%DigiCert17.8%21.20%GoDaddy10.6%1.44%Sectigo6.8%5.23%Other CAs11.2%17.06%Certificates: WebPKI Observatory, 22 Oct 2026, unexpired certificates.Firefox: GLAM cert_validation_success_by_ca_2, Firefox 157–158 release channel, via mcpherrinm/cert-count.

The two columns rank the same CAs differently. Google Trust Services leads Firefox validations at 28.53% while holding 21.0% of certificates, and ISRG leads certificates at 32.6% while carrying 26.54% of validations. GoDaddy shows the widest gap, with 10.6% of unexpired certificates but only 1.44% of Firefox validations. Amazon Trust Services, which sits inside the remaining certificate share, accounts for 10.93% of validations on its own. A provider can serve millions of lightly visited names while another protects a smaller population of sites people load many times a day, so the two measures diverge for ordinary reasons. The columns come from different measurement windows and should be compared as rankings of dependence rather than as a synchronized time series.

The telemetry counts successful validations rather than distinct users, websites, or visits, and it samples participating Firefox installations rather than the browser population as a whole. StatCounter's September 2026 measurements put Firefox at 2.80% of worldwide browser share across devices, or 6.11% on desktop, and Firefox users need not browse the way Chrome or Safari users do. Browser share shows how narrow the sample is, and multiplying Firefox's CA percentages by it cannot reveal the CA mix in other browsers.

What happens when a CA certifies the wrong website?

Both measures describe CAs that websites chose or that Firefox users encountered. Neither says what happens when a CA certifies a website it has no right to certify, and answering that requires looking at what a browser actually checks.

When you visit a website over HTTPS, the site presents a certificate naming the domain and a public key. Your browser checks that the certificate chains to a CA in its root store and that the domain name and other requirements match. The browser does not normally require that the website chose that particular CA, and it accepts a valid chain to any CA authorized for that name in its root store. This is what makes competition and interoperability possible, since a site can change providers without asking anyone to update a browser.

It also means that, unless a CA's authority is limited by name constraints or other validation controls, a CA serving a tiny local market can make a certificate claim about an unrelated website anywhere in the world. The victim does not have to be that CA's customer. Successful impersonation still depends on the attacker's ability to present the certificate and on client protections such as Certificate Transparency, revocation, and blocklists, but the CA's share of legitimate certificates places no bound on the set of names it might wrongly authenticate.

A CA responsible for 0.001% of deployed certificates does not have 0.001% of the ability to issue a certificate for a major website. Issuance share describes commercial and operational dependence, and Firefox validations describe observed browser exposure. Neither describes how broadly a CA is allowed to sign, which is what determines the cost a CA imposes on the rest of the web.

When a small CA fails, the rest of the world pays

The cost of adding another broadly trusted issuer goes well beyond that issuer's audits or the number of certificates it issues. Every relying party that accepts it takes on that issuer's key management, validation behavior, incident response, ownership changes, and eventual failure.

On October 17, 2026, a researcher reported that the private keys for eight eMudhra subordinate CA certificates had been generated with insufficient randomness and could be recovered. Chrome's account says its team corroborated the report and blocked the affected intermediate public keys through CRLSet within roughly 100 minutes, and Mozilla distributed a OneCRL block. Cloudflare describes remediation spanning clients, proxies, services, trust stores, containers, and applications that do not inherit browser updates automatically.

Chrome's report also says that no TLS certificates chaining to the six affected intermediates within the scope of its CT check had been logged before the block, so CT-enforcing clients could not have accepted unlogged forged certificates through that publicly trusted path. eMudhra's incident statement says none of the eight intermediates had issued end-entity certificates. That statement describes its legitimate issuance records, and anyone able to recover a signing key could have produced certificates outside them. The remaining risk sat with clients that trusted the keys but did not enforce CT, receive timely blocklist updates, or inherit browser protections.

Chrome and Mozilla acted decisively, and the incident still required coordinated remediation well beyond browsers, because public trust extends into applications and devices with different validation logic and update cadences. eMudhra Technologies accounted for 0.025% of the Firefox validations in the telemetry above. It did not have to matter to those clients' legitimate traffic in order to matter to their security.

The benefit of local issuance can be concentrated in one country or customer segment, while the authority conferred by broad trust, and the work of cleaning up its failures, are spread across everyone.

Where the WebPKI’s 16 recorded trust actions occurred

The Observatory's historical list records 16 trust actions against CAs from 2011 through 2024. The Observatory labels them distrust events, but they range from complete root removal to narrower restrictions. ANSSI's 2013 incident, for example, ended with the offending intermediate blocked and the root's scope limited. By jurisdiction of the CA operator, five involved European CAs (31.25%), four involved Asian CAs (25%), three involved U.S. CAs (18.75%), two involved CAs in Türkiye (12.5%), and two involved Latin American or Caribbean CAs (12.5%). In total, 13 of 16, or 81.25%, involved CAs outside the United States. The chart treats Türkiye as its own transcontinental category and places StartCom under its historical Israeli operating identity, with its later Chinese ownership covered in the methodology notes.

Where the 16 historical trust actions occurredRegional counts: Europe 5, Asia 4, United States 3, Türkiye 2, Latin America and Caribbean 2, summing to 16; 13 outside United States.Where the 16 historical trust actions occurred2011–2024 · jurisdiction of associated CA · recorded trust actions, not failure ratesEurope5 · 31.25%Netherlands: DigiNotar · 2011 · France: ANSSI · 2013; Certinomis · 2019Spain: Camerfirma · 2021 · Austria: e-commerce monitoring · 2024Asia4 · 25%India: India CCA / NIC · 2014 · China: CNNIC · 2015; WoSign · 2016Israel*: StartCom · 2016United States3 · 18.75%U.S.: Symantec · 2017; Visa · 2022; Entrust · 2024Türkiye2 · 12.5%Türkiye: TÜRKTRUST · 2018; e-Tugra · 2023Latin America & Caribbean2 · 12.5%Venezuela: PROCERT · 2020 · Panama: TrustCor · 2022 Outside the United States: 13 of 16 (81.25%) * StartCom began as an Israeli CA and was Chinese-controlled by its 2016 distrust. Years mark listed trust actions, not original incident dates. Includes restrictions (e.g., ANSSI).

These are descriptive counts of trust actions. They are neither per-CA failure rates nor a standardized record of root removals, and the cases span different eras, root programs, ownership structures, and kinds of action. Estimating comparative risk would require historical denominators covering how many CAs operated in each region each year, how much trust and issuance each held, and consistent inclusion criteria. The counts show where trust actions have arisen and say nothing about any national characteristic of European or non-U.S. CAs.

A narrower and more serious category consists of unauthorized certificates capable of impersonating domains that were actually issued. That category is distinct both from a CA's mere ability to issue such certificates and from proof that traffic was intercepted. Several cases on the list have direct evidence of it.

DigiNotar (Netherlands, 2011). A fraudulent *.google.com certificate was observed being presented to users in Iran, as documented in the independent Fox-IT investigation.
TÜRKTRUST (Türkiye, discovered 2012). A mistakenly issued subordinate CA was used to produce a fraudulent *.google.com certificate. This was the 2012 to 2013 misissuance, which predates the Observatory's 2018 entry for TÜRKTRUST. Google's incident report and Microsoft's advisory document it.
ANSSI (France, 2013). An unauthorized subordinate certificate was installed in a TLS interception appliance and used for domains the holder did not control, according to Mozilla's advisory and Google's disclosure.
India CCA / NIC (India, 2014). NIC issued unauthorized TLS certificates covering Google and Yahoo domains. Google noted that the Indian government roots were then trusted through the Microsoft root store rather than by every browser or platform, and it constrained its own trust in response. Google's investigation and Microsoft's advisory give the details.
CNNIC / MCS Holdings (China, 2015). An intermediate issued under CNNIC produced unauthorized certificates for Google domains, and Google documented the incident and its immediate block.

The list is illustrative rather than exhaustive, and interception was not demonstrated in every case. Counting leaf misissuance, trust restrictions, and CA removals in a single number would combine different events, and an unauthorized subordinate CA certificate belongs in this category only when later leaf issuance is documented.

The geographic chart shows how trust actions are distributed, and the incident reports show how the failures happen. Both point to the same mechanism. A CA with little legitimate traffic can produce an authentication claim that is accepted far outside the market it serves, and a single unauthorized certificate is enough to demonstrate it.

A country can control local identity without controlling global TLS trust

The European Union’s eIDAS framework puts this tension into law. Articles 45 and 45a of Regulation (EU) 2024/1183 require browsers to recognize qualified certificates for website authentication, display specified identity attributes, and observe procedural requirements when taking certain precautionary security measures. Article 45(1b) also limits additional mandatory requirements. Member states determine which providers are qualified under the European framework. The accompanying recitals express an intention to preserve browsers’ freedom to secure connections, but recitals guide interpretation; they do not cancel the operative obligations. The unresolved tension is between independent browser security judgments and the legal requirements attached to recognizing a government-approved trust service.

These requirements arise within the EU’s jurisdiction, but their practical effects are difficult to confine there. A website generally presents the same certificate chain to visitors around the world. If that chain is accepted in one region and rejected in another, the site can fail for visitors outside the first region. Maintaining region-specific chains or browser trust policies across the global web would require complex changes to certificate automation, deployment, location handling, caching, and compatibility. That is not a realistic general answer to conflicting trust requirements. Consequently, European decisions about browser obligations can affect relying parties far beyond Europe, even though the EU does not directly regulate every user or application worldwide.

The history makes this concern concrete. European CAs account for five of the sixteen historical trust interventions in the Observatory’s list, more than any other region in that dataset. Those entries include ANSSI, where browsers independently restricted a CA’s scope after unauthorized issuance. The historical count is not a regional failure rate, but it demonstrates why browsers need to retain the ability to intervene on technical security grounds. Article 45a places procedures and conditions around certain precautionary measures involving qualified certificates. That creates a pointed governance question: when the security interests of global relying parties and the qualification decisions of a regional regulator conflict, who gets to decide what a browser must trust?

The WebPKI has already used geographic scope in incident response. After the 2013 ANSSI misissuance, Chrome announced that it would limit the ANSSI CA to .fr and a defined set of French territorial top-level domains, and Mozilla documented the same restriction. The offending intermediate was blocked, and the CA's wider trust was restricted rather than removed. After the 2014 India CCA / NIC incident, Chrome announced a similar restriction covering specified Indian government, academic, banking, and technology domains. In neither case did the browsers have to conclude that every domestic certificate was worthless, and both responses reduced where the trust could be exercised.

These were restrictions enforced by browser root programs, and other platforms did not necessarily enforce equivalent X.509 name constraints. Country-code scopes also fit the real DNS poorly, since German and Swedish organizations use .com and a country-code suffix maps cleanly to neither citizenship, ownership, nor regulatory jurisdiction. Even so, the ANSSI and India CCA cases show that global trust scope is a policy and implementation choice, and a functioning domestic CA does not depend on it.

The European framework bears directly on this kind of response, and the irony is hard to miss. Europe accounts for more of the sixteen trust actions than any other region, including ANSSI, the case that gave browsers their clearest precedent for scoped trust. The ANSSI restriction was a decision Chrome and Mozilla made on their own and carried out quickly. For QWACs and the qualified providers that issue them, Article 45a now attaches a prescribed procedure, including notification of the Commission and the relevant supervisory body, to precautionary measures of that kind. Europe leads the documented CA trust-intervention count in this dataset, while also having written the rules that most directly constrain browsers' ability to respond to failures involving qualified providers.

A local-language issuance service, a domestic registration authority, or a regulated provider of verified legal identity can meet much of the practical customer demand without adding another unconstrained global root. A government may want something stronger, namely full cryptographic independence from foreign CA operators. That is a genuine sovereignty objective with real benefits and trade-offs, and it should be debated on those terms rather than treated as a free property of a shared global root store.

Post-quantum migration gives us a chance to change the trust model

The classical public WebPKI is not the endpoint. Quantum computers threaten the signature algorithms its authentication model relies on, and large post-quantum signatures create practical scaling problems for classical X.509 certificate chains. As explored in The Post-Quantum WebPKI and A Deep Dive on the Classical WebPKI, the transition involves much more than swapping one signature algorithm for another.

Chrome's post-quantum WebPKI work includes Merkle Tree Certificates and a proposed separate quantum-resistant root program. The details are still evolving, and classical and post-quantum systems will coexist through the migration. The cryptographic pressure behind the migration has nothing to do with sovereignty, but it creates a rare moment when root program admission and trust scope can be reconsidered deliberately. A Merkle proof cannot tell a browser whether a legal or domain control assertion was authorized, and nothing about quantum resistance requires granting every admitted issuer authority over every domain. The longer technical analysis covers the MTC machinery, and the decision at issue here is a governance one.

Who should bear the security cost of sovereign CAs?

Limiting the scope of trust does not require reducing the number of CAs. Concentrating issuance in a few very large providers would create its own points of operational failure, commercial dependency, censorship pressure, and geopolitical leverage, and a CA's nationality says nothing about the competence of its staff. Scoping trust leaves room for many issuers while limiting what each one can sign.

The practical choices are narrower. Sovereign or locally focused CAs can be constrained to country-code top-level domains and to specifically enumerated government or other approved domains, including names under generic top-level domains where a meaningful scope can be defined. Unconstrained global authority should be reserved for CAs that meet a substantially higher bar for technical quality, operational discipline, incident response, and demonstrated value to the web as a whole. That bar is hard to quantify fairly, but the difficulty of drawing it does not justify making worldwide authority the default.

This is a cost accounting problem as much as a technical one. The provider, country, or customer base that receives the local benefit does not bear all of the security cost. When a broadly trusted CA fails, browsers, independent software vendors, enterprises, service operators, and relying parties elsewhere absorb the assessment and remediation burden, and that burden bears no relation to legitimate issuance or usage share.

One way to distribute that cost would be to require globally trusted CAs to post a bond or carry comparable financial assurance. That would at least make admission carry a price tied to the harm an issuer can impose. But it could also turn access to global trust into a test of capitalization rather than operational quality. The Internet Security Research Group, which operates Let's Encrypt, is a nonprofit and one of the web's best-run CAs; a large bond could exclude or burden it without proving that a wealthier operator is safer. Any financial mechanism would therefore have to avoid treating the ability to pay as evidence of competence.

The tax is paid continuously, not only after an incident. Every additional broadly trusted issuer adds another organization whose key management, validation controls, software, personnel, governance, and incident response everyone who accepts it must depend on, even if those relying parties never intentionally transact with that CA. A compromise makes that dependency visible, but the dependency existed long before the compromise was discovered.

A Swedish company can reasonably want a Swedish-language CA. A German ministry can reasonably want a German trust provider. An Indian company can reasonably compete for global customers. We may choose to support all of those aims for the sake of competition, resilience, and sovereignty, but technical necessity, customer preference, and national policy are three different arguments, and only technical necessity answers what the web requires in order to function.

We are about to migrate one of the internet's oldest shared trust systems to post-quantum cryptography. Before we do, it is worth asking whether to carry forward a design in which a CA with little legitimate global market share can still authenticate names everywhere. If the local benefit is real, why should the global scope be automatic?

A CA that is trusted everywhere has impact everywhere, and that is the sovereignty tax.

Data and methodology notes

Observatory snapshot. WebPKI Observatory landing page and JSON snapshot viewed October 22, 2026. Market-share figures are proportions of unexpired certificates, not counts of distinct site operators. The Observatory reports 88 CA organizations in at least one major root program and 84 geographically classified organizations, and the geographic comparison uses the 84. Top-five shares are rounded to one decimal place.
Jurisdiction method. The Observatory's 16 historical trust actions are assigned to the CA operator's country, not to the location of affected websites or users. Europe (5) covers DigiNotar (Netherlands), ANSSI and Certinomis (France), Camerfirma (Spain), and e-commerce monitoring/GLOBALTRUST (Austria). Türkiye (2) covers TÜRKTRUST and e-Tugra. Asia (4) covers India CCA/NIC (India), CNNIC and WoSign (China), and StartCom (historically Israel, later Chinese-controlled). Latin America and the Caribbean (2) covers PROCERT (Venezuela) and TrustCor (Panama). The United States (3) covers Symantec, Visa, and Entrust. The list mixes eras and kinds of trust action, including ANSSI's scoped restriction, so 13 of 16 is descriptive and is not a risk estimate. Mozilla's PROCERT root inclusion record identifies its Venezuelan operating identity, TrustCor identifies its Panamanian registration, and Mozilla's StartCom ownership discussion documents its ownership history.
Firefox method. Shares are computed from CA.csv in the mcpherrinm/cert-count repository as of its October 22, 2026 update, which sums Mozilla GLAM cert.validation_success_by_ca_2 histograms for Firefox 157 and 158 on the release channel and maps histogram bins to CA names using Firefox's KnownRootHashes.txt. The total is 178,009,832,890 successful validations. The telemetry reflects participating Firefox clients only, and its window differs from the Observatory snapshot.
Legal timeline. Regulation (EU) 2024/1183 amended eIDAS. Articles 45 and 45a are discussed in their adopted form.
Quantum timeline. MTC and Chrome's proposed quantum-resistant root program are an announced program and an active standards effort. Proposed policies and rollout dates are not presented as completed facts.

Primary and supporting sources. WebPKI Observatory · Observatory data snapshot · Mozilla CA telemetry metric · Firefox CA validation dataset · StatCounter browser share · Chrome eMudhra notice · Mozilla eMudhra notice · Cloudflare eMudhra notice · eMudhra incident statement · eIDAS amendment · Fox-IT DigiNotar report · Google TÜRKTRUST report · Google ANSSI response · Mozilla ANSSI advisory · Google India CCA response · Google CNNIC response · Chrome's MTC announcement · Classical WebPKI explainer · Post-Quantum WebPKI explainer

The Private Key Was Four

Thu, 22 Oct 2026 16:00:00 GMT

Every time a browser loads a page over HTTPS, it is trusting a chain of signatures. A certificate authority signs the website's certificate, and the browser accepts it because that authority sits on a short list the browser ships with. Publicly trusted CAs keep their top-level root keys offline and do their daily signing with intermediates, subordinate keys the root has vouched for. Whoever holds an intermediate's private key can issue a certificate for any website, and browsers will accept it.

So everything rests on the private key staying secret and unguessable. For the elliptic curve in this story, P-384, a private key is a number picked at random from a range about 116 digits long. The assumption being that no computer that will ever be built could search that range.

In April 2025 eMudhra, a certificate authority based in India and trusted by Apple, Google, Microsoft and Mozilla, created a batch of new intermediates. The private key of one of them, EM OV TLS CA - G3C, was the number 4. Written out as a P-384 key it looks like this:

00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000004

A properly generated one looks like this:

84ac06790886c80c48d31691fd4c0fe9
ce3207f9807f2fe571d6e4ec426d7864
4298fc53df2b92566970e757ac998c34

The keys came out of a key ceremony, these are supposed to be a scripted and witnessed session in which a CA generates keys inside its hardware security modules. Seven sibling intermediates from the same ceremony were no better. Their private keys were 56, 66, 115, 155, 156, 230 and 237. Every one was below 256. Anyone with a copy of the certificate could recover the key by trying each number from 1 to 255 until one produced the public key printed in the certificate. I checked all eight myself. It takes less time than reading this sentence.

Those certificates were public the whole time. CAs must disclose every intermediate in the CCADB, the shared database the browser root programs use to track CAs, and these went in on 10 May 2025. On 17 October 2026 a researcher who goes by Wayne ran paranoid_crypto, a free key-testing tool Google released in 2022, over the CA certificates in CCADB. Twenty-seven minutes after he started, it flagged the eight keys. About two hours later he reported them to Apple, Chrome, Microsoft, Mozilla and eMudhra, with a message signed by each key as proof. Chrome blocked the six intermediates that could issue website certificates within two hours of his email, and Mozilla within five. By 20 October both had decided to remove the eMudhra root above them, emSign ECC Root CA - G3, entirely. On 22 October eMudhra announced it was leaving the global TLS certificate business.

The proof was barely needed. Dennis Jackson of Mozilla recovered all eight keys independently within two hours of receiving the report, and noted that AI agents running on his own machine recovered them on the first try when told the task was a capture-the-flag puzzle. Within three days several people had posted the keys, and one showed a forged google.com certificate working against an unpatched client.

For seventeen months, then, the private keys of six website-signing intermediates trusted by every major browser could be computed by anyone who looked. Wayne had even seen the first symptom. In June 2025 he commented on one of two public bug reports that, as it turns out, documented these exact keys misbehaving. eMudhra had the same evidence in hand that June. It knew its certificates did not hold the keys it meant them to hold, switched to the keys the certificates named, and kept operating for sixteen more months.

The CRLs that told the truth

The 30 April 2025 ceremony produced 24 issuing CAs in three series (A, B and C), each with RSA-4096 and P-384 versions for DV, OV, EV and code signing. The HSM mapping eMudhra posted shows the A series was generated on an Entrust nShield Connect XC, and the B and C series on a Utimaco Se-Series Gen2. Only the P-384 keys from the Utimaco went wrong. The RSA siblings from the same Utimaco look sound on every cheap test I ran (no shared factors with any other CCADB key, no Fermat weakness, no small factors), though cheap tests prove nothing about generation. The B series belonged to eMudhra BV in the Netherlands and the C series to eMudhra Inc in the US. eMudhra says they were built for data-sovereignty customers and only one, EM DV TLS CA - G3C, was ever switched on. None issued a certificate.

Date	What happened
30 Apr 2025	Ceremony creates 24 CAs. Per eMudhra, a wrap and unwrap test produces a second, weak key pair for each B and C P-384 CA
10 May 2025	All 24 published to CCADB, two days late (Bug 1965559). From here the public keys are open to anyone
14 May 2025	Sectigo cross-signs all 24 from AAA Certificate Services
22 May 2025	Chrome tells eMudhra in Bug 1965559 that its earlier preventive actions failed and it is not learning from other CAs' incidents
23 May 2025	Mozilla later reports an archived crt.sh page showing the six P-384 CRLs failing from this date
29 May 2025	The CRLs eMudhra dates the problem to are published. The A series is revoked and replaced by A-1 the same day, for reasons not yet explained
3 Jun 2025	Rob Stradling asks eMudhra whether the CRLs were signed with the wrong private key. The email is quarantined
5 Jun 2025	Stradling opens Bugs 1970727 and 1970728
6 Jun 2025	eMudhra publishes CRLs that verify, signed with the weak keys
18 Jun 2025	Sectigo cross-signs the eight A-1 replacements
16 Jul 2025	Bug 1970728 closed
8 Nov 2025	Chrome reviews eMudhra's hierarchy metadata during a root inclusion evaluation
6 Aug 2026	BDO issues unmodified WebTrust opinions naming Bug 1970728
2 to 9 Oct 2026	Mozilla's intent to approve eMudhra's new roots and last call
14 Oct 2026	An eMudhra root CRL fails signature verification
17 Oct 2026	Wayne reports. Chrome and Mozilla block within hours
20 Oct 2026	Mozilla removes emSign ECC Root CA - G3 from NSS. Chrome commits to the same
21 Oct 2026	Chrome posts twelve questions for eMudhra in Bug 2083449
22 Oct 2026	eMudhra announces it is leaving the global TLS certificate business

On 29 May 2025 eMudhra published CRLs for six of the new intermediates, the same six TLS CAs whose keys were later recovered. None of the signatures verified. crt.sh flagged them, and on 3 June Rob Stradling of Sectigo sent eMudhra a problem report asking whether the CRLs had been signed with the wrong private key. The email sat in a spam quarantine. On 5 June he opened Bug 1970728.

eMudhra's incident report said a manual CRL process had used an incorrect CA key configuration. The keys came from the issuing environment but did not match the keys expected for those CRLs. It regenerated the CRLs, and committed to automating CRL signing and verifying every CRL after signing. The bug closed on 16 July 2025.

The CRLs eMudhra attached to that bug say more than the report does. From an ECDSA signature and the signed data you can compute the small set of candidate public keys that could have made it, usually two. eMudhra attached five distinct CRLs, uploading one of them twice and leaving out EM DV TLS CA - G3B. I did that for each. Every candidate verifies the signature, none has a small private value, none matches the key in the CA's certificate, and no candidate key appears in any of the 10,313 CA certificates in CCADB. So if we believe what they said in June 2025 these CAs each had two keys on the HSM. One was a normal key that eMudhra's systems treated as the CA's key. The other was the key the certificate actually certified.

In October 2026 eMudhra explained what happened next. The CRLs had been signed with the original keys, the ones generated with appropriate entropy. Those signatures failed, so eMudhra re-signed them with the flawed keys. The fix for the invalid CRLs was to start using the weak keys, because they were the ones that matched the certificates.

That was the decision that mattered. eMudhra held cryptographic proof that its certificates did not contain the keys its systems treated as the CA's, and it resolved the discrepancy in favor of the certificates. Then it kept those CAs in service for another sixteen months. The 2025 investigation reached the right symptom and stopped one question short. Both of its action items concerned the CRL signing path. Neither asked why a CA had two keys, which one was legitimate, or whether the certified key had been generated as intended. Answering that required no cryptanalysis, only treating "the expected key does not match the certificate" as a question about the certificate rather than about the CRL tooling.

According to eMudhra's preliminary root cause, the 30 April ceremony included an approved deviation to test key wrap and unwrap, because the B and C series keys were meant for export to European and US data centers. A flaw in the script generated a second, weak key pair on the Utimaco HSM. The key labels were similar, and the CSRs were built from the weak keys.

Each CA held two keys, and the fix put the weak one to workOne of the six TLS CAs, from eMudhra's account and the keys recovered from its CRLs30 Apr 2025 key ceremony on the Utimaco HSMStrong keyGenerated as intendedMatches no certificateWeak keyLeft by the wrap and unwrap testPrivate value below 256Treated as the CA keyby eMudhra's CRL signingCSR built from the weak keySimilar key labels on the HSM29 May 2025 CRLsSigned with the strong keyNo signature verifiesCA certificateCertifies the weak keyCCADB 10 May, Sectigo 14 May6 Jun 2025 fixCRLs re-signed with the weak key, so they verifysignatures failkey matches the certificate

eMudhra has not published the script, the mechanism and parameters it passed to C_WrapKey, or the attribute template it gave C_UnwrapKey. A wrong parameter, or a mismatch between the encoding the wrap produced and the one the unwrap expected, could turn a 384-bit scalar into a single byte. Every recovered key is below 256, which fits a parsing error far better than a weak random number generator. The ceremony needed a known-answer test before any production key went through that path. Wrap a key generated in software under a wrapping key the operator also holds in software, decrypt the blob to confirm its encoding, unwrap it on the HSM, and check that the resulting public key matches the original. That dry run belongs after every change to the script or the HSM firmware as well.

Commenters in the bug have pressed the same point. eMudhra's own CP says issuing CA keys are generated in a device that is at least FIPS 140-2 Level 3. A P-384 scalar below 256 is not something approved in-module generation produces, so the weak keys most likely came from material the script supplied, through a call such as C_UnwrapKey or C_CreateObject. In that case the weak keys may never have been confined to an HSM at all. The private value existed outside the module wherever the script ran, and possibly in its logs, before it was loaded. With a key below 256 the secrecy question is moot, since anyone can compute it. The control question is not. A CA key that was loaded rather than generated inside the module does not meet eMudhra's own CP. One commenter has asked eMudhra for the script, the exact call, whether the module was in its FIPS-approved mode, and who approved the deviation. Another proposed that root programs require a manufacturer-signed attestation that every CA key was generated inside the HSM rather than loaded into it. eMudhra's public statement says it has not yet confirmed the exact root cause.

The 2025 report even has a section titled "Where we got lucky". It says the new intermediates had not yet issued any end-entity certificates, which limited the exposure. That was true, and for a reason the authors could not see. Its lessons-learned section called for automating manual steps in creating new issuing CAs. Automating CRL signing against the certified key would only have put the weak keys to work sooner.

The auditors saw this incident too. BDO's WebTrust report for June 2025 through May 2026 lists all eight affected intermediates in scope, names Bug 1970728 as an Other Matter, and says the auditor examined the disclosed root cause. The opinion was unmodified. The ceremony itself fell in the prior audit period, and I have not been able to find that report.

Why no one else looked

Once eMudhra chose the certificates over its own key records, nothing else in the system was positioned to catch it. Six publicly trusted keys sat in plain view for seventeen months because no rule asked anyone except eMudhra to look, and the one check the rules recommend would have passed them.

The Baseline Requirements treat CA key generation as a process control. For root keys and for subordinate CAs run by an unaffiliated operator, a Qualified Auditor must witness or video the ceremony and then report on whether it followed the script. eMudhra BV and eMudhra Inc are affiliates, so for these keys both were a SHOULD. A commenter in the bug has asked whether Sectigo's cross-certification two weeks later changes that, since relative to Sectigo's root eMudhra is an unaffiliated operator. The ceremony would still have had internal attendees and witnesses. eMudhra still had to keep the script, logs and witness records as cradle-to-grave evidence of each key, and the period audit is where that evidence gets examined. The weak-key rules in section 6.1.1.3, covering Debian keys, ROCA and close primes, apply to subscriber keys. Nothing tells an issuing CA, a cross-signer or a root program to screen a subordinate CA's key that way.

The ECDSA quality check in section 6.1.6 is NIST public-key validation, and it is a SHOULD. It confirms the point is on the curve and has the right order. A private key of 4 produces a perfectly valid point. Public-key validation looks only at the point, so by design it cannot tell how the private key was chosen. The recommended check would have passed every one of these keys.

The check that did work has existed for years. Google published paranoid_crypto in June 2022. Its CheckWeakECPrivateKey test, a default check rated critical, looks for EC keys whose private value fits in 32 bits. It was public nearly three years before these keys were generated.

I wanted to know what running it at disclosure time would cost, so I pulled every CA certificate in CCADB, 10,313 of them, and tested each EC key for a private scalar below 65,536. The scan took 0.3 seconds after a one-time table build of under a minute. It returned exactly sixteen certificates. Eight were the eMudhra intermediates and eight were Sectigo's cross-certificates of the same keys. There were no false positives. A batch GCD across all 6,669 unique RSA moduli found no shared factors either.

Several parties had a reason to look. Chrome reviewed eMudhra's hierarchy metadata in November 2025 during a root inclusion evaluation. Mozilla posted its intent to approve eMudhra's new roots, including emSign Root TLS CA - G3, the root that issued the six TLS intermediates, on 2 October 2026. The CCADB public discussion ran from August to September without objection. Sectigo cross-signed all 24 keys from the April ceremony. None of these processes is designed to test key values, and none of them caught the problem.

Who	What they had	What the rules asked of them	What happened
eMudhra, at the ceremony	The HSM, both key pairs, the CSRs	Effective controls so keys are generated as the CPS and script describe (BR 6.1.1.1)	A script flaw and similar key labels; CSRs built from the weak keys
eMudhra, in June 2025	Six CRLs signed by keys matching no certificate, and a report asking about the wrong key	A root cause analysis that reaches the real cause	Fixed the CRL path, re-signed with the weak keys
Third-party auditor at the ceremony	Internal attendees and witnesses were present; unknown whether a Qualified Auditor attended	Auditor witness or video, and an auditor's key generation report, are a SHOULD for affiliate sub-CAs (BR 6.1.1.1)	No public record of auditor attendance or a key generation report
Auditor for the ceremony period (June 2024 to May 2025)	The ceremony script, logs and witness records eMudhra must keep as key lifecycle evidence	WebTrust key generation and lifecycle criteria, tested in the period audit	Period audit report not found
BDO, June 2025 to May 2026	Bug 1970728 and eMudhra's root cause	Reasonable assurance on key integrity	Examined it, unmodified opinion
Sectigo, as cross-signer	32 eMudhra keys across two batches	No rule to test a subject CA's key	No evidence of a key test
Sectigo, via crt.sh	Automated CRL verification	None	Raised the only public warning
Root programs	CCADB records from May 2025, an inclusion review	Disclosure, audits, incident handling	No key scan; nothing requires one
Certificate Transparency	Nothing, since no certificate was issued	None	Could not have seen it
Researchers	Public keys from May 2025, a free tool from 2022	None	Found in 27 minutes, seventeen months late

Each layer of WebPKI assurance was built for a different failure. Audits test whether controls exist. Root programs watch disclosure and incident handling. Certificate Transparency watches issuance, and these CAs never issued a logged certificate, so CT had nothing to see. Key validation tests curve membership. The one property that was wrong sat outside all of them.

Where the keys reached

The six TLS intermediates were issued by emSign Root TLS CA - G3, a 2024 root that only Microsoft includes directly. Mozilla, Chrome and Apple trusted them through a cross-certificate from emSign ECC Root CA - G3, which has been in all four programs since 2019. The two code-signing CAs mattered only on Windows, where anyone could have signed software that chained to a trusted root until Microsoft's Disallowed list update reached the machine.

The third path is the one most coverage has missed. On 14 May 2025 Sectigo cross-signed every CA from the April ceremony, 24 certificates, from its AAA Certificate Services root. The eight affected keys therefore appear in sixteen certificates.

Sectigo's cross-certs carried the weak keys past eMudhra's rootsPaths from the eight keys to trust anchors, as of 21 Oct 20268 weak keysP-384 scalarsbelow 256in 16 certs6 TLS CAsEM DV/OV/EV TLS CAG3B and G3CAny domain, pathlen 0Root TLS CA - G3Root in Microsoft onlyCross-certified in 2024by ECC Root CA - G3ECC Root CA - G3In all four programsNSS removed 20 OctChrome removing, v438 cross-certsIssued by Sectigo14 May 2025Same public keysAAA Cert ServicesApple, Microsoft for TLSPre-2025 Mozilla bundlesExpires 31 Dec 2028Reaches clients thatnever trusted eMudhra2 code-signing CAsEM OV CS CA - G3B/CCode signing EKU onlyRoot CS CA - G3Microsoft onlyAlso via AAA forWindows code signing

AAA Certificate Services dates from 2004 and sits in most trust stores built in the last twenty years. Mozilla removed its Websites trust bit in April 2025 and Chrome dropped it from the Chrome Root Store the same month. Apple and Microsoft still trust it for TLS, and Microsoft for code signing. Every Mozilla-derived bundle from before April 2025 trusts it for TLS. A device with a 2015 trust store never trusted eMudhra, and it still accepts a forged certificate chained through Sectigo's cross-cert. That path closes when AAA expires on 31 December 2028. The path through emSign ECC Root CA - G3 lasts until 2039 in any store that keeps it.

The constraints narrow what a forger can do, but not by much. The TLS intermediates carry no name constraints, so they can sign for any domain. They are limited to path length zero, so a forger cannot create a new intermediate. Their EKU includes client authentication as well as server authentication.

The timing of the cross-signs is the governance story. Mozilla requires approval before a root in its program cross-signs a CA outside it. Chrome requires notice and express approval for certificates that extend its trust boundary. Sectigo's cross-signs came three weeks after AAA left Chrome and a month after Mozilla stopped trusting it for TLS, so neither rule applied. Apple added a prior-approval rule for external cross-signs on 1 August 2026, fifteen months later. Microsoft requires disclosure and audits, not approval. The cross-signs reached exactly the stores with no approval gate.

Program	Rule for cross-signing a CA outside the program	Since	Applied to the 14 May 2025 cross-certs?
Mozilla	Approval before cross-signing a CA not in Mozilla's store; the signer is accountable for it (MRSP 8.4)	2022	No. AAA lost TLS trust on 16 April 2025 (Bug 1957685)
Chrome	Three weeks' notice and express approval for certificates that extend the trust boundary (policy 1.6.1)	2022	No. AAA removed on 24 April 2025
Apple	Approval before each cross-signed certificate to an externally operated CA (policy 1.4)	1 Aug 2026	No. The rule came later
Microsoft	Disclosure and audits only (requirements)		No rule to apply

Precedent says this matters. Mozilla removed Certinomis in 2019 partly because it cross-signed StartCom, a CA Mozilla was in the middle of distrusting. Today's approval rules each cover their own store. None can reach a root that has already left. Revocation follows the same boundary. Asked in the bug whether the AAA cross-certs belonged in OneCRL, Mozilla replied that AAA is not in its root store, so they do not.

When one store retires a root and another keeps it, cross-signs from that root become the route around the first store's rules. Sectigo's cross-signs took that route. Cross-signing from a root that browsers are retiring is common practice, Sectigo did the same for SSL.com in 2024, and no rule asks a cross-signer to test the subject CA's key. Two questions still belong to Sectigo. It cross-signed eight more eMudhra CAs on 18 June 2025, two weeks after its own staff flagged wrong-key signatures on CAs it had already cross-signed. And it extended a root that browsers had just retired into the platforms with the least oversight of cross-signs.

The response, and who it missed

The browser response was fast and well run. Chrome pushed a CRLSet block on the six TLS keys at 19:45 UTC on 17 October, under two hours after Wayne's email. Mozilla, which received the report through Chrome at 20:06, recovered the keys independently by 21:48 and completed its OneCRL block at 22:59. Google extended the block to Android through Play Services on 19 October. Apple blocked the certificates on 19 October and Microsoft added all sixteen to its Disallowed list on 20 October. eMudhra revoked its eight between 09:24 and 09:38 UTC on 18 October, and Sectigo revoked its cross-certs at 10:35 UTC on 19 October.

The mechanisms that worked best matched on the public key rather than the certificate. Chrome's CRLSet, Microsoft's list, Go's patched certificate verifier and the CT log software patches all blocked by key, so the cross-certs were covered without anyone having to enumerate them. Cloudflare's log patch says so in its commit message. Mechanisms keyed on issuer and serial need an entry per certificate, and every cross-sign is one more entry to miss.

Mechanism	What it blocked	Live	Matches on
Chrome CRLSet	Six TLS intermediates and their cross-certs	17 Oct, 19:45 UTC	Public key
Mozilla OneCRL	Six TLS intermediates	17 Oct, 22:59 UTC	Issuer and serial
Android, via Play Services	Same as CRLSet	about 19 Oct	Public key
Apple	The affected certificates; mechanism not documented	19 Oct; iOS clients still accepting on 21 Oct	Unknown
Microsoft Disallowed list	All sixteen certificates	20 Oct	Certificate
CT log software (Sunlight, TesseraCT, certificate-transparency-go, Azul)	Submissions chaining to the eight keys	Released 19 to 20 Oct; deployed log by log	Public key
Go crypto/x509 (CVE-2026-111943)	The eight keys, hard-coded	Go 1.27.3 and 1.26.10	Public key
NSS root removal	emSign ECC Root CA - G3	NSS 3.131.1, 20 Oct	Root
Chrome Root Store	emSign ECC Root CA - G3	v43, about 27 Oct	Root

Pushed lists like CRLSets are how browsers revoke today. Live CRL and OCSP checking is broken on the web for many legitimate reasons, including latency, privacy and unreachable responders, so each browser ships its own revocation data instead. Chrome uses CRLSets, Firefox uses OneCRL and CRLite, and Apple and Microsoft push their own lists.

Certificate Transparency did not detect the weak keys, but it recorded their use once they were public. Chrome reports that no certificate chaining to these CAs was logged before the block. The log software fixes that reject these chains by key shipped on 19 and 20 October, but each operator deployed them on its own schedule, and until it did, its logs accepted whatever chained to a trusted root. People used that gap. Cloudflare closed submissions to three shards of its Raio log on 20 October to stop what it called the eMudhra DoS vector, and the ctlogs.dev monitor counted 105,901 certificates naming google.com, issued from EM EV TLS CA - G3B and logged on 21 October.

I scanned every entry in two of those logs, Microsec Eszigno2027h1 and GoDaddy Aquamarine2027h1, from 18 October to the morning of 22 October. Both tell the same story. The first certificate chaining to a weak key was logged at 09:40 UTC on 20 October, for emsign.filippo.io. That evening came pairs issued two minutes apart with the same key, one chained through eMudhra's root and one through the Sectigo cross-cert, for markgamache.com and emudhra.salesforce.com, carrying no AIA, CRL distribution point or policy extensions. Three more single certificates followed on 21 October. Then, between 21:42 and 21:57 UTC, came 1,042 at once in each log, 1,030 of them for .invalid names and 12 for the community site's own test hosts. By the morning of 22 October Microsec's log held 1,050 such certificates and GoDaddy's held 1,055, one of them logged twice. All of them look like researchers and pranksters testing the keys, not like an attack in progress.

Microsec and GoDaddy are separate operators, and Chrome had qualified both logs in September, so each of these precertificates collected SCTs from two independent qualified logs. Two of the pairs, all issued by EM DV TLS CA - G3B with a P-256 subject key, as recorded in each log:

Names	Chain submitted	Serial	Logged (UTC)	Microsec index	GoDaddy index
emudhra.salesforce.com	eMudhra root	39d03c305fc13f907618681fe3cf7cb3	20 Oct 2026, 23:46:08	21768393	141565655
emudhra.salesforce.com	Sectigo AAA cross-cert	6b5995b57a535b8ecf7839a79064afaf	20 Oct 2026, 23:44:30	21768345	141565082
markgamache.com, www.markgamache.com	eMudhra root	02e0978c639f7e464f9ba5a4e46d9e85	20 Oct 2026, 23:38:47	21768199	141563038
markgamache.com, www.markgamache.com	Sectigo AAA cross-cert	228939ef131f959dcb20510501886c35	20 Oct 2026, 23:37:08	21768160	141562540

During that gap, CT on its own stopped nothing. A forgery could collect SCTs from logs that still accepted these chains, and what refused it in a browser was the browser's own key block. Only once a log enforced the fix did it become containment, because a forged certificate could no longer be logged there. A forged certificate presented to a client that does not enforce CT was never going to be logged or seen.

On 20 October Mozilla removed emSign ECC Root CA - G3 from NSS. Firefox was already protected by OneCRL. Mozilla removed the root for the software downstream of NSS that copies its root list but never reads OneCRL. That is a root program acting on behalf of clients it does not ship, and it is the clearest admission I have seen that browser-side revocation does not reach the rest of the Internet.

The rest of the Internet is patching unevenly. certifi, webpki-roots, Node.js and Debian removed the root within days, and Go shipped a hard-coded key block. As of 21 October the OpenJDK cacerts file and the AOSP system store still contain both emSign ECC Root CA - G3 and AAA Certificate Services. Java does not check revocation by default.

Trust store	Status on 21 October
python-certifi	2026.10.19 removes emSign ECC Root CA - G3. Versions from before April 2025 also carry AAA for TLS
rustls webpki-roots	1.0.10 removes emSign ECC Root CA - G3
Node.js	v24.22.1 removes emSign ECC Root CA - G3
Debian	Blocks emSign ECC Root CA - G3 and AAA
Ubuntu	Blocks emSign ECC Root CA - G3
Go	Hard-coded key block
OpenJDK cacerts	Still contains both roots
AOSP system store	Still contains both roots; Play Services block covers some apps

Announcing a block is not the same as delivering it, and in this incident the gap can be measured. The community site that carries the proof, patch list and workarounds, we-have-a-problem.com, runs test hosts that serve certificates from each compromised intermediate, directly and through the Sectigo cross-certs, and counts the clients that accept them. In the five minutes before 23:04 UTC on 21 October, 19,415 of 139,023 sampled connections succeeded. Safari on iOS was the largest group that accepted, two days after Apple's block. Edge on Windows, Chrome on Android and the curl that ships with macOS appeared too. The counts come from people testing their own devices, and user agents can be faked, so they show which kinds of clients still accepted the certificates rather than how many. Firefox had a delivery gap of its own. Some Firefox for Android profiles did not apply the updated blocklist at startup until Mozilla shipped a fix in 158.0.2. CT records what was issued under these keys, not which clients would accept it, so measurement like this is the only view of that side. Root programs timestamp their blocks, but I have found none that publishes how far a block has reached. Current counts are on the site's live statistics page.

A client is exposed only if four things are true together. It trusts either root for TLS. It has not received a block list or root update, which for a browser is its revocation check. It does not enforce CT. It does not fetch CRLs or OCSP responses itself, or fails open when it cannot reach them. Patched browsers fail the second condition. Many appliances, embedded systems and long-lived applications meet all four. We cannot rule out that someone found these keys before Wayne and used them against such clients, because an attacker never needed eMudhra's HSM and never needed to log anything.

A forged certificate gets through only where all three defenses are missingTrusts an affected root for TLSemSign ECC Root CA - G3 or AAA Certificate ServicesnoNever at risk1. Pushed revocation lists and root updatesHow browsers and OSes revoke now that live checks are brokenpresentBlockedUpdated browsers and OSes2. Certificate Transparency enforcementWorks once the logs it relies on enforce the fixpresentBlockedCT-enforcing browsers3. Its own CRL or OCSP check, failing closedThe fallback for software outside the browserspresentBlockedyesmissingmissingmissingForged certificate acceptedMany appliances, embedded systemsand long-lived applications

What happens to eMudhra

On 22 October eMudhra announced that it is leaving the global SSL/TLS certificate business. The board called the decision voluntary. It cited TLS being about 1 percent of revenue, the cost of data-sovereignty requirements, competition from free certificates, and a "level of social media amplification around this incident" that "has brought scrutiny that is out of proportion to SSL/TLS's place in our business." That treats public attention as the problem. The problem is CA keys anyone could compute, which eMudhra had evidence of sixteen months earlier. The release does not mention the June 2025 CRLs or Chrome's twelve questions, and it presents the exit as a business decision made under unfair pressure. The scrutiny was in proportion to the failure. A browser does not trust a CA any less because TLS is 1 percent of its revenue, and a CA key anyone can compute does no less damage. By then Mozilla had removed emSign ECC Root CA - G3, Chrome's removal was set for 27 October, and Chrome had said it was investigating eMudhra's remaining CAs. The exit is voluntary only in the sense that it came before the distrust that was coming anyway.

The same release says eMudhra handled the incident diligently, revoked the affected CAs within 14 hours of discovery, and suffered no compromise of its infrastructure. Its incident statement says it learned of the issue late on 18 October, India time, and revoked on 19 October. The revocation dates recorded for the eight certificates are 09:24 to 09:38 UTC on 18 October, mid-afternoon in India and about fifteen hours after Wayne's report was sent to eMudhra directly. eMudhra's public timeline puts its revocations a day after its own revocation records do. Nobody broke into eMudhra's infrastructure because nobody needed to. The statement also says the impact is "contained to a single customer ecosystem in the United States." Chrome's CT count shows which one. Of about 30,800 unexpired precertificates chaining to the root, 30,791 came from two eMudhra-operated intermediates branded for InCommon, InCommon Intermediate CA - OVG3C and DVG3C, which serve the InCommon certificate service for US universities. They are separate CAs from the eight with weak keys, which issued nothing. Eighteen came from eMudhra's own intermediates. eMudhra put the number needing migration at about 30,000, 97 percent issued through ACME, and is moving them to its RSA root, emSign Root CA - G1, which Mozilla kept "at this time".

On 21 October Chrome posted twelve questions in the bug. They include when eMudhra first knew the B and C CAs had two key pairs, why it re-signed the CRLs with the second key instead of halting, why its 2025 report left out the duplicate keys and the ceremony deviation, and whether Sectigo or any root program was told that the cross-certified keys were not the intended ones. Chrome also noted that eMudhra's status page and press release refer only to "eMudhra's ECC root" and do not name the affected root or intermediates, give removal dates, or tell subscribers how to check their chain. eMudhra announced its exit the next day.

The precedents that matter are the recent ones. Chrome distrusted Entrust in 2024, and Chunghwa Telecom and NetLock in 2025, citing a pattern of compliance failures and unmet improvement commitments. Each case used a forward-dated distrust, so existing certificates kept working while new ones stopped. eMudhra fit that pattern more closely than any of them. It has 31 compliance incidents in Bugzilla since 2020, twelve of them in 2026. In May 2025 Chrome wrote in Bug 1965559 that eMudhra was not applying lessons from other CAs' incidents and that its earlier preventive actions had failed. The fix for the 2025 CRL incident did not just miss the weak keys. It put them into service.

One more recurrence sharpens that. On 14 October 2026, three days before Wayne's report, eMudhra published CRL 11 for emSign Root TLS CA - G1. Wayne flagged it in the bug, and I checked it. The signature does not verify under the root's key, and CRL 12, with identical entries, verified fourteen hours later. Invalid CRL signatures were the failure eMudhra promised in 2025 to catch by verifying every CRL after signing.

The rest of the record shows a CA whose own monitoring finds less each year. Through 2024 eMudhra detected and filed most of its own incidents. From June 2025 most were found by others, including Sectigo, Chrome, outside researchers, customers and problem reports. All five incidents eMudhra filed between 12 and 16 October 2026 name a third party as the source. A 2024 incident found by Google (Bug 1931683) is the closest relative of this one. It involved a key-blocking control that checked only some reissue paths. Key validation implemented narrowly is a pattern, not a one-off.

One removal that landed the same week is not a consequence. On 19 October eMudhra asked Mozilla to remove two older dual-purpose roots, emSign Root CA - C1 and emSign ECC Root CA - C3, completing a retirement plan from February 2026 (Bug 2083709). Coverage that counts three roots removed is counting a scheduled retirement as a sanction.

Year	CA	Trigger	Outcome
2011	DigiNotar	Intrusion and rogue Google certificates, concealed	Full removal everywhere
2011	DigiCert Sdn Bhd	22 certificates with 512-bit keys	Intermediate distrusted
2019	Certinomis	Cross-signed StartCom, plus misissuance	Mozilla removal
2021	Camerfirma	Repeated misissuance and delayed revocation	Chrome, Mozilla, later Apple
2023	e-Tugra	Insecure systems attached to the CA, weak response	Mozilla removal
2024	Entrust	Compliance failures, unmet improvement commitments	Forward-dated distrust in Chrome, Mozilla, Apple
2025	Chunghwa Telecom, NetLock	Same rationale	Chrome forward-dated distrust
2026	eMudhra	CA keys anyone could compute	Intermediates blocked in hours, one root removed in three days, exit from global TLS announced in five

This case is new in two ways. No earlier WebPKI CA key was derivable from public data while trusted. DigiNotar's keys were used by an intruder, not exposed to everyone. And no earlier root removal was justified mainly by clients the root program does not ship.

Mozilla was days from approving a new eMudhra root when the keys surfaced. It posted its intent to approve emSign Root TLS CA - G3 on 2 October 2026. In that thread, a participant asked for an overview of eMudhra's incidents and Mozilla replied that it lacked the capacity to prepare one. Last call closed on 9 October. Eight days later, the root it would have approved turned out to have issued six intermediates with keys anyone could compute. Root programs are run by small teams, and that capacity is the resource this incident spent.

After classical PKI

Merkle Tree Certificates would have removed most of the structures this incident moved through. Under draft-ietf-plants-merkle-tree-certs a CA's key signs batches of its own issuance log, clients require cosignatures from independent mirrors, and there are no intermediates or cross-signs. Chrome's draft policy for its quantum-resistant root store forbids subordinate CAs outright. There would have been no side ceremony for regional intermediates and no cross-cert to carry them into Apple and Microsoft. A client could distrust a CA's issuance after a given log index, one cut in place of per-certificate block lists, and would never accept an unlogged forgery.

MTC does not fix key generation. A CA key with a private value of 4 is just as forgeable in a Merkle tree, and mirrors only make the forged entries visible. It also does nothing for the clients exposed here, because they are the ones that will never adopt it. Chrome plans an MTC-only quantum-resistant root store for the third quarter of 2027, alongside the existing one. Removing non-MTC CAs from the Chrome Root Store appears only as a second-half 2028 target on a slide titled "An (optimistic) timeline", and that slide notes no impact on user-added roots or private PKI.

When the browsers that fund root-program oversight stop depending on classical CAs, those roots will stay in frozen trust stores. The path through emSign ECC Root CA - G3 stays valid until 2039 in any store that keeps it. Classical WebPKI could outlive the people supervising it, and the eMudhra inclusion review shows how thin that supervision already is. A device that will run for fifteen years should trust a private root its operator controls and can rotate, with a managed path for trust updates. I covered the MTC side of this in From Chains to Trees and The Post-Quantum WebPKI.

What we still don't know

When eMudhra's staff first realized each CA had two keys, and who decided to sign with the certified ones instead of halting. eMudhra says it re-signed the CRLs with the flawed keys, so someone made that choice. Chrome asked both questions in the bug on 21 October.
What the wrap and unwrap script actually called. The mechanism, its parameters and the C_UnwrapKey template would show how a 384-bit key came out as a single byte.
Who audited the ceremony period, and what they tested. The June 2024 to May 2025 report covers 30 April 2025. I have not found it. Nor have I found an auditor's key generation report for that date, which for affiliate sub-CAs is optional.
What BDO did with Bug 1970728. Its report says it examined the root cause. Whether that included reconciling HSM keys to certificates is not public.
Whether Sectigo tested anything when it cross-signed. Its policy for external cross-certification has not been discussed in the bug. Chrome has asked whether Sectigo or any root program was told in 2025 that the cross-certified keys were not the intended ones.
Why the A series was revoked and replaced on 29 May 2025, the same day as the bad CRLs. Chrome has asked. eMudhra says the A and A-1 series were generated on Entrust HSMs for use in India, and Mozilla found none of their keys recoverable.
Which keys went to the European and US data centers. The B and C series were generated for export. If the strong keys were the ones shipped, the regional deployments held keys that matched no certificate and could never have produced a valid signature. That would mean the export path was never tested end to end.
What signed the bad root CRL on 14 October 2026, and whether other eMudhra roots have more than one key pair. Chrome has asked both.
What Apple and Microsoft will do with the remaining eMudhra roots, and whether Microsoft acts on the code-signing roots.
Whether anyone used the keys before Wayne. HSM logs cannot answer this, because an attacker never needed the HSM.

Who runs the check

A CA has one job. It holds keys that only it can use, and it knows which keys those are. Ceremonies go wrong, and the script bug is not the failure that matters most. The failure is that eMudhra had cryptographic proof in June 2025 that its certificates did not hold the keys it thought they held, resolved the discrepancy in favor of the certificates, and kept going.

Every layer of WebPKI oversight did what it was designed to do. Audits examined controls. Root programs examined disclosures and incident reports. CT examined issuance. Formal key validation examined curve membership. None of them examined the keys, and nothing required any of them to. We have become good at documenting that controls ran. We are much worse at checking what those controls produced.

The fix is cheap enough that the absence of an owner is the only obstacle. CCADB already receives every CA certificate within seven days of creation. Running paranoid_crypto, or the twenty lines it took me to scan for small scalars, at upload time costs less than a second for the whole corpus. Run it on the certificate, again on any cross-certificate of the same key, and publish the results. Make a failure block disclosure until the CA explains it. Root programs could require it in a policy update, and CCADB could run it without one. They could also require what one commenter in the bug proposed, a signed attestation from the HSM manufacturer that each CA key was generated inside the module.

This is not over. The bug is still open, eMudhra has promised root programs a full root-cause report, and none of Chrome's twelve questions has been answered. I am sure we will learn more, and I will update this post as we do.

Wayne found these keys with a free tool and twenty-seven minutes. The rest of the system had seventeen months, a public incident on the same six certificates, two audit reports and an inclusion review. Plenty of people could have caught it. Nobody was assigned to.

Sources

Mozilla Bugzilla: 2083449 (compromise), 1970728 (2025 CRLs), 1970727, 1965559, 1999241, 2084101, 1957685, 1931683, 2083709
dev-security-policy: incident and operator guidance thread, eMudhra inclusion thread
Google: incident post, quantum-safe HTTPS plan, PQWS slides, paranoid_crypto
Community: we-have-a-problem.com (proof, patch list, workarounds), connection statistics
Audits: BDO WebTrust for CAs, June 2025 to May 2026, BDO WebTrust BR
Policies: Baseline Requirements, Mozilla Root Store Policy, Chrome Root Program, Apple Root Program, Microsoft Trusted Root Program
Patches: Go 1.27.3 and 1.26.10, NSS 3.131.1, certifi 2026.10.19, webpki-roots 1.0.10, Node.js v24.22.1, Debian, Ubuntu
Data: CCADB resources (certificate records and PEMs), crt.sh, ctlogs.dev, Cloudflare's Azul commit, Raio shard closure, impacted certificates and revocation dates
draft-ietf-plants-merkle-tree-certs, Mozilla Security Blog post
eMudhra: incident statement, exit announcement

Key recovery, CRL key recovery and the CCADB scans are my own analysis of the public artifacts listed above.

Rejection Is Not a Verdict

Tue, 15 Sep 2026 19:31:09 GMT

Life offers an almost unlimited supply of chances to be rejected, and it is easy to take each one personally.

Picture a young man who works up the nerve to cross a room, tell a woman he finds her beautiful, and ask for her number. She says no, and he walks away believing she looked him over, weighed what she saw, and found it lacking.

That may not be what happened at all. She may have slept badly, or been running late, or just ended a relationship, or be seeing someone already. He may have reminded her of someone she can’t stand. She may simply not have wanted to meet anyone that afternoon. If he approached the same woman a year later and said nearly the same words, he might get a very different answer, and nothing about him would need to change for that to happen. The timing would be different, and so would her circumstances and priorities.

A rejection arrives as a yes or a no, which makes it feel like a clean judgment. The decision behind it is anything but clean. It is an equation with many variables, most of which you can’t see and many of which have nothing to do with you.

That is true of nearly every rejection you will face, whether it comes from an employer, an investor, a customer, an editor, or an admissions committee, and learning to read rejection accurately is a large part of what long-term success requires. Internships make a good example because they are where most people first meet rejection in large numbers, usually before they have learned how to interpret it.

What the rejections seem to say

You apply for an internship and get turned down. You apply for another and get turned down again. After twenty of these it is natural to conclude that nobody wants you, and after fifty that conclusion can harden into a belief that you just aren’t good enough.

Consider what is happening on the other side of those applications. For the student, landing an internship may be the most important thing in their life that year. For the company, filling it may sit somewhere around number 87 on someone’s list. Internships serve real purposes. They build a pipeline of future hires, give junior employees practice at mentoring, bring in fresh ideas, and let successful people and organizations give something back. They also cost time. Someone has to read the applications, run the interviews, scope a project an intern can actually finish, and answer that intern’s questions all summer.

So the person whose future seems to hang on the decision is often dealing with someone for whom that decision is one small part of a much larger job. That alone injects a great deal of randomness into the outcome, and it is only one variable among many.

Equally capable, different outcomes

Imagine two applicants who are, by any reasonable measure, equally capable. One lives ten minutes from the office and the other would have to move across the country. The local candidate gets the offer, and nobody involved has concluded that the other person is worse.

The two could also differ only in when they applied. One sends an application the week the posting goes up, and the other sends one a month later, after the hiring manager already has a shortlist and has stopped reading new applications closely. Applying early doesn’t settle the matter either, because the first applications can sit untouched until someone finds time for them, and whichever resume happens to be on top of the pile when the reviewer finally sits down gets the careful read. Neither applicant can see where they landed in that order, and the result says nothing about which of them is better.

Or imagine that one of two otherwise identical candidates spent more time on their resume. Instead of listing the name of a class project, they explain what they personally built, and their accomplishments are easy to follow. They get the interview even if they aren’t the stronger engineer, because they did a better job of showing what they had done.

Cover letters work the same way. Two letters can be equally well written while only one explains why this particular company interests the applicant and connects something the applicant has already done to the company’s work. That letter lands with the person reading it. Hand the same two letters to a different reader and the result could flip.

One candidate might figure out who leads the group they want to join, reach out on LinkedIn with a thoughtful question, and become a real person instead of another PDF in an applicant tracking system. Another might have an uncle who knows someone at the company and says, “You should at least talk to this kid.” Neither move guarantees anything, because the candidate still has to perform. But one application now gets ten minutes of attention while the other gets twenty seconds, and that difference can decide the outcome.

Differences that seem too small to matter

Suppose an aerospace company receives applications from two computer science students with similar grades, similar projects, and no prior internships. One of them happened to take private pilot ground school, and the hiring manager flies airplanes. It isn’t hard to guess which resume gets a second look.

The deciding detail could just as easily be a class project in computer vision when the group has an imaging project planned for the summer, or a university paper supervised by a professor the interviewer knows, or a personal project written in Rust at a company that builds in Rust. The hiring manager might have gone to UW and remember it fondly, or might hold an irrational grudge against it. People make these decisions, and people bring their interests, associations, biases, and moods along with them.

Sometimes the first filter isn’t the hiring manager at all. A recruiter works from a checklist an engineering manager handed over, without understanding the technology behind it. Your experience is relevant, but you described it in different terms, so you are screened out. Another candidate used the exact words on the checklist and gets an interview. That result says a lot about the screening process and very little about the relative potential of the two applicants.

The variables that belong to you

I keep the Serenity Prayer on my desk. Whatever you make of its religious framing, its central idea is practical. Learn to tell the things you can change from the things you can’t, and put your effort into the first group.

You have no say over who reviews your resume, how much time they have to spend on it, or where your application sits in the pile when they start reading. You can’t choose who else applies, including the CEO’s neighbor’s daughter. You can’t stop a company from losing a contract and freezing hiring the week before your interview, and you can’t pick where the hiring manager went to school.

You do have a say in whether your resume communicates clearly, and in whether you apply soon after a posting opens instead of the day before it closes. You can research the company and learn enough about its business to explain why you care about it. You can build things, write about what you built, and develop interests that make you worth talking to. You can contact people directly instead of relying entirely on an applicant tracking system, and you can ask family, professors, and friends whether anyone in their networks works in the fields that interest you.

You can also make it obvious that you have a direction. Compare a student who says, “I need an internship this summer,” with one who says, “I want to work on robotics and control systems. I’ve enjoyed building autonomous rovers in a campus club, I’m interested in how machines make decisions from sensor data, and I’d like to spend the summer somewhere I can see how those systems are built in the real world.” The second student may have no more experience than the first, but they have given the reader a reason to picture them inside the organization.

Rejection will still happen

You can do every one of those things well and still be turned down, sometimes over and over. You might apply to a hundred internships and hear no from ninety-nine of them, but you never needed a hundred internships. You needed one.

Those ninety-nine rejections don’t add up to a verdict on your worth. Each was a separate decision, produced by its own mix of timing, circumstance, competition, priorities, presentation, relationships, luck, and ability. Ability matters enormously over a career, but in any single decision it is just one of the variables.

A long run of rejections can still tell you something, because a few variables appear in every one of those decisions. Your resume, the roles you chose to pursue, and the way you describe your work went into all ninety-nine. When the same result keeps coming back, those shared variables are the first place to look, and they happen to be the ones you can change.

Internships are only the first place most people meet rejection in volume. Later it arrives as a job that goes to another candidate, an investor who passes on a pitch, a customer who picks a competitor, an editor who declines a manuscript, or a promotion that goes to a colleague. The equation is the same in each case, with many variables you can’t see, a few that travel with you, and a single yes or no at the end. People who build something meaningful usually collect far more rejections along the way than anyone else sees, and what distinguishes them is how they respond. They adjust the variables that belong to them and keep going, long after others have read the same results as a verdict and stopped.

My father likes to ask why God made men so stupid, and then he answers his own question. It was so they could do the impossible. He means the particular bravado of young men, which is often misplaced but sometimes useful. A young man who doesn’t appreciate how long his odds are will still cross the room, make the call, and send the application. Every so often that persistence pays off precisely because he never stopped to calculate how unlikely success was. As we get older, many of us get better at calculating the odds and worse at ignoring them. We start treating each rejection as a judgment about ourselves, and eventually as a reason to stop trying. Understanding what a rejection actually tells us lets us keep the persistence without depending on the ignorance.

Rejection will still hurt, because you wanted the thing you didn’t get. The useful response is a narrower question than “What is wrong with me?” Ask which variables in the equation belong to you, work on those, and then try again.

The Amnesia Cycle and Why AI Is Turning Developers Back Into Testers

Wed, 19 Aug 2026 17:34:06 GMT

I started working in technology around 1993. One of my first jobs was in quality assurance, partly because there was no security profession to join yet.

There were people doing the work, but few companies were hiring for it. That changed within a decade. Until it did, people with the instincts that would later define security engineering landed in adjacent disciplines. Test was one of them.

That was true for me. I became a test manager fairly quickly, later worked as a test architect, then went on to software development, security, and a bunch of other things.

More than thirty years later, I find myself watching something funny happen.

AI is turning software developers back into testers.

Not the kind of testers we were in 1993. What it means to test software has changed several times since then. But at a more abstract level, the work is surprisingly familiar.

The person is no longer primarily producing the thing. They are increasingly trying to determine whether the thing that was produced is any good.

And we have been around this loop before.

We develop a specialty because a problem is hard. We get good enough at it to encode pieces of the expertise into process, tools, and automation. Eventually the machinery works well enough that the underlying expertise starts to look unnecessary. We distribute the responsibility, automate more of it, and convince ourselves that the problem has largely been solved.

Then the system grows, the environment changes, or a new technology arrives and exposes all of the judgment that never made it into the machinery.

The responsibility never left. It just changed costume.

When testing was a profession

In the 1990s, software development and software testing were much more clearly separated. Developers wrote software. Test organizations tried to figure out where it broke.

A lot of the work was manual. People installed builds, exercised features, constructed strange states, tested boundaries, wrote bug reports, and tried to reproduce failures. By modern standards, much of it would look labor-intensive. Some of it really was scut work.

But the separation had an important organizational property. The person who built the thing and the person whose job was to find out why it was wrong were different people.

Their incentives were different too. Developers were trying to make the product work and get it shipped. Testers were rewarded for finding the circumstances under which it did not work.

Those are complementary responsibilities, but they are not the same responsibility.

Over time, testing acquired a status problem. It was increasingly treated as work that did not require the same level of engineering skill as implementation. One response was to give more of it to junior developers.

Then we moved further. Instead of a separate organization owning quality, developers would test their own software.

Much of that change was good.

Unit testing was good. Test-driven development was good. Continuous integration was good. Automated regression testing was good. Testing closer to the point where software was written eliminated entire classes of expensive downstream failures.

The nature of testing changed too. It moved away from a model dominated by manually exercising a finished product and toward one where tests could become part of the way software itself was specified and constructed.

So the story is not that we eliminated QA and that was simply a mistake.

The mistake was gradually convincing ourselves that because we could automate more of the mechanics, we needed less of the judgment.

A test that passes is not evidence that the system is good

As automation improved, we became extraordinarily good at running tests.

A modern software project can execute tens of thousands of tests on every change. We can measure coverage, reject regressions, test configurations, fuzz interfaces, spin up entire environments, and tear them down again without anybody touching them.

The machinery works.

But there is a distinction that became easier to overlook.

A test can work perfectly and still tell you almost nothing useful about the quality of the system.

The hard question is not always whether the test passed. It is whether passing that test is evidence of the property you actually care about.

That gets harder as systems become more complicated. A system can have an enormous green test suite while failing in a way nobody thought to represent in the suite.

There is an important difference between mechanical verification and judgment about what deserves to be verified. One asks whether the checks we wrote passed. The other asks whether those were the right checks.

Automation became exceptionally good at answering the first question. It did much less to eliminate the difficulty of the second.

A test suite is an encoding of somebody's model of how the system can fail. It captures the failures we anticipated, the properties we chose to represent, and the assumptions we knew enough to challenge.

It says much less about the failures nobody imagined.

Automation did not eliminate the test problem. It moved the test problem up a level.

The scarce skill became figuring out what to test, what failure looks like, which assumptions need to be challenged, and what evidence should actually make us confident in the result.

That is why what is happening with AI feels so familiar.

We thought AI would do the testing

One of the obvious expectations around generative AI was that it would automate still more testing.

If the AI can write the implementation, it can certainly write unit tests too. And it can.

But that misses the more important change.

AI is making implementation cheap.

A developer can already cause far more code to be produced than they could reasonably have written themselves. As agents improve, that multiplier gets larger. It is not difficult to imagine one engineer directing dozens, hundreds, or eventually thousands of concurrent software-producing processes.

At that point, traditional code review is not merely inefficient. It becomes physically impossible.

Nobody is going to carefully read every line produced by a thousand coding agents.

The generated output is also not quite like the output of an old deterministic compiler or code generator. These systems are probabilistic. Run them again and you may get a different implementation, a different decomposition, or a different mistake.

That makes the old assumption that we can inspect the artifact into correctness even less plausible.

So the human moves up a level.

Instead of spending most of the time constructing the implementation, we increasingly construct the conditions under which an implementation will be accepted. We write tests, evals, invariants, acceptance criteria, adversarial cases, and constraints. We decide what evidence is sufficient to tell us that the machine-produced result is actually good.

Software developers are becoming testers again.

It is not 1993-style manual QA. It is closer to specification, TDD, evaluation design, systems validation, and the construction of executable evidence.

But abstractly, it is the same work happening in a different way.

The form changes. The responsibility does not.

What the new testing actually looks like

The easy question is whether generated code passes the existing test suite.

The harder question is whether the suite represents the properties we actually care about.

Does the generated component behave correctly under inputs we did not anticipate? What happens when independently generated components interact and two locally correct decisions compose into a globally bad result? What happens after the system has been operating for days, accumulating state and acting on the consequences of its own earlier decisions?

What happens when the environment differs from the conditions represented in our evals? And what happens when an adversary deliberately searches for the assumptions we failed to encode?

Those questions are not answered by producing more tests mechanically.

They require somebody to form a theory of failure.

The scarce skill is increasingly understanding the mechanics of failure. That means knowing where the system boundaries are, which properties must remain true across those boundaries, what assumptions are hidden inside the architecture, and how reasonable local behavior can produce unreasonable global outcomes.

It also changes what a useful test looks like.

When implementations are relatively stable, testing specific examples can tell you a lot. When an AI can regenerate the implementation tomorrow, durable properties become more important. The question shifts from whether this implementation produces the expected result for this input toward which properties must remain true across whatever implementations the system produces.

Those properties might concern correctness, authority, state transitions, isolation, safety, or the relationship between components. The implementation can change while the invariant remains.

That is a different kind of leverage.

Verification also stops neatly ending at release. Some failures only emerge through interaction with real environments, long-running state, changing inputs, or behavior that was not represented during development. That pushes part of the evidence gathering into operation through telemetry, observability, runtime checks, and the behavior of the deployed system itself.

The new tester is therefore not just checking whether an implementation conforms to a specification somebody else already wrote.

Increasingly, they are responsible for deciding what the specification must say, which properties must survive implementation changes, which failures matter enough to detect, and what evidence is sufficient for the resulting system to deserve trust.

That is a substantially higher-order form of the same old responsibility.

High-quality scut work

There is a funny status inversion hiding in this.

Test used to be treated as scut work. Then we gave more of it to junior developers. Then we told every developer they were responsible for doing it themselves. Then we automated as much of it as we could.

Now we are automating the part we used to think was the prestige work, writing the software, and leaving the developer sitting above the machinery trying to determine whether any of what it produced is good.

We may have turned one of the industry's prestige jobs into very high-quality scut work.

The funny part is that the scut work may now be where much of the value lives.

When implementation is expensive, the person who knows how to implement something is scarce. When implementation becomes cheap, the scarce person is the one who knows what should be built, what properties it needs to have, how it is likely to fail, what assumptions are hidden inside it, and what evidence would convince us that it works.

Generation gets cheaper. Judgment does not.

Security already did this

I have seen almost the same cycle happen with security.

In retrospect, it is probably not an accident that so many early security people came through test. Both disciplines train you to look at a system somebody else believes works and ask what they have failed to consider.

Testing asks how the behavior can violate what was intended. Security asks how trust, authority, or assumptions can be violated even when the system appears to be functioning normally.

Both reward a certain kind of skepticism.

When I started, security engineering barely existed as a normal software profession. By the late 1990s it was becoming one. By the early 2000s, it was clearly a distinct industry with dedicated teams and career paths.

That specialization happened because security was hard and ordinary development organizations were not consistently good at it.

Then we started saying something that was also fundamentally correct. Security should not be something another group bolts onto the product afterward. Developers should build secure systems themselves.

"Security is everyone's responsibility."

There is nothing wrong with that principle.

The problem is what happens when we confuse responsibility with expertise.

A product developer is trying to ship a product. They have schedules, features, performance requirements, compatibility issues, reliability problems, customer demands, and dozens of other things competing for attention.

Security becomes one of many things they are supposed to get right.

When that proves insufficient, organizations build machinery around the problem. We add secure development lifecycle processes, scanners, policy gates, paved-road platforms, and controls designed to make the safe thing easier than the unsafe thing.

All of those things can help.

But notice what we are doing.

After deciding the specialist function should be distributed into the rest of engineering, we are encoding pieces of that specialist judgment back into systems and processes.

A scanner encodes somebody's knowledge of what a vulnerability looks like. A secure-by-default platform encodes somebody's judgment about which choices should be permitted. A policy gate encodes somebody's model of what conditions need to hold before software should be released.

The specialist may become less visible as that expertise gets embedded into the platform, but the expertise did not cease to exist. It became infrastructure.

Eventually there is enough machinery that it again becomes tempting to ask whether we really need the specialists.

Then the abstraction leaks.

A threat appears outside the model encoded in the scanner. A platform assumption no longer holds. A new system does something the existing governance framework was never designed to reason about.

Then we discover that we automated the known answers, not the ability to recognize new questions.

AI security and the latest rediscovery

AI makes this pattern almost comical because we are currently rediscovering old classes of security problems with new names.

Prompt injection is obviously not literally SQL injection. The implementation is different, the interpreter is different, and the failure modes are different.

But someone who spent the 1990s and 2000s dealing with SQL injection, command injection, script injection, confused deputies, trust boundaries, privilege separation, and the consequences of letting untrusted input become control should find the family resemblance hard to miss.

We have spent decades learning that you should be very careful when information from an untrusted party can influence what a privileged system interprets as instructions.

Now we have built enormously capable interpreters whose primary interface is natural language. We mix instructions and data in the same context, connect them to tools, and act surprised when hostile input changes what they do.

The technology is new. The institutional failure mode is not.

The same thing is happening in conversations about containment and sandboxing. We are once again discovering that powerful systems need boundaries, that those boundaries need to be enforced rather than merely described, and that capabilities should be constrained by something stronger than an instruction asking the system to behave.

None of that makes AI security trivial. The new systems create genuinely new problems. But novelty at one layer does not erase the accumulated lessons at another.

We knew versions of these things thirty years ago. What keeps recurring is not the exact vulnerability. It is the belief that a new abstraction has somehow relieved us of the old responsibility.

That is where the amnesia comes in.

The amnesia cycle

You could describe the pattern as specialization, codification, automation, perceived redundancy, loss of judgment, scaling failure, and rediscovery.

That is the amnesia.

We rarely forget the artifacts of the previous generation. We keep the test frameworks, scanners, development processes, controls, and automation.

What we forget is why the people who created those things thought the problem was hard in the first place.

Eventually we confuse the existence of the machinery with possession of the expertise that created it.

A passing test suite becomes evidence of quality. A security scanner becomes evidence of security. An AI eval becomes evidence that the AI is doing the right thing.

Until the system moves outside the assumptions those mechanisms encode.

Then the old problem appears again, wearing different clothes.

The responsibility never leaves

That is the through line I see between testing, security, and what is now happening with AI-assisted development.

We are very good at abstracting away mechanics. That is what engineering does.

But when we successfully automate the mechanics, it is easy to convince ourselves that we automated the underlying responsibility too.

We did not.

Testing did not disappear when the dedicated QA organization disappeared. Security did not disappear when we made it everyone's responsibility. Verification will not disappear because an AI can generate both an implementation and a test suite that says the implementation is fine.

Somebody still has to decide what "fine" means. Somebody has to recognize the assumptions the automation does not know it is making. Somebody has to decide which failures matter. Somebody has to determine what evidence would falsify the claim that the system is working.

Somebody has to distinguish a system that successfully passes its tests from one that deserves to be trusted.

The responsibility never leaves. It changes costume.

I started my career in a world where developers wrote the software and people like me tested it. We spent the next thirty years treating test first as work developers should not have to do, then as work developers should do themselves, and finally as work machines should increasingly do for them.

Now the machines are starting to write the software.

And the developers are increasingly responsible for figuring out whether any of it is good.

Apparently the tester won.

Hurst University

Mon, 17 Aug 2026 17:33:41 GMT

For as long as my children can remember, I have told them that they are students at Hurst University.

It has no campus, no accreditation, and no admissions process. Joining the family is enough to get you enrolled. Graduation is another matter.

There is only one requirement. By the time you leave the house, you should be capable of building a future for yourself.

I don't mean that you should know what you are going to do for the rest of your life. That is one of the stranger questions we ask young people. Most adults I know have changed direction enough times that pretending an eighteen-year-old is making a permanent occupational choice is unserious.

What I mean is more fundamental. You should know how to learn, and you should know how to work. When you encounter something you don't understand, you should have some idea how to get from ignorance to competence. You should know how to take a hit without deciding the hit defines you, how to recognize when your assumptions were wrong, and how to change direction without treating everything behind you as wasted.

Most importantly, you should increasingly understand that the responsibility for what happens next belongs to you.

That is Hurst University.

There is nothing new about the idea. If anything, it is an old model of education with a family name attached to it.

For most of human history, becoming educated and becoming useful were close to the same thing. Children watched adults do real work. Then they helped. Then they were trusted with some small part of it. They made mistakes where mistakes were survivable, were corrected by somebody who knew more than they did, and tried again. As competence increased, the work got harder and supervision got lighter. Eventually the person who had been taught became somebody who could be trusted to act without waiting to be told what to do next.

That transfer is the part I care about most. The student eventually has to become responsible for the education.

I learned that the hard way, and then I had to watch one of my children learn it too.

He was not slow. He was seeking out serious material for pleasure at an age when adults found it surprising. But the way he took in information did not match the way the school delivered it, and the instruments the school trusted returned the wrong answer about him. The system then acted on the wrong answer. He began to notice he was being handled as a different kind of student, and he could not work out why.

The part I remember is not the meetings. It is that he started to wonder whether the school knew something about him that he didn't.

I eventually moved him somewhere more willing to respond to an individual child, which helped. The more important intervention was a conversation.

I told him what I believed to be true. No institution was ever going to be able to take full responsibility for his education. Some things were going to be harder for him than for other people, and that was not going to change.

Then I told him that his challenges were his superpower.

I meant it literally, and I explained why. Most people are never forced to learn how they learn. They get through on the method they were handed, and they only discover its limits much later, if ever. He was going to have to build his own method starting now, at nine, because the handed one did not work for him. That is a brutal assignment and it is also an enormous head start. The world keeps rewarding people who can acquire what they need without being given it, and he was going to be practicing that while everyone else was still being taught.

Then I told him he was a student at Hurst University, and that when he graduated from this house he was going to be fine.

He is grown now. In his twenties he built a business and sold it to one of the largest financial services companies in the world, which is a thing you cannot do without becoming a fast and relentless student of whatever is in front of you. Entrepreneurs are not people who already know how. They are people who find out in time.

He is one of the sharpest people I know and one of the most prepared, and the second of those is the one he built. He worked out early that being the smartest person in the room and being ready for the room are different things, and that only one of them was under his control. He stopped waiting to find out which he would get. That habit came out of the friction, and I am not sure anything else would have produced it.

I was a student at Hurst University long before I had a name for it.

I was dyslexic and dysgraphic and did not present well to the educational system. At one point my parents were told they should prepare themselves for the possibility that I would never be capable of supporting myself.

That prediction did not age well.

I went to college young and moved out at sixteen. Some of that was rebellion, but mostly I wanted independence in the literal sense. I wanted control over my own life, including the economic responsibility that came with it.

A lot of my education after that happened without anybody designing it. Computers gave me problems I cared enough about to solve. Programming led into systems. Systems led into networking. Networking led into security. Security eventually required understanding companies, incentives, law, economics, organizations, and people. I kept encountering things I did not know and learning enough to get through the next door.

One capability created a reason to acquire another.

None of it started with me.

My father grew up on a subsistence farm, where you fix what breaks with what is in the barn because the alternative is doing without. He taught himself rocket chemistry as a kid off that same principle and was working on satellite hardware by his early twenties. I have written about him before, so I will not tell it twice.

What matters here is that he never called any of it an education. It was just what you did when you needed to know something.

So I did not invent this. I inherited it, gave it a name, and made the handoff deliberate. That last part is the part that matters, because none of it moves on its own. It has to be handed over on purpose, by somebody who decides to bother.

Years ago, when I wrote about apprenticeship, I described four things that had mattered enormously in my own development: access, direction, challenges, and support. I still think that framework is right, but I now see something underneath it. Those are not merely the ingredients of a good apprenticeship. They are the ingredients of an environment that gradually teaches someone to direct themselves.

Give somebody access to things worth learning. Put people around them who know more than they do. Give them problems slightly beyond their current ability. Support them enough that failure remains recoverable. Then, slowly, stop telling them what to do next.

That last part is the actual transfer.

Having three children made this clearer, because the same philosophy looks completely different depending on the student.

My second child has always worked. At four he was doing long division and by 6 or 7 he could recite the major bone and muscle groups. As a teenager he became a nationally ranked fencer, which is not something that happens to a person for being quick. It happens through years of drilling the same movement badly until it is good, losing in front of people, and going back the next day.

So the story I used to tell myself about him, that things came easily and he had therefore never built the habit, does not survive the evidence. He built it early. He built it in a domain nobody assigned him.

What was true is that school rarely asked him for it. He was admitted to several of the best aerospace programs in the world and ultimately chose computer engineering, and he is now somewhere the standard is set by people who are also very good and where thinking fast is the baseline rather than the edge.

So what I am watching is not a young man learning to work. It is someone moving a work ethic he already owns out of the place he first built it and into the place he intends to live. That transfer is the entire point of this essay, and he is doing it in front of me.

My third is different again. Nearly all of her identity is currently organized around a single competitive sport, roughly twenty-six hours a week of it. That is not a complaint. Twenty-six hours a week of anything difficult teaches repetition, correction, pain tolerance, delayed gratification, and performing while people watch. The work ethic already exists.

The problem is that she believes the work ethic belongs to the sport. Her brother has already shown that it doesn't, which is the most useful thing an older sibling can do.

Activities end. The machinery that produced the excellence should travel.

Aptitude changes the educational problem. It does not remove the educational problem.

A child who struggles may need to learn that difficulty is not the same as inability. A child who rarely struggles may need enough friction to discover the value of preparation. A child who becomes excellent in one domain may need to discover that excellence is partly a process that can be carried elsewhere.

The curriculum changes because the student changes. The goal does not.

The goal is agency. By agency I mean the ability to encounter something unfamiliar and say, with some credibility, "I don't know how to do this yet, but I know how to begin."

That may be the most durable thing an education can produce.

My father was not the only one. My mother was doing the same thing in a different register, and I was even slower to see it.

She started as a hair stylist. Later she became a tool-and-die worker at Boeing. Then she moved into knowledge work and eventually retired as a business analyst.

Described by job title, those look like unrelated careers, the sort of résumé that gets read as drift. Described by behavior, they are the same story repeated several times. She kept becoming qualified to do things she had not previously known how to do.

Nothing about cutting hair prepares you to hold tolerances on a machined part. Nothing about machining prepares you to take apart how a business actually works and put it back together as a requirement. What carried across was not the content. It was the practice of arriving somewhere without the necessary knowledge and acquiring it in public, in front of people who already had it, while the work still had to get done.

She did that at least three times, each time later in life than the last, each time with more to lose. Watching it happen taught me more than any explanation of it would have.

I think we put far too much weight on occupational identity. We ask people what they "are" when what we mean is what they are being paid to do right now. Those are not the same thing.

A mechanic who becomes an engineer does not arrive empty-handed. A construction worker who moves into software already understands sequencing, dependencies, tolerances, physical constraints, customers, mistakes, and what happens when plans encounter reality. A salesperson who becomes a product leader knows things about incentives and people that do not appear in a product-management textbook.

Learning one serious domain teaches you more than the facts of that domain. It teaches you something about systems, and, if you are paying attention, something about how you yourself become competent.

That is why the question "What are you going to do with your life?" is not useful. Ask instead what you are going to do next.

You are not choosing forever. You are choosing next.

That does not make the decision unimportant. It changes what makes the decision good. A good next step should leave you with more than you had before. More competence, more judgment, more context, more relationships, more credibility, more capital, or simply more options.

Then you choose again.

This is also why I have never been comfortable with "follow your dreams" as career advice. Dreams are useful. They give us energy and they make us try things. But desire and strategy are not the same thing.

People have aptitudes whether we like that fact or not. The world has needs whether we like that fact or not. Some capabilities are scarce, some are common, and some interests map more naturally than others onto work that can support a life. If you expect something to support your life, understanding how it creates value is part of taking responsibility for the decision.

Years ago I was riding the gondola between the peaks at Whistler with three young women. One of them had recently graduated and was telling the others about an argument she had had with her boss. She had a degree now, and she thought she should be paid more. Her boss told her he could not pay her more simply because she had acquired the degree.

She was furious. Why had she bothered getting it?

Then, somewhere in the conversation, she mentioned that she worked at a sandwich shop.

I never learned what the degree was in. That is probably why the story stuck with me. The degree may have been enormously valuable, and may have opened an entirely different career a month later. But the credential itself had not changed the economic value of the work she was performing that afternoon.

The point is the distinction between learning something, possessing evidence that you learned something, and becoming capable of doing something the world values. Those things overlap, but they are not identical.

Eventually reality gets a vote.

That phrase matters to me because it applies well beyond credentials. You can believe you understand a system until you have to build one. You can believe you understand customers until you have to sell something to them. You can believe you understand leadership until somebody else's livelihood depends on your judgment. You can believe you understand risk until the decision is yours and the consequences arrive with it.

Capability develops when knowledge begins colliding with consequence.

That is why work, projects, apprenticeship, competition, and responsibility matter so much. They are not simply places to apply learning. They are part of how learning becomes judgment.

And judgment is difficult to acquire without being wrong.

My father used to ask, "Do you know why God made young men so stupid?"

"So they could do the impossible."

He usually said it about the early space program, and that was not a coincidence. The young men in the joke were him and the people he worked with. He was not describing a category of person. He was describing a room he had been in.

There is a limit to the principle. Ignorance can get you killed. But experience accumulates reasons a thing will not work, and there is a danger in becoming so sophisticated about risk that the sophistication becomes a reason never to take one.

If you are going to attempt difficult things, some of them will not work. What matters is what happens next.

Failure itself is not automatically useful. Failing repeatedly without changing anything is repetition. The useful part is the loop afterward. What happened, what assumption was wrong, what did I misunderstand, what was inside my control, and what should change next time?

The attempt failed. That is information.

"I am a failure" is something else.

Rejection works much the same way. Many worthwhile things require volunteering for outcomes somebody else partly controls. Apply for the job. Ask for the opportunity. Pitch the customer. Publish the idea. Start the company. Compete.

Somebody gets to say no.

If no is psychologically intolerable, you eventually begin designing your life so nobody ever gets the opportunity to say it. That feels safer, but it also shrinks the range of possible futures.

This is why resilience is less something you explain to a child than something you progressively load. An athlete does not begin with the maximum weight. A gymnast does not begin with the hardest skill. You give somebody difficulty at a scale they can survive, let reality push back, help them understand what happened, and then load a little more.

Over time they accumulate evidence about themselves. Discomfort ends. Embarrassment is survivable. Criticism can contain useful information. Preparation changes outcomes. Being wrong does not destroy you. Failure can be followed by another attempt.

This is what I mean when I tell my children that the hardest thing in life is managing your own psychology. Intelligence does not save you from fear. Talent does not save you from insecurity. Being right does not save you from ego. Knowing what you ought to do does not guarantee that you will do it.

A surprising amount of adulthood is remaining capable of acting while your own psychology is trying to convince you not to.

Which brings me to the message I worry about most, because it arrives sounding like sophistication.

Somewhere between the start of high school and the end of it, one of my children began telling me a story about his own future. Previous generations had taken the housing. Previous generations had taken the wages. The arithmetic of an ordinary adult life no longer worked, and there was not much point pretending otherwise.

I pushed back, and I want to be careful about what I was pushing back on. I was not arguing that housing is affordable or that the numbers are fine. Constraints are real, and a young person who cannot see them is not being educated, he is being flattered.

What I objected to was the shape of the conclusion. A structural fact had quietly become a personal verdict. He was not describing a difficult environment he would have to navigate. He was describing an outcome that had already been decided, which meant navigation was beside the point.

That is the mirror image of "follow your dreams," and it fails for the same reason. One says the world will accommodate you. The other says the world will not permit you. Both remove the part where what you actually do makes a difference.

Narratives shape agency. Give young people tools, not verdicts.

The same idea applies to the people and environments we choose.

I have told my children for years that you cannot aspire to what you have not seen or experienced. We like to think our imagination is independent. It isn't.

If you have never met anyone who built a company, building a company feels like something done by a different category of human being. Then you spend time around someone who has done it and the thing moves from abstract possibility into the set of things ordinary humans apparently do.

Spend time around engineers and engineering becomes concrete. Spend time around excellent tradespeople and craftsmanship becomes visible. See someone change careers later in life and reinvention becomes less frightening. Watch someone take a serious risk, fail, recover, and try again, and failure becomes less terminal.

This is part of what apprenticeship does extraordinarily well. The apprentice is not merely receiving instruction. The apprentice is watching what competent people consider normal.

And normal is contagious. So are standards. So are ambitions. So are fears.

Whether people like the implication or not, we become, in meaningful ways, like the people with whom we spend our time. Which is why I have come to think of the whole environment as three inputs worth choosing deliberately. People shape your standards. Problems shape your capabilities. Feedback shapes your calibration.

Those are decisions, and like any decision they are worth revisiting. I am not disciplined about this and I have stayed in lanes longer than I should have. But without some mechanism for reconsideration, inertia starts impersonating intention.

You do not need to predict your life. You do need to keep steering it.

All of this was already true before AI. AI makes one part of it harder to ignore.

Instruction is getting cheap and abundant. The judgment that used to accumulate as a byproduct of doing unimportant work is not. I have written about both of those elsewhere, the vanishing on-ramp and what turns scarce once reasoning is cheap, and will not argue them again here.

The piece that belongs in this essay is smaller and harder. Every one of those questions is a decision about what to do next, and there is no longer anyone obvious to hand it to. A model will tell you what is true. It will not decide what you are willing to own.

That does not require a new educational philosophy. It makes an old one newly relevant.

You need access. You need direction. You need problems that matter. You need people who know more than you do. You need enough consequence for reality to get a vote. And eventually you need responsibility for deciding what happens next.

Which brings me back to the thing Hurst University was actually built around.

Family.

When children are young, the family carries almost everything. Food, shelter, transportation, money, protection, opportunity, judgment, and most of the consequences of decisions all sit primarily with the parents.

Childhood is, in part, the gradual transfer of that weight. Not all at once. That would be abandonment. A little at a time, while mistakes are still cheap and there is somebody nearby who can help make sense of them.

At first we choose for them. Then we let them choose between things we have selected. Then they make decisions we would not have made. Then they live with some of the consequences. We advise more and decide less.

If the process works, responsibility moves almost imperceptibly from one side of the relationship to the other.

Eventually they leave the house. They do not leave the family.

That distinction is more important than I understood when I first started making the Hurst University joke. The goal was never independence in the sense of needing nobody. Families do not work that way, and neither does the rest of life. The goal was to change your position inside the family.

When you are small, the family carries you. For a long time that is its job. Then you begin carrying more of yourself. Your decisions become yours. Your mistakes become yours. Your work becomes yours. Your education becomes yours. Eventually the roof over your head becomes yours too.

And somewhere along the way, if things have gone well, another transition begins. You become capable of carrying some weight for the people who once carried all of yours.

That is much closer to what I mean by graduation.

Not that you know what you are going to do for the next forty years. Not that you have accumulated the correct credentials. Not that you have stopped needing your parents, your siblings, or anyone else.

You leave the house able to participate in the family as an adult. You can build enough of a life to carry yourself, and enough capability that when the people you love need something from you, you have something to give.

The graduate of Hurst University is not the person who has all the answers. It is the person who has become difficult to make helpless.

Thirty years ago I did not have all of this worked out. I had a joke about Hurst University and a conviction that my children needed to leave the house able to make a future for themselves.

I understand the joke a little better now.

The future was never the thing I could give them. The best I could do was help them become people capable of building one.

From Periodic Audit to Continuous Assurance

Thu, 06 Aug 2026 23:13:04 GMT

I have been writing about the limitations of audits and compliance systems for several years.

In Accountability and Transparency in Modern Systems, I wrote about systems producing evidence continuously rather than assembling it periodically for an auditor.

In First Principles for Root Store Management, I looked back at the decision to require WebTrust for publicly trusted CAs and argued that, if we were designing the system today, much more of the trust decision should be based on continuously verifiable behavior.

That led to The Limitations of Audits, Rethinking Compliance, and Compliance at the Speed of Code.

The common thread was that the systems we are trying to assure change much faster than the mechanisms we use to understand them.

Over the last year, I have spent considerably more time on this problem, both thinking about it and building systems intended to work differently. That work convinced me that the problem is deeper than periodicity alone.

I have pulled that thinking together into two new long-form pieces.

The Assurance Model Was Built for a World That No Longer Exists looks at how the modern assurance model developed, what it actually establishes, and five distinct ways the evidence available can fail to justify the conclusion people ultimately rely on.
Why Continuous Assurance Did Not Happen Until Now asks why decades of automation gave us continuous evidence without continuous reasoning, what AI changes in that equation, and what a different assurance architecture might look like.

They are intended to be read together.

The first explains how we got here.

The second explores what comes next.

From Chains to Trees

Tue, 04 Aug 2026 22:34:37 GMT

The WebPKI has two structures that are not the same shape.

One is a cryptographic graph of signed bindings. Public keys, names, entitlements, and the keys that authorized them. The other is a governance hierarchy of accountability. It explains why a relying party accepts that authority at all, and when it stops accepting it.

Nearly every interesting failure in the history of the system lives in the gap between them. Misissuance, compromise, distrust events, and the long struggle with revocation are all stories about that mismatch.

I wrote two long-form pieces that try to make the distinction legible.

The first walks through the classical system as it actually exists. What a certificate is, how trust is delegated, how root programs and policy actually work, and why the governance layer has always mattered more than the certificate chain itself.

A Deep Dive on the Classical WebPKI

The second examines the redesign now underway. Post-quantum signatures are simply too large for the classical model at public scale. The response is Merkle Tree Certificates. A CA logs certificates into its own tree, signs the tree head, and each certificate carries a short inclusion proof. One signature covers the batch. The proof is the path.

This is not merely a cryptographic migration. It is the ecosystem cashing in a forced upgrade to close a decade-old compromise in Certificate Transparency. Transparency stops being a post-issuance promise and becomes the issuance mechanism itself. The wire format changes. Most of the governance carries forward.

The Post-Quantum WebPKI

The Status Quo Outlived Its Status

Fri, 17 Jul 2026 14:40:54 GMT

In security we like to say that the problems live in the gaps between systems. Each system, on its own, is usually coherent. It has a threat model, invariants, and someone who owns it. The seam between two systems is owned by nobody, and each side quietly assumes the other is handling the thing that neither is. The load balancer assumes the backend validates. The parser assumes the canonicalizer normalized. The audit covers the software but not the network it runs on. Attackers don't have to beat either component. They just have to find the assumption neither side wrote down. The attacker gets to pick the threat model, and they pick the one that lives in the seam.

Once you see this pattern, you see it everywhere, and not just in security.

Quality Lives in the Gaps of Ownership

In software quality, the same topology produces a different failure class. Security gaps produce exploits. Ownership gaps produce jank.

A user's journey through a product is inherently horizontal. They sign up, configure, use, get billed, get help. But ownership is vertical, carved along team boundaries. So the experience degrades precisely at the handoffs. The onboarding flow owned by one team dumps you into a product owned by another, with terminology that doesn't match, settings that don't carry over, and an error message that references a concept from a third team's domain model. Every screen passed its own review. The journey never got one, because the journey has no owner.

This is the same root cause as the security version. Contracts between components are written in terms of what each side provides, not what the whole must feel like or withstand. Functionality composes. Quality attributes don't. Not security, not usability, not performance, not consistency. Those are emergent properties of the composition, and emergent properties are exactly what per-team accountability structures can't see. A dashboard that takes eight seconds to load is usually five services each meeting their SLO.

You Ship the Org Chart in N Dimensions

Conway's law is usually quoted as a statement about architecture, that organizations design systems which mirror their communication structures. But architecture is just the most legible projection. The org chart also manifests in the security posture, where trust boundaries land wherever the reporting lines do. It shows up in the latency profile, where every org boundary becomes a network hop plus a queue plus a retry policy. It shows up in the data model, where the same "customer" is defined four ways because four VPs own four systems. It shows up in the compliance scope, where audits map to cost centers rather than to where the risk actually lives. It even shows up in the documentation, where each team documents its interior and nobody documents the crossings.

You don't ship your org chart once. You ship it in every dimension at the same time.

And it's stickier than the chart on the wall, because the formal org chart is only the visible part. The real partitioning is cultural. It is who trusts whom, which teams have history, where the scar tissue from the last reorg sits, and who won the last budget fight. Systems calcify around those boundaries. That's why reorgs so rarely fix seam problems. You can redraw the chart in a day, but the shipped artifact embodies the org chart as it existed at every point in the system's history. Legacy code is really legacy org structure. You're maintaining the fossil record of decade-old turf wars, and the team that could explain a given seam disbanded three reorgs ago.

Bureaucracy Is the Fixative

Here's where it hardens. Process is how organizations serialize distrust between units. Every approval gate, every ticket queue, every review board is a treaty boundary between fiefdoms, and treaties optimize for non-aggression, not for the emergent properties of the whole.

The bureaucratic instinct when a seam fails is to add process at the seam. A checklist, a sign-off, a form. This papers over the gap without giving it an owner, and now the seam has a compliance artifact defending its existence.

Which brings us to the uncomfortable part. Bureaucracy defends the status quo long past the point where the status quo lost its status. Not out of malice, and usually not even out of preference. The mechanism is simpler and more forgivable than that. Process is memory without comprehension.

Every rule is a compressed lesson. Some incident happened, someone got burned, a control was born. But the compression is lossy. The rule survives while the context doesn't. The organization keeps executing the answer long after everyone who understood the question is gone. It's Chesterton's fence, except nobody can find the fence's author, the field it enclosed is now a parking lot, and there's a Fence Compliance team whose headcount depends on the fence.

That's the inversion point. Controls that began as instruments become constituencies. A process accretes staff, tooling, budget, an annual review cycle. It stops being a means and becomes a stakeholder. And stakeholders defend themselves.

Asymmetric Bookkeeping

The genius of bureaucratic self-defense is that it never has to argue the status quo is good. It only has to make change expensive.

Every proposal to remove a control gets evaluated by asking what risk removal creates. Nobody asks what risk retention creates. The cost of the existing process is denominated in currencies the review process can't count, things like velocity, morale, and opportunities that quietly went elsewhere, while the cost of change is denominated in the one currency it's built to count. With bookkeeping that asymmetric, the ratchet only turns one way.

There's a reliable tell for when status is lost but the defense continues. The justifications go circular. Ask why we do this and the answer stops referencing a threat or an outcome and starts referencing the process itself. It's required for the audit. It's policy. That's the template. When a control's referent is another control, you are no longer managing risk. You are maintaining a liturgy. And liturgies are stable. That's what they're for.

The people defending the liturgy usually aren't cynics, either. Institutions promote the people who thrived under the current rules, which means the people with the authority to change the system are precisely the ones whose careers validate it. They don't defend the status quo because they've weighed it and found it good. They defend it because it's the ladder they climbed, and it's genuinely hard to see your own ladder as arbitrary. The system doesn't need guards. It manufactures believers. That's why correction so often comes from outside, from a competitor, a collapse, or a technology that routes around the institution entirely, rather than from reform. Reform requires the institution to metabolize the idea that its own selection function is the problem, which is roughly asking the liturgy to audit itself.

Deletion Has No Constituency

So what do you do about it? Two things, and both are harder than they sound.

First, admit that the seams are the system, and staff them accordingly. You can't fix an n-dimensional Conway problem with a one-dimensional intervention. A design system fixes the UX projection. A service mesh fixes the network projection. A GRC tool fixes the audit projection. But the generator is the accountability topology itself, so the pathology just re-expresses in whatever dimension you didn't treat. The only durable moves are changing the human topology, which is rare, painful, and temporary, or forcing the interfaces to be explicit, adversarially specified, and owned as products. That's the real lesson of the Amazon API mandate. It wasn't about services. It was about giving the gaps owners.

Second, build a decay function. Controls get created by incidents, which are vivid and have advocates. Control removal has no incident, no advocate, no ceremony. The beneficiary of deletion is diffuse while the loser is a specific person in the room. So organizations accumulate process the way arteries accumulate plaque, one reasonable deposit at a time. Sunset clauses, zero-based process reviews, and deletion treated as a first-class ritual with the same ceremony as launch are ideas everyone nods at and almost nobody funds. The organizations that stay fast are the ones that treat removing a rule as an achievement, not an admission.

Because the status quo isn't defended because it won an argument. It's defended because it's the null hypothesis, and the burden of proof only ever runs one direction. Every so often, you have to flip the burden and make the process re-justify itself in terms of an outcome rather than another process. If it can't, it isn't protecting you anymore.

It's just protecting itself.

Why FIPS 140 Means Running Old Code

Mon, 13 Jul 2026 15:33:16 GMT

You need to use FIPS 140 because of compliance, but have you ever asked what that requirement is actually for? What security properties are the authors of these policies trying to achieve?

In high-assurance deployments, the practical goal is usually to establish a meaningful security boundary around cryptographic keys. Organizations want explicit controls over who and what can use a key, and they do not want the answer to be every application or administrator with access to the host. They are also worried about key theft and abuse. For important signing and decryption keys, keeping the key out of the hands of the application and host OS is often the simplest way to force reasonable key-protection practices.

These are real problems. When the threat group Storm-0558 acquired a highly sensitive Microsoft MSA signing key, operational failures allowed key material to escape the isolated signing environment and become accessible from a compromised engineering environment. That single extraction let the attackers forge tokens and compromise customer email accounts at scale. It is much harder to see that happening when a key is non-exportable and managed inside a hardware boundary. HSMs are not the only way to get these properties, but they are the one tool that forces you to think hard about how you operationalize a key, and that discipline has value.

The trade-off is that you end up running old software you cannot patch for upstream security vulnerabilities. In the best case, you are years behind.

Worse, this is usually non-memory-safe code that is entirely opaque to you. The firmware, the middleware, and the technical documentation are kept strictly behind lock and key by the vendor. You cannot inspect the code to see if it is vulnerable, and independent review is virtually impossible. The HSM vendor may have proactively patched those vulnerabilities, but more likely they have not.

The cryptography in these modules almost never breaks. What breaks is the plumbing. For example, when an HSM takes in a payload like an administrative command or an authentication token of some sort, it has to parse it. This may mean relying on complex parsers or business logic, often written in aging C code. Because that plumbing is not memory safe, a single malformed input can lead to a buffer overflow or remote code execution right past the validated boundary, or even worse, a long-forgotten feature combined with new code could bypass policy controls around accessing the module altogether.

Take the U-Boot forks as another example. When you look at how these embedded systems are actually built, they lean heavily on bootloaders, embedded operating systems, and vendor firmware that sit outside the cryptographic boundary but inside your trust story. A vendor might fork U-Boot, validate their module, and then that code is essentially frozen.

Think about what boot verification actually involves. Something has to parse the firmware image, figure out which bytes are covered by the signature, compute the digest, and check the signature. The math for the signature might be FIPS validated, but the parsing, the offset arithmetic, and the handling of attacker-controlled structures is all just non-memory-safe C code.

When researchers find vulnerabilities in X.509 parsing or U-Boot image processing, the flaws are not in the cryptography. They are in the plumbing. And because vendor version strings in this opaque, locked-down firmware often have no meaningful relationship to upstream release numbers, that fork may be a decade old.

The real danger is that two curves are moving in opposite directions. The attack surface is being examined continuously. AI and automated variant analysis are getting exponentially better at finding previously unknown exploitable memory-safety bugs in old C code. Meanwhile, the validated base firmware remains essentially fixed.

We do not have to guess about this. I recently published a data project analyzing recent FIPS 140-3 validations. The data shows how frozen these systems actually are. Out of 415 validated modules we looked at, 324 of them, 78 percent, showed no recorded public update after their initial validation. The knowledge of security issues compounding every week is colliding with code that barely moves.

FIPS 140 compliance buys you necessary domain separation and forces good operational habits. But the way the market achieves that separation often leaves you depending on a static, aging, and opaque codebase. The certificate tells you that a particular version met the requirements when it was evaluated. It does not tell you that the same code remains secure years later. At some point, the certificate becomes evidence not only of what was validated, but of how long the underlying system has stood still.

The Certification Ends Where the Code Begins

Fri, 10 Jul 2026 22:36:46 GMT

Disclosure: I am an advisor to Binarly.

I recently built the FIPS 140-3 Corpus, a dataset that pulls together the public record of FIPS validations. It combines CMVP certificate records, Security Policies, implementation details, operational environments, firmware versions, algorithm claims, and lifecycle data into something you can actually query and analyze rather than read one certificate at a time.

I built it because I have spent enough years around certification programs to know that the interesting information is rarely in any single document. It emerges when you look at the record as a system. Once you do, a pattern shows up that I think deserves more attention than it gets. The public evidence tells you a great deal about what was evaluated and almost nothing about whether the code that shipped actually behaves the way the evaluation assumed.

What the paper trail shows

When you read Security Policies in bulk, you start seeing the same dependencies over and over. Validated modules lean on bootloaders, embedded operating systems, update agents, and vendor firmware that sit outside the cryptographic boundary but inside the trust story. The certificate covers the module. The security property depends on everything around it.

U-Boot is a good example. Several modules in the corpus disclose it as part of their firmware or boot environment, and in some of those cases it participates directly in verifying firmware integrity before execution. Think about what that verification actually involves. Something has to parse the firmware image, figure out which bytes are covered by the signature, compute the digest, check the signature, and then decide what to run. RSA and SHA-256 handle two steps in that sequence. The parsing, the offset arithmetic, the decision about which fields are authenticated and which are attacker controlled, all of that is ordinary C code, and it is exactly where things tend to go wrong.

Binarly's researchers recently published Unfit to Boot, which found previously unknown vulnerabilities in U-Boot's FIT image processing and signature verification path. The flaws were not in the cryptography. They were in the handling of attacker controlled structures before and around the verification operation. The math was fine. The plumbing was not.

This is the oldest lesson in applied cryptography and we keep relearning it. The primitive is almost never the weakest link. The code that feeds the primitive is.

Where the evidence runs out

Here is where it gets uncomfortable. A Security Policy might identify its bootloader with a string like CNN35XX-UBOOT-4.03-03. That tells you a vendor U-Boot derivative is present. It tells you almost nothing else. Which upstream revision was it forked from? What did the vendor change? Which FIT features were compiled in? Were the fixes for known parsing flaws ever backported? Can externally supplied firmware even reach those code paths in this product?

None of that is answerable from the certification record. Vendor version strings in embedded firmware often have no meaningful relationship to upstream release numbers. The fork may be a decade old. The fixes may have been applied selectively, or renamed, or lost in a rebase nobody documented.

Conventional software composition tools do not close this gap either. They work by matching. Filenames, manifests, version strings, hashes, YARA rules, CVE mappings. That approach answers a useful question, namely whether a binary appears to contain a component already known to be vulnerable. Firmware defeats it routinely. Dependencies get statically linked into larger executables, symbols get stripped, and vendor forks drift far enough from upstream that the signatures stop matching anything.

And matching cannot help with flaws nobody has found yet. Before the Unfit to Boot research existed, there was no CVE to map, no affected version range, no signature to match. Someone had to go look at the implementation first. Credit to the Binarly team for doing that work, and disclosure noted, but the point stands independent of any vendor. Until somebody examines what actually shipped, every downstream tool, database, and compliance process is working from an empty record.

Why I care about this for HSMs and BMCs

The corpus is full of devices that sit in unusually trusted positions. HSMs hold the keys for certificate authorities, payment systems, and governments. BMCs sit beneath the host operating system with control over firmware updates, recovery, and remote administration. I have spent much of my career depending on the first category and being quietly worried about both.

These devices are exactly where the paper trail is weakest. They accumulate long lived vendor forks, inherited open source components, proprietary parsers, and hardware specific code written over many years, most of it statically linked and distributed only as compiled firmware. HSM firmware makes the visibility problem even worse. It is almost never publicly accessible, and on the rare occasion you do get an image, it is often encrypted or obfuscated, frequently with a key shared across the product line. Whatever that design accomplishes, it means customers and independent researchers see less of the code than a motivated attacker willing to recover the key. So we end up trusting these devices on the strength of certifications that, as the U-Boot example shows, stop well short of the code paths where real failures happen.

That does not make the certifications worthless. It makes them a starting point. A validation record that discloses a U-Boot derivative in the boot chain has handed you a concrete question to ask your vendor. What evidence supports the claim that your product is unaffected by this class of flaw? Has anyone analyzed the released binary, or is the answer derived from a spreadsheet of version strings? Which fields of an incoming update are actually authenticated before any code touches them? Vendors who can answer those questions with evidence are telling you something important. So are vendors who cannot.

Documentation, inference, evidence

The way I think about it, assurance comes in layers and each layer answers a different question. The certificate tells you what was evaluated and under what assumptions. The corpus connects those artifacts across the whole ecosystem and exposes the shared dependencies and recurring architectures the individual documents obscure. The final layer is evidence about what actually shipped, and it can come from several places. Vendors tracking their forks against upstream and documenting backports. Independent analysis of released binaries. Researchers publishing the failure modes of mechanisms everyone assumed were sound.

Most of the industry stops at the first layer. Procurement checks for the certificate, the checkbox gets ticked, and the boot chain full of forked bootloader code goes unexamined until someone publishes research like Unfit to Boot and everyone scrambles to figure out whether they are affected.

I built the corpus to make the second layer easier, so that the public record can generate the right questions instead of just decorating RFP responses. The questions are the point. A validation record that names a bootloader fork should end in a conversation with the vendor about what is in it, not in a filed PDF. Trust in labels got us the last twenty years of firmware security. Trust supported by evidence is going to have to get us the next twenty.

Steve Jobs, AI, and the Problem of Analysis Without Ownership

Thu, 09 Jul 2026 16:28:09 GMT

There is an old Steve Jobs clip from a 1992 MIT Sloan talk that feels newly relevant in the age of AI. In the talk, available here as Steve Jobs MIT 1992 Lecture, Jobs is asked about consultants. His answer is not that consultants are unintelligent or useless. His criticism is more subtle. He says consultants often get to see a lot, analyze a lot, and recommend a lot, but they do not stay with the work long enough to own the consequences.

They do not spend years living with the product, the team, the tradeoffs, the mistakes, the customers, the budgets, the bugs, or the recovery. They may see the fruit, as Jobs put it, but they “never really taste it.”

That distinction matters.

There is a kind of knowledge that comes from observation, and there is a different kind of knowledge that comes from ownership. Observation can make you articulate. Ownership makes you careful. Observation helps you describe what should happen. Ownership teaches you what actually happens when a recommendation meets constraints, incentives, politics, timelines, systems, and human behavior.

There is also a kind of knowledge that only comes from time.

Some problems cannot be understood in a single sitting. You need to carry them around for a while. You read, step away, come back, notice what still bothers you, test a different framing, sleep on it, and then see the thing that was hiding in plain sight. That kind of soaking is not inefficiency. It is often how judgment forms.

That is the parallel to AI.

AI is making analysis abundant. It can read more than we can read, summarize faster than we can summarize, find patterns across larger datasets, generate plausible options, and produce recommendations that sound polished and confident. That is useful. But it is not the same as judgment.

Used poorly, AI becomes consulting at machine scale. It is fast, articulate, and superficially impressive, but disconnected from whether its recommendations actually survive contact with reality.

It can say what should be done without knowing what happened after someone tried to do it. It can identify risks without understanding which ones mattered. It can produce a roadmap without living through the missed dependency, the customer objection, the policy constraint, the budget cut, the migration failure, or the second-order effect six months later.

It can also make us confuse speed of response with depth of understanding. That may be the deeper risk. AI can collapse the slow work of thinking into the first plausible answer. It can make a problem feel resolved before we have really spent time with it. It can produce fluency before we have earned conviction.

That does not make AI useless. It makes the design problem clearer.

The lazy version of the AI story is the self-driving car analogy, namely once the machine becomes safer, faster, or more consistent, the human gets pushed out of the loop. There will be domains where that is true. But much of knowledge work is different. The goal is not only to execute a task correctly. The goal is to understand the problem well enough to make better decisions the next time.

Execution tools can displace. Reasoning tools should compound.

That is why the most interesting promise of AI is not simply that it becomes a better consultant or even a better operator. It is that AI can help humans become better operators.

Used well, AI becomes a way to think with the material. It helps us understand datasets that are too large to hold in our heads. It lets us explore problem spaces from more angles. It helps test assumptions, compare interpretations, surface edge cases, and ask better questions. It can show us patterns we would have missed, but the value is not just the pattern. The value is that, through the process of interrogation, we understand the problem more deeply ourselves.

AI should not shorten our attention so much as deepen what our attention can hold.

A good AI system should help us return to a problem with more context than we had the last time. It should preserve the questions we asked, the assumptions we tested, the contradictions we found, the evidence that mattered, and the places where our understanding changed. It should make it easier to spend real time with the problem, not merely produce an answer faster.

In that sense, the best use of AI is not instant certainty. It is structured patience.

It lets us soak in a problem more effectively. By that I mean it enables us to hold more evidence in view, revisiting prior interpretations, comparing today’s answer to yesterday’s uncertainty, and gradually turning analysis into understanding.

The goal should not be to outsource judgment to AI. The goal should be to use AI to improve the conditions under which judgment is formed.

A good AI system should not merely say, “Here is the answer.” It should help us see why the answer might be true, where it might be fragile, what evidence supports it, what alternatives exist, and what would change our mind. It should help us move from diagnosis to action, from action to feedback, and from feedback to learning.

That is the line between AI as consultant and AI as learning partner.

This is also where many AI products will disappoint. Dashboards full of findings, risks, summaries, and recommendations can look impressive while still leaving the actual burden on the human team. They create the appearance of progress without necessarily improving understanding. The human still has to decide what matters, translate the finding into action, make the change, verify the outcome, and remember the lesson later.

The point is not that findings and recommendations are useless; they are necessary. Findings are the beginning of the loop, not the end of it. Systems that stop there are not doing judgment automation. They are doing analysis transfer.

The more interesting systems will close the loop. They will connect analysis to execution, execution to verification, and verification to institutional memory. Not because humans should be removed from the process, but because humans should be able to reason from a better substrate.

This is where Jobs' point lands today. The scarce thing is not access to analysis. AI will make analysis abundant. The scarce thing is accumulated judgment, something that only comes from acting, observing, correcting, and learning over time.

Observation gives you language. Ownership gives you consequence. Time gives you depth. Feedback gives you judgment.

Jobs’ critique of consulting was not just a warning about consultants. It was a warning about any tool, process, or person that gets rewarded for sounding right without having to live with whether they were right.

AI will be most valuable not when it becomes the smartest consultant in the room, but when it helps teams build judgment faster, seeing more, acting sooner, sitting with the problem longer, verifying outcomes, and remembering what reality taught them.

The future of AI in knowledge work should not be analysis without ownership. It should be ownership made smarter.

The Breaker, the Priest, and the Philosopher

Thu, 18 Jun 2026 17:19:19 GMT

Spend enough years in security and you notice that the people whose judgment you actually trust are rarely the ones with the cleanest credentials.

They are the ones who have been wrong in public often enough to develop taste. Their authority is earned backward, from scars rather than definitions. When they look at a scheme and say, no, that is wrong, and here is the deeper reason, they are not deriving it from first principles. They are recognizing a shape they have been cut by before.

That is worth taking seriously. In security you get little standing to philosophize until you have shipped something, broken something, defended something, or watched something fail. The person who starts from what is identity, or what is trust, but has never lived with the consequences of an answer, barely exists as a respected type. The people who carry real philosophical weight almost always came up through contact with failure first.

So the security philosopher is not the pure theorist. The philosopher is what a survivor of failure becomes once the failures start to rhyme and a reflective habit sets in. Most often that survivor is a breaker, because breaking is the most direct contact you can have with the gap between how a system should work and how it does. But breaking is not the only contact that leaves marks, and that turns out to matter later.

Everyone who matures this way is answering one question, whether or not they say it out loud.

Why does this keep happening?

You earn the right to answer by watching it happen enough times that fixing the bug stops feeling like an answer. Whatever conclusion you settle on is what you turn into. Some decide people do not understand the systems deeply enough. Some, that bad claims go unchallenged too long. Some, that security is downstream of engineering. Some, that the bytes are downstream of institutions and power. Some, that the abstractions themselves are broken. Some, that the failure is intrinsic and the only honest response is to keep hunting.

Those answers create the taxonomy. The genus is philosopher. The species are sorted by the answer each one gives, and by the way each one tries to make that answer true for other people.

The Sage

The Sage believes it keeps happening because nobody understands the systems deeply enough.

The Sage transmits by instantiation rather than argument. The worldview gets built into a tool, written into a book, embedded in a way of working, and you absorb it by use. A fuzzer can teach an entire philosophy of bug finding. Coverage becomes the thing worth chasing. The tool makes the argument every time it runs, so nobody has to persuade you in a thread. The position is already standing in the room, made of working code.

The Sage is contemplative, the monk of the genus. The work is not quiet because it is timid. It is quiet because it expects reality to do the teaching.

The Gadfly

The Gadfly believes it keeps happening because people are wrong in public and nobody corrects them.

The Gadfly's philosophy exists only in motion. It lives in the thread, the argument, the review comment, the refusal to let an incorrect claim stand. This is Socratic in the original and irritating sense. You learn what is true by watching the argument refuse to die.

The Gadfly may share the Sage's diagnosis exactly. People do not understand the system. But the method is the opposite. The Sage builds, the Gadfly fights. Both believe misunderstanding is the enemy. They differ on whether the cure is a tool or a wound.

The Builder Evangelist

The Builder Evangelist believes it keeps happening because security is downstream of bad engineering.

This is the breaker who concludes the dramatic failure is only the visible symptom. The real incident happened earlier, in how software was designed, reviewed, deployed, owned, or forgotten. The answer is not more heroics. It is to change how teams build, with security folded into normal engineering life rather than bolted on as a gate at the end or a priesthood that arrives with findings after everyone has moved on. The insight only counts if it propagates into how people actually work.

The missionary energy is the tell. Converts always have it, and this is the convert's slot. Philosophy that has to spread to count as true.

The Statesman

The Statesman believes it keeps happening because the bytes are downstream of institutions, incentives, and power.

This one did not go deeper into the stack. They went up, out of the assessment shop and into platforms, governments, trust programs, procurement, incident governance, the places where the adversary is sometimes the org chart and sometimes a nation. The wisdom gets cashed out in policy and in who sits at which table.

The risk of the type is floating clear of the actual bytes. The best Statesmen never do. They remember the policy is only real if it changes what happens at the machine, the credential, the incident bridge. The worst become fluent in altitude and lose contact with the ground.

The Theorist

The Theorist believes it keeps happening because the abstractions themselves are unsound.

This is the opposite vector from the Statesman. Where the Statesman goes up toward power, the Theorist goes down toward formalism. Weird machines. Exploitation as programming a machine nobody meant to build. Security as a subset of reliability. Trust as an operational claim, not a noun.

This is the species that comes closest to the academic register, but the ticket was still bought through breaking first. The formalism is trusted because the person doing it has felt the abstraction fail in their hands. That is the difference between earned theory and decorative theory.

The Refusenik

The Refusenik is the apex breaker who declines to metamorphose at all. Not from inability. From principle.

The refusal is itself an answer. It keeps happening because failure is intrinsic to systems of any real complexity, and pretending a framework or a doctrine can end it is the deeper error. There is no theory waiting at the top of the climb, only the next finding. So the Refusenik stays the predator and calls the philosophizing a comfortable retreat from the only thing that is real, the work in front of them.

This is the lower bound of the taxonomy. It proves that becoming a philosopher of the discursive kind is a choice, not an inevitability, and it does so by holding a real position rather than an empty one. Some of the best who ever lived remain here permanently, and they are right to.

The Priest

The Priest is the auditor, the compliance keeper, the custodian of the framework. The Priest is the scandal of the taxonomy, but not for the reason the field assumes.

The reflexive complaint is that the Priest arrived without breaking. No public exploit, no system torn open, no credential earned the old way. By the breaker's accounting, no scars at all. That accounting is the actual error.

Much compliance really is theater. Much audit work mistakes evidence for reality. Much framework worship trains people to pass inspections while risk keeps moving underneath them. None of that is in dispute. But at scale the Priest is often the only force keeping an organization's vital signs visible. Compliance read generously is not theater. It is a pulse, one of the few observable ways to ask whether the organization is doing the things it claims to.

And the Priest does have scars, just not the kind the breaker recognizes. The Priest learns by watching organizations lie to themselves in patterns, watching the same control fail the same way across a dozen audits, watching risk reappear in process and ownership long after the technical finding was closed. That is sustained contact with failure. It leaves marks. The breaker simply does not read them as marks, because they did not draw blood the familiar way.

So the scandal is not that the Priest skipped the initiation. It is that the breaker cannot see the Priest's scars as scars. Seeing them requires fusing two diagnoses that rarely live in the same person. The Builder Evangelist says automate or drown. The Statesman says the real system is organizational health. The generous read of the Priest requires both at once, and most people who came up breaking cannot hold both, so they read the Priest as the enemy.

Sometimes they are right. Sometimes they are only defending their own credentialing system.

The seam in the taxonomy

Two forces are doing the work here. One is diagnosis, the answer to why this keeps happening. The other is temperament, how you try to transmit that answer. The clean version of the theory says these collapse into one, that the way you transmit is downstream of what you concluded and who you are. The Sage builds because he decided depth is the problem and because he is contemplative. The Gadfly fights because he decided public error is the problem and because he cannot leave it alone.

But the Sage and the Gadfly may share the same diagnosis. If they do, they are one species wearing two faces, and the real joint is diagnosis, with transmission as a surface effect. The alternative is that transmission is itself fundamental, that how you choose to make a thing true for other people is a deeper fact about you than the proposition you are trying to make true.

I do not think this is settled, and I am not sure it should be. A taxonomy that resolved it cleanly would claim to know which of two people who believe the same thing is the more serious, on the sole basis of whether they build or fight. That is a claim worth resisting.

The point

Security does not really trust credentials. It trusts scars. That instinct is mostly healthy. It keeps empty abstraction out of the room and gives weight to people who can smell a failure coming rather than just describe one.

But every credentialing system has a blind spot, and the breaker's is believing that only breaking confers standing. The whole taxonomy is one argument against that belief, because every species on it earned its authority through a different kind of contact with failure.

The breaker learns by cutting into systems.

The builder learns by watching teams repeat the same mistakes.

The statesman learns by watching incentives defeat correctness.

The theorist learns by watching abstractions collapse.

The priest learns by watching organizations lie to themselves in patterns.

The refusenik learns by never looking away from the hunt long enough to be comforted by a story about it.

Five are reflective and one refuses reflection, but all six are forms of earned contact, and none is the only one that counts. What you become is the answer you settle on for why this keeps happening, and the way of making it true for others that you cannot help but reach for.

The breakers who refuse the question stay hunters. The keepers who never broke anything are mistrusted for it, sometimes fairly and sometimes not. Everyone else becomes a philosopher of one species or another, whether or not they would accept the word.

The only real question is which one you became.

The Prompt Is an Argument

Thu, 18 Jun 2026 16:51:45 GMT

The prior piece made a narrow claim. The prompt is the record, because a system can only act on what reaches it. Intent that stays in your head does not govern anything. Context that never reaches the model does not constrain anything. Purpose that is not in the prompt, the retrieved material, the tools, the policies, or the examples is not part of the decision environment.

If you accept that, the next question is unavoidable. What kind of record should a good prompt be?

For goal-oriented work, it should be an argument. More precisely, it should be enthymematic, an argument that leaves its obvious premise unstated for the audience to supply.

The form is Aristotle's. He called the enthymeme the body of proof, the strongest of rhetorical proofs, a kind of syllogism with a premise left out. Its strength comes from the omission. Stating what the audience already accepts is tedious, while assuming the right premise makes the conclusion feel almost self-evident. That power holds on one condition. The missing premise has to be one the audience can already supply.

A capable model supplies a premise readily. It does not reliably supply yours, and from the output alone you cannot tell which one it used. Where your premise and its default mostly agree, the gap is cheap. Where the wrong one is costly, or where you have to reconstruct the reasoning later, it is not.

Look at a bad prompt with that in mind.

"Write about DevOps automation."

That is a topic, not a goal. The model can go anywhere. CI/CD, infrastructure as code, job displacement, Kubernetes, a bland survey that explains nothing. It did not disobey. The prompt simply did not contain an argument for it to advance, so it advanced none.

Now the same subject with the premise restored.

"Explain why DevOps automation is becoming more viable now that production systems can be modeled, tested, and validated in synthetic environments."

This carries a claim the first version left in your head. Automation gets safe where results can be checked. Coding agents became useful because code has feedback loops. It compiles or it does not, tests pass or fail, the type system objects. As production systems become modelable and testable, operations starts to acquire the same machinery, so it starts to look more automatable for the same reason coding did. You could not spell all of that out if you tried. The context behind any goal is unbounded, so every prompt is already a compression, and the only real choice is what to keep. Keep the load-bearing premise and drop what the model already handles. Here that premise is the causal claim, present enough that the model knows which answers would count as success, the one it would otherwise have to invent.

That is the difference between a task prompt and a goal prompt. A task prompt says do X. A goal prompt says do X in service of Y, because Z is the relationship that matters. The "because Z" is where most prompt failures live.

People leave it out because they assume the model already holds the frame. They write as if it knows the business context, the adversary, the institutional scar tissue, the product strategy, the audience, the risk appetite, the standard of correctness. Then a plausible, fluent, useless answer comes back and surprises them. The model was not confused. It was unconstrained. It got a task without the premise that made the task mean anything.

Humans skip premises constantly and get away with it. A lawyer says "that creates a reliance problem" and the room knows the shape of the issue. An engineer says "that breaks rollback" and the team feels the operational cost. A security person says "that moves the trust boundary" and the people who have lived with the system know why it matters. The shorthand works because the history is shared. A model does not inherit that history unless you hand it over. It does not know which premise is obvious inside your company, which analogy is carrying the real weight, which part of the request is decorative and which part is load-bearing. It does not know that "make this clearer" meant keep the technical claim but cut the wording that lets a reader hear monocausality.

This is why writing a good prompt feels more like drafting than asking. A good lawyer does not write words that merely sound like the client's intent. A good lawyer writes words that survive interpretation by someone else, later, under pressure, with incentives to read them the wrong way. The document has to carry its operative meaning forward once the author has left the room. A prompt carries the same burden into a system that completes patterns from whatever materials it has. Leave the wrong premise implicit and the model may complete the argument in the wrong direction. Omit it and the model substitutes a generic one. Supply several competing premises and the model optimizes for the one easiest to write about rather than the one that matters. That is how a prompt produces fluent nonsense. Not that the model cannot write, but that the prompt did not preserve the reasoning.

So before writing a goal-oriented prompt, ask one question. What has to be true for this output to be useful? Not what topic it should cover, what format it should take, how long it should run. What has to be true. The answer is usually the missing premise. For a product strategy it is often the market wedge. For a security review it is the attacker's actual path. For an executive summary it is the decision the executive has to make. For a critique it is the standard the work should be judged against. For a rewrite it is the misunderstanding you are trying to prevent.

That premise does not have to appear as a formal sentence. It can live in the framing, in an example, in the acceptance criteria, in the source material, or in an instruction about what not to optimize for. It only has to be somewhere in the record. Otherwise the system is not helping you pursue a goal. It is producing text near a topic.

In a single prompt you can hold the whole record in view. A production system spreads it out. The effective prompt is no longer the user's sentence. It is that sentence plus the system instruction, the developer instruction, the retrieved documents, the available tools, the policies, the examples, the memory, the model version, and the configuration around all of it. The argument is distributed across those layers, which makes enthymematic design more important rather than less. The question stops being what the user asked and becomes what argument the system received. Did retrieval supply the missing premise or omit it? Did the tool definitions encode the right standard for action? Did the policy layer quietly override the user's goal? Did the examples teach the wrong pattern?

Those are governance questions. A system that cannot reconstruct its effective prompt cannot reconstruct the argument it acted on. And a system that cannot reconstruct that argument cannot explain why its output was reasonable or unreasonable, compliant or not, safe or merely plausible.

The enthymeme works when the missing premise is shared enough to be supplied safely. That is the one condition these systems do not meet on their own.

So the work is not to make prompts longer. It is to make them carry the right inference.

The missing premise will be supplied either way. The only question is by whom. You provide it, or you let the system invent it.

The Prompt Is the Meaning

Tue, 02 Jun 2026 22:50:30 GMT

Why textualism, original public meaning, and AI governance all turn on the same uncomfortable fact: intent does not travel unless it becomes part of the record.

There is an old fight in legal interpretation about where meaning lives.

Intentionalists look for purpose. They ask what Congress meant to do, what the drafters were trying to accomplish, and what problem the law was meant to solve. Legislative history matters in this view because floor speeches, committee reports, drafter notes, and surrounding debate can reveal the intent behind the enacted words.

Textualists are skeptical of that move. They argue that the law is the text that was enacted, not the private intentions of the people who helped write it. The words are the law. The meaning is what the text would have conveyed to a reasonable reader, not what a motivated advocate can later reconstruct from a convenient committee report.

Originalists make a related move in constitutional interpretation. Original public meaning says the Constitution means what its words would have been understood to mean by the public at the time of ratification. Not what a drafter secretly intended. Not what a later judge wishes it said. The meaning is anchored in the text, the historical context, and the interpretive record available at the relevant time.

You do not have to be a textualist or an originalist to see the infrastructure point.

Once a system has to act on language, intent is not enough. Intent has to travel through something. It has to travel through text, context, rules of interpretation, and a record of what was available to the interpreter when the decision was made. Otherwise, “what I meant” becomes an after-the-fact story.

Anyone who has spent time debugging prompts has run into the same problem.

You thought you were telling the model to be careful. You were actually telling it to hedge every answer into uselessness. You thought you were asking it to be concise. You were actually removing the context it needed to be right. You thought you were giving it freedom to reason. You were actually giving it permission to invent.

The words on the page were doing work you did not realize they were doing.

That is the uncomfortable part of working with language models. A prompt is not what you meant. It is not the conversation you wish you had with the model. It is not the background assumptions in your head. It is not the thing you would have clarified if another person had looked confused.

A prompt is the artifact the system received, interpreted, and acted on.

That is why the analogy to textualism matters. In human organizations, we constantly rely on unwritten context. We rely on shared history, institutional memory, tone, relationships, and the ability to stop and ask, “wait, what did you mean by that?” Human communication survives ambiguity because humans have recovery mechanisms.

AI systems do not get those recovery mechanisms for free.

There is no hallway conversation where you explain that when you said X you obviously meant Y. No shared institutional memory unless it is supplied. No unstated assumptions unless they are embedded somewhere in the context. No ability to rely on “what everyone knew” unless what everyone knew made it into the materials the system was given.

The system only has the record.

This does not mean the model is a textualist judge. It is not. The originalist’s reasonable reader is a legal construct. A model is a versioned, probabilistic system operating inside a specific runtime. The same words can produce different behavior under a different model, a different instruction hierarchy, a different retrieval result, a different tool definition, a different policy layer, or a different configuration.

So the lesson is not that the model found the one correct meaning of your prompt. The lesson is that your intent did not travel.

Intent does not become operational just because it existed in your head. Context does not exist just because your team would have understood it. Purpose does not govern the system unless it is encoded in the materials the system actually sees.

This is the prompt engineering lesson people often miss. Prompt engineering is not merely a collection of magic phrases, though some prompt patterns work for model-specific reasons that are not obvious from the surface text. At its core, prompt engineering is drafting. It is the work of turning intention into operative language.

That is why it feels so much like legal drafting. You are not writing what you wish the system understood. You are writing the thing the system will act on.

In production AI systems, that “thing” is larger than the sentence the user typed.

The operative prompt includes the system instruction, the developer instruction, the retrieved documents, the tool definitions, the prior turns, the examples, the policies, the memory, the model version, the ranking logic that decided which facts were included and which were left out, and the configuration that shaped how deterministic or creative the output could be.

That is the effective prompt.

This is where context engineering starts to absorb prompt engineering. The hard problem is no longer merely finding better words to ask the model. The hard problem is constructing the interpretive environment in which the model can behave reliably enough, predictably enough, and accountably enough for the job it is being asked to do.

Legal interpretation has always depended on more than raw text. Textualists still need grammar, usage, canons of construction, dictionaries, historical context, and an account of the reader. Original public meaning still needs evidence of how words were used at the relevant time. Even the most text-centered theories need a record.

AI systems make that dependency operational.

What was the user prompt? What system instruction controlled it? What policy applied? What documents were retrieved? What facts were omitted? Which tool schemas were available? Which model ran? Which version of the surrounding system produced the answer?

These are not merely implementation details. They are the interpretive record of the system.

That is why the prompt is the meaning. Not because the model is a perfect reader. Not because the words have one stable meaning across all systems. Not because intent is irrelevant. But because the system can only act on what made it into the operative prompt. If something was not in that record, it was not available to govern the system’s behavior.

At that point the issue changes. It is no longer just an interpretation problem. It becomes an evidence problem.

For a toy prompt, failure looks like annoyance. The answer was too verbose. The model misunderstood. The prompt needed tuning.

For a production system, failure looks like a governance gap. The system made a decision, but nobody can reconstruct what it was asked, what it knew, what it was allowed to do, what policy constrained it, what model produced it, or which version of the surrounding context shaped the result.

If you cannot reconstruct the effective prompt, you cannot explain the output. If you cannot explain the output, you cannot evaluate whether the system behaved correctly. If you cannot evaluate whether it behaved correctly, you cannot govern it.

This is why prompts need version control. Retrieved documents need provenance. Tool definitions need change history. Policy layers need to be preserved. Model versions need to be tied to outputs. Evaluations need to capture not just the answer, but the context that made the answer plausible.

But preservation is only the first step. A record without evaluation is archaeology. A record without monitoring is trivia. A record without enforcement is a diary. Governance requires the record, but it also requires machinery that acts on it: tests, policy gates, drift detection, human review, incident analysis, and change control.

Otherwise, the organization has outputs but not governance.

A log tells you what happened. A record tells you what the system was asked to do, what it was allowed to know, what constraints it was operating under, and why the resulting behavior was plausible from the materials available to it.

Without that, accountability collapses into storytelling.

The team says the model was supposed to be careful. The prompt says “avoid unsupported claims,” but the retrieved material was stale. The policy says “follow the customer’s procedure,” but the procedure was missing from context. The audit says the system made a decision, but nobody can reconstruct the versioned bundle of instructions, documents, tools, model, policies, and configuration that shaped it.

At that point, you are not governing the system. You are narrating around it.

That is the deeper lesson from textualism, originalism, and prompt debugging.

Text is never just text. It is text plus an interpretive frame, text plus context, text plus a record of what counted as meaning at the time. In law, we fight about that because rights, obligations, and institutional power depend on it. In AI systems, we are turning that fight into software.

The prompt is not your intent.

The output is not self-explaining.

The effective prompt is the record.

You may not fully own the model. You may not own the provider’s policy stack. You may not own the training process, the safety layers, or the hidden machinery that shapes the output.

But you can own your side of the interpretive record. You can preserve what you supplied, what the system saw, what it was allowed to do, what it returned, how it was evaluated, and what changed afterward.

That is the infrastructure of accountable AI.

The prompt is the meaning because the prompt, properly understood, is the record the system acted on.

You either govern that record, or you inherit someone else’s explanation.

A CA That Produces Evidence, Not Promises

Sat, 23 May 2026 23:32:34 GMT

In my last post I argued that high-assurance systems should stop asking to be trusted on the basis of institutional promises and start producing verifiable runtime evidence about what actually happened. This post is the worked example. A certificate authority built that way, what choices it forced, and what is and is not done yet.

When I was at Google I got to work a bit with the BeyondCorp folks. What most didn’t understand is that the BeyondCorp Google used internally was substantially different from the BeyondCorp they launched to customers. On Windows and Linux, the path was TPM-backed device credentials associated with each machine, turning possession of the laptop and an authenticated credential into another factor. The deployment timelines varied by platform, but the important point was the architectural shape hardware-bound device identity became part of the access decision.

You will notice I didn't mention Macs. That's because Apple, although it had a similar secure processor on its devices, did not give customers attestations over keys stored inside it. We could put keys in the Secure Enclave. We could not prove to a third party that we had. For years we tried to get Apple to change that. Eventually they did, and what they shipped was not what we asked for.

We didn't get the ability to put arbitrary keys in the enclave under our control with attestation. We got a device-bound credential signed by Apple that let us verify, at enrollment time, that a request was coming from one of our devices. We used that as a bootstrap to enroll a short-lived credential that the OS stored and used for day-to-day authentication. Apple's attestation answered the which device question. Our short-lived credential answered everything that came after.

That worked. The standardization piece is draft-ietf-acme-device-attest, an IETF working group document I co-author, which lets the same ACME flow carry an Apple Managed Device Attestation statement, a TPM key attestation, or a YubiKey assertion without the CA needing to special-case each platform. Apple's adoption was what unlocked normalizing on a single way across the fleet to authenticate these devices.

That normalization is the relying-party side of the credential story. The hard-edge property, this key lives on this specific device, signed by a chain you can verify back to the manufacturer, is now what we expect from a workload, a laptop, a phone, a passkey, a SPIFFE workload identity. MFA was the first compensating control for the weaknesses in passwords and API keys and bearer tokens. Passkeys, SPIFFE, and certificate-based zero-trust segmentation are the structural answer. We replaced the secret-you-know with a key-you-hold and bound it to hardware where we could.

Google's internal production systems run the same shape, with Titan chips as the foundational substrate for the devices that get on the network for internal and cloud workloads.

That shift is well underway on the side of the wire that uses credentials.

The side that issues them is still mostly software on a server with an API key and a SOC 2 report.

If issuance hasn't kept up with what we now expect from credential holders, what would catching up actually look like? The previous post made the general argument. The CA should be asked to produce evidence, not to be trusted. This post is what that actually looks like when you build it. A certificate authority where the issuance path itself is the evidence, where the policy that fired is part of what was measured, where the key release is gated on the measurement of the binary asking for it, and where every issued certificate is accompanied by a portable bundle a relying party can verify against trust anchors they already hold.

The architecture is designed to run on both AWS and Google Cloud. On AWS, enclave-based deployments use Nitro Enclave attestations, while VM-level deployments can use NitroTPM-backed evidence for measured boot, instance identity, and workload state. On Google Cloud, Confidential VM deployments use AMD SEV-SNP or Intel TDX attestation for the protected execution environment, with Cloud HSM as the key custodian. Where the system needs VM identity, boot-state evidence, or platform posture outside the confidential-computing attestation itself, it can also use vTPM-based evidence.

That breadth is intentional, and I'll come back to why. But the real load-bearing architectural choice is not which cloud, HSM, enclave, VM, or attestation primitive we use. It is the shape of the issuance system inside each trust boundary.

Two components, one evidence chain

The split is structural, not procedural.

In a single-binary CA, policy enforcement and signing authority are separated by code paths inside one process. A vulnerability in the policy evaluator is a vulnerability in the signing authority, because they share an address space. The compartments exist in the source code. They do not exist at runtime.

Here, the compartments are real. The architecture splits issuance into two attested components.

The registration authority receives the certificate request, resolves identity from authoritative sources, evaluates the issuance policy against the requester's posture and attestation, and produces a signed authorization context. The signing oracle holds the path to the CA's signing key and produces the signature. Each runs in a separate attested environment. Each is measured separately. Each has its own keys. Both are written in Go. Memory-safe in normal code, reproducible without ceremony, and a standard library that already covers most of what a CA needs.

The RA and the oracle are different binaries, with different measurements, with different keys, on different network endpoints, joined by mutually-attested TLS where each side has independently verified the other's measurement before either will talk. A compromise of the RA does not give the attacker the signing key, because the signing key is not in the RA's address space and is not reachable from the RA's network position. A compromise of the oracle does not give the attacker the policy evaluator's identity providers, because the oracle has no identity providers wired into it. The interface between them is narrow, signed, and replay-protected.

This is the cross-machine version of the kernel-userland boundary. The kernel does not trust userland's claim that a syscall is authorized. It does the check itself, every time. The oracle does the same thing for the parts of issuance it can verify independently.

What this buys is a bounded blast radius on a compromised RA.

If an attacker takes over the RA, including the RA's signing key, the damage is bounded only to the extent that the oracle requires independently verifiable evidence for the facts that matter. For profile authorization, key binding, replay protection, RA authorization, and certificate structure, the oracle can check those facts locally.

For domain control validation, the same is true only when the evidence is cryptographic or independently corroborated, such as DNSSEC-validatable DNS evidence or signed observations from independent multi-perspective validators. The current implementation does not yet do either of these things, but adding support for them is a straightforward extension of the model. Without that, however, an RA compromise can still become a validation compromise.

That distinction matters. The oracle does not make an RA trustworthy. It makes the RA's assertions conditional. Where the RA presents verifiable evidence, the oracle can check it. Where the RA presents only its own statement about what it observed, the oracle can enforce structure, freshness, profile policy, replay protection, and RA authorization, but it cannot turn that statement into ground truth.

The architectural property is not that the oracle magically knows every fact the RA observed. It is structural separation plus independent verification wherever the fact is independently verifiable. Every other property the rest of this post will describe, measured policy, gated key release, per-operation attestation, portable evidence, depends on that boundary.

What I mean by evidence

Before walking through the attestations, it is worth being precise about the word evidence.

A raw measurement is a narrow statement, this binary had this digest, this key was generated by this HSM, this quote was signed by this platform, this public key matches the attested key. Those statements matter because they are the hard edge of the system.

But they are not enough to explain issuance. A certificate is issued because policy evaluated a set of facts and authorized a specific profile for a specific requester at a specific time.

So evidence here means the decision record, the raw attestations where they can be disclosed, the verifier results, the policy digest, the profile binding, the facts the policy evaluated, the signed authorization context, the oracle's per-operation attestation, the custodian's key-release evidence, and the transparency proof.

draft-ietf-acme-device-attest helps with the requester-side device and key attestation. The CA-side runtime evidence chain is still built from platform-specific TEE, TPM, HSM, KMS, and transparency-log evidence. The point is not that one standard covers all of it. The point is that the CA should preserve the evidence chain instead of collapsing it into a one-bit pass/fail result or a promise.

What each attestation lets you verify

Cross-checking only matters if the things being checked are concrete. Four attestations cross the issuance flow, each one produced by a different party, each one letting the next party check something specific. It is worth walking through what each one actually lets you verify before going further.

The client's attestation

What the RA verifies: that the requestor controls the private key whose public half is in the certificate request, that the key lives on a specific piece of hardware, that the hardware has a manufacturer-vouched identity, and that the key was generated under conditions the policy can reason about.

The shape of the attestation changes between platforms. A TPM-bound key on a Windows or Linux machine, a Secure Enclave key on an Apple device, and a key in a YubiKey's PIV slot each arrive in a different format with a different signing chain and different platform-specific fields. The function is the same. Each one carries a binding between the key in the CSR and a piece of hardware, an identifier for that hardware, the conditions under which the key can be used, and a certificate chain back to a manufacturer the policy can decide whether to trust.

draft-ietf-acme-device-attest is the wire format that carries any of these in the same ACME flow, so the CA doesn't need to special-case each platform. What the policy sees in the end is the same five things regardless of which platform produced them. Does the requestor control the private key. Is the key on a piece of hardware. Whose hardware. Under what conditions can it be used. Does the manufacturer chain trace back to a root the policy will accept.

The RA's environment attestation

What anyone holding the document can verify: that the RA ran a specific measured image inside an attested execution environment, that the cloud provider signed off on the measurement, that the workload identity or role is the one the deployment expects, and that the public key bound to that measured environment for the rest of its lifetime is the one in the document.

AWS Nitro, AMD SEV-SNP, and Intel TDX each produce a different document, but the load-bearing contents are the same: a measurement of what was loaded, an identifier for the platform that produced it, the operating context, and a vendor chain back to a root the relying party can verify.

Worth being explicit about what the document does not let you verify. It does not tell you who deployed the measured environment, who runs the cloud account, or what the operator intended to run. It tells you what was measured. The operator's intent is a separate question, answered by the published image list and the policy that names which measurements are acceptable. A relying party who trusts the hardware vendor's chain can verify the measurement. They still have to verify, separately, that the measurement is one they should accept.

The oracle's environment attestation

What it lets the RA verify, during handshake: that the oracle the RA is about to send an authorization context to is running the published image, and that the signing key the oracle will use for its half of the mutual TLS is the one bound to that image at boot.

What it lets a relying party verify, after issuance: the same thing, plus one additional binding. The oracle produces a fresh attestation for every signing operation, and the attestation is bound to the certificate that operation just produced and to the RA on the other end of the conversation that authorized it. Where the RA's boot-time attestation carries the long-lived public key the measured environment will sign with, the oracle's per-operation attestation pins this specific certificate to this specific oracle measurement with this specific RA.

The difference between boot-time and per-operation attestation is the difference between "the box looked right when it started" and "the box looked right when it did the thing you actually care about." Boot-time attestation is what most confidential-computing deployments do today. It tells a relying party the deployment was valid at startup. It tells them nothing about whether the deployment was still valid five hours later when an actual issuance happened. Per-operation attestation closes that gap.

The custodian's attestation

The CA private key is not loaded by the operator. It is held by a custodian, an HSM with discrete-silicon attestation, a cloud KMS that gates use on attestation, or another measured execution environment. The custody model matters. In the HSM case, the CA key is non-exportable. The signing oracle never receives the private key. It sends a signing request to the HSM, and the HSM signs only when the relevant policy, authorization, and attestation conditions are satisfied. In the software-protected or KMS-protected case, the key, or the key-encryption material needed to use it, may be wrapped so it is usable only inside a signing oracle whose attestation matches a published image.

What the custodian's evidence lets a relying party verify depends on that model. For an HSM-held key, the evidence is not that the key was released. It is that the non-exportable key was used by a particular custodian, under particular firmware, configuration, policy, and authorization conditions. For a wrapped software or KMS-backed key, the evidence can show that key material was made usable only because the oracle's measurement matched a measurement on the published list. Either way, if an operator runs a different binary, however benign the reason, the signing path fails. The dragon's teeth around the data center are still there. They no longer have to do the whole job.

For a concrete look at what one of these documents actually contains, rather than just what it proves, the Peculiar Ventures attestation library parses examples from each of these platforms, including the Marvell HSM attestation produced by Google Cloud HSM. An attestation without a verifier is a claim. With a verifier, it is something a relying party can act on.

Policy as mechanism, not promise

Reading all of those attestations means the policy that evaluated them has to be something concrete enough to read.

Today's CAs publish a CP/CPS. The Certificate Policy and Certification Practice Statement is a document describing what the CA will and will not do. An auditor samples evidence once a year against the document. The document and the system that produces certificates are not cryptographically linked. The relying party trusts that the document describes the system. The auditor's annual report is the closing of the loop.

Cedar policies are different. They are a domain-specific language with declarative semantics, written under version control, statically analyzable, and small enough to read in a sitting. The policy that fires inside the signing oracle is compiled into the measured binary. The digest of the policy travels in the evidence bundle that accompanies the certificate. A relying party can re-fetch the source at the named digest, read the rules, and decide for themselves whether the policy that authorized their certificate is a policy they accept.

The contrast that matters operationally is the one between policy-as-promise and policy-as-mechanism. A CP/CPS is a promise. The auditor verifies, by sample, that the practice resembles the promise. A Cedar policy compiled into a measured binary, with its digest in the bundle, is a mechanism. The relying party verifies, per certificate, that the rules that fired are the rules the CA published.

There is a sharp footgun specific to Cedar that is worth naming because the answer to it is part of the architecture. Cedar's evaluator skips a policy that throws while accessing an attribute that was not present. The convenient result is that policies stay readable in the absence of optional context. The inconvenient result is that a forbid policy with an unguarded attribute access can silently drop, which is a fail-open. The lint at build time requires a has guard before any optional attribute access. A policy that would have failed open is now a build error. The defense-in-depth move is structural. The policy author cannot ship a fail-open by accident.

What this looks like in practice

An employee gets a new smart card from corporate IT in a sealed blister pack. They plug it in.

The enrollment client on the laptop sees a card it doesn't know. It reads the card's GlobalPlatform Card Production Lifecycle data and the card recognition data, learns this is a factory-fresh retail token from a manufacturer it has a trust root for, and verifies the card identity attestation back to that root. The card is genuine. It has never been provisioned. The enrollment client knows what kind of token it is looking at and what the policy says to do with one.

The client provisions the token. It generates a new keypair on the card under a policy that requires user PIN for use and marks the key non-exportable. The card produces a key attestation: a statement signed by the card's manufacturer-installed attestation key asserting that this specific public key was generated on this specific token, has these specific usage constraints, and will never leave the hardware. The enrollment client builds a CSR for the new key and has the card sign it, which is the standard proof that the requestor controls the corresponding private key. It then packages the CSR, the user's identity claim, and the card's attestation into an ACME request and sends it to the RA.

That is the first link. The chain is going to have several.

The RA receives the request inside its measured execution environment. It verifies the CSR signature against the public key in the CSR, which proves the requestor controls the corresponding private key. It verifies the card's attestation against the manufacturer chain. Yes, this is a genuine retail token from a manufacturer we trust. Yes, the key in the CSR is on the card. Yes, the usage constraints match policy. It resolves the identity claim against the corporate identity provider. Yes, this user exists. Yes, they are entitled to this credential type. Yes, their device posture matches. It evaluates the Cedar policy against the cross-product of the device, the identity, and the requested profile. Yes, all of it permits issuance. It builds an authorization context, signs it with the key bound to its measured environment, and sends it to the signing oracle.

The signing oracle receives the context inside its own measured execution environment, over mutually-attested TLS where both sides have verified the other's measurement before they would talk. It does not simply believe what the RA told it. For facts backed by independently verifiable evidence, the oracle re-verifies that evidence against its own configured verifier. In this example, it re-verifies the card's attestation, cross-checks the claims the RA made about the card, the requested profile, and the requester, validates the profile binding, the certificate type, the validity window, the structural invariants the profile requires, and confirms that this RA is authorized to ask for this kind of issuance. It rejects replays. Only then does it ask the custodian for the signing key. The custodian releases it because the oracle's attestation matches the published image. The oracle signs. It produces a fresh per-operation attestation binding this certificate to this measured execution environment and to the RA that authorized it. The issuance is written to a transparency log that independent witnesses cosign.

The bundle that comes back to the enrollment client contains the card's attestation that the key is on the token, the RA's signed authorization context naming the identity, the profile, the policy digest, and the verifiers it ran, the oracle's per-operation attestation binding the signature to a measured binary on a measured platform, the custodian's evidence that the key was released only because the oracle's measurement matched, the transparency log inclusion proof and witness cosignatures showing the issuance was published before the certificate was returned, and the certificate itself.

Every party in the flow did the logical equivalent of what every other party did. It verified upstream evidence against a manufacturer, platform, custodian, or witness root it already trusted, did its work, and produced its own evidence for the downstream party to verify. The bundle packages those attestations and proofs so the subscriber, or anyone the subscriber shares it with, can re-walk the chain end to end against the same trust roots, without having to take any single party's word for it.

The card manufacturer's root says the key is on the hardware. The chip vendor's TEE root says the RA ran the measured image. The chip vendor's TEE root says the oracle ran the measured image. The HSM vendor's root says the key was released to a measurement that matched. The witness network's cosignatures say the log is what the operator published, not a fork served to one relying party. A relying party who trusts each of those roots can verify the certificate's basis of issuance from the evidence bundle, instead of relying only on the CA's institutional promise.

The CA is not asked only to be trusted. The CA produced evidence.

What is built

The architecture above runs in preproduction today on AWS and Google Cloud.

On AWS, that means Nitro Enclaves for enclave-based issuance components and NitroTPM-backed evidence for VM-level identity, measured boot, and workload posture. On Google Cloud, it means AMD SEV-SNP and Intel TDX Confidential VMs for protected execution, Cloud HSM as custodian, and vTPM-based evidence where VM boot state or workload identity needs to be represented.

Classical ECDSA and post-quantum ML-DSA-65 (FIPS 204) hierarchies operate in parallel. ML-KEM-768 (FIPS 203) is the subject key for TLS key-exchange certificates. Cedar policy with the fail-open lint is enforced in the oracle. Per-operation attestation, evidence bundles, the custodian gating key release on measurement, mutually attested TLS between RA and oracle, end-to-end on both clouds.

Each trust domain signs from its own sub-CA, and classical and post-quantum issuance never share a key. Machine, machine-with-EAP, user, group, workload, smart card, TPM AK, and SSH all sit under separate sub-CAs; classical and PQ are separated within each family. A compromise of any single signing key bounds the damage to one family-and-algorithm slice. The architecture treats hierarchy multiplicity as security-domain separation, not as an algorithm-bridging side effect.

Profiles wired up today cover machine authentication including EAP-TLS, DNS-validated TLS server certificates, workload identity including SPIFFE-style URI identifiers, user and group signing and encryption, smart card and PIV logon, TPM AK bootstrap, and SSH user, workload, and host certificates. Each family that supports both has a classical and a post-quantum variant.

The platform breadth is there because no single TEE family fits every customer environment, and the architecture should not be hostage to one chip vendor or one cloud.

The 2029 problem

None of what I've described above is algorithm-bound. That matters because the algorithms are about to change.

In March 2026, Google's Heather Adkins and Sophie Schmieg set 2029 as the target for completing Google's migration to post-quantum cryptography. Google's timeline matters beyond Google. They run Chrome and Android, and when they move, the WebPKI moves with them. CNSA 2.0 puts 2027 on software and firmware signing in National Security Systems and 2030 on general use. The CABF is working through its own timeline. Federal procurement requirements are already moving.

The CA infrastructure that exists today was designed for the snapshot of math problems that the 2029 transition invalidates. Every CA in production is going to be re-architected before it lands. The algorithms expire. The migration is not optional.

The transition itself is going to be heterogeneous. The classical-PQC X.509 path is going to run for a long time alongside what eventually replaces it. Merkle Tree Certificates — batched, transparency-native issuance with much smaller per-certificate overhead — are a likely part of the answer to ML-DSA's signature size on the wire. The architecture above does not care which container format the certificate is in. The attested issuance pipeline, the custodian-gated key release, the evidence bundle, the transparency log — all of it operates on the issuance side. MTC issuance benefits from runtime evidence the same way X.509 issuance does, and the patterns in this post carry over.

A CA built on the runtime-evidence pattern does not cost more to deploy at the moment you are already rebuilding. It costs more only if you skip the rebuild, and skipping is not on the table. The hardware-anchored credential side of the wire has been arriving in production for a decade. The issuance side is the part still running on the old shape. The PQ deadline is the forcing function that makes the issuance side move. The choice is between rebuilding the old shape with new algorithms, and rebuilding it with the same discipline that the relying-party side has spent the last decade adopting.

Short lifetimes with ARI make the operational side tractable. Seven-day certificates with ARI-driven renewal turn the PQ migration from a flag day into a moving window. The fleet rotates without an emergency, without anyone touching a machine, because the CA can shorten the renewal window for specific machines or profiles whenever it wants to.

That is what the next CA looks like. It is not a different CA than the one I have been describing. It is the same one.

A CA Built for the Threat Model We Actually Have

Sat, 23 May 2026 20:35:05 GMT

This builds on earlier posts on what attestation actually proves, what confidential computing is and isn't, and an honest accounting of the problems with the current generation of TEEs. None of those problems go away here. The argument is that despite those limitations, attestation is an important tool. Certificate issuance is overdue to use it.

Back in the 1990s, I was doing some consulting for DigiNotar, yes, that DigiNotar. They had CA facilities in a data center whose perimeter still had WWII-era anti-tank obstacles, large concrete barriers sometimes called "dragon's teeth." Of course, this was an artifact of the facility's history, but data centers are designed from a security perspective with layers of physical protection, including barriers, mantraps, biometrics, individual vaults with cages, individual racks with their own locks and biometrics, cameras, and more. The threat of physical theft, destruction, or manipulation is exactly what these facilities are designed to mitigate.

When building a CA inside one of these facilities, we design yet another layer of protection. Administration networks are segmented from transaction networks, interconnects from supporting infrastructure, the issuance environment from the systems holding root keys. We add our own physical segmentation on top of that so we can build controls around multiple parties being necessary for the more sensitive operations, while still letting routine hardware maintenance happen on the schedule the SLA needs.

These are all useful and important things, but the reality is that CA key material is not likely to be physically stolen. It is more likely to be compromised from the outside. We solve these problems through design, not by writing more code.

Meaningfully measured code forces design upstream of the code itself.

A measurement of a monolithic blob proves almost nothing useful. A measurement that names a specific role, a specific security domain, and a specific assertion the verifier is supposed to act on, proves something. The roles, the domain boundaries, and the questions the verification has to answer have to exist before any code is written, or the attestation is a signature on nothing in particular.

In operating system design, we have similar problems. In naive systems, we load cryptographic keys into memory on a running, network-connected system, accidentally exposing ourselves to memory-disclosure bugs where a network attacker can steal keys. Heartbleed is the canonical example, but the class is what matters. We do this because it is simpler and faster, but it is also less secure. As systems designers, we address this by moving those keys out of the process of the network-connected application and into a different user context. That way, an attacker cannot simply get the network-connected service to dump memory. They have to get persistence and cross a kernel-enforced user boundary.

This is old wisdom. Least privilege and privilege separation exist because network-facing code should not also be the thing that controls the keys.

A parallel showed up in early cryptocurrency exchanges. Hot wallets were used as signing oracles because the design and deployment work needed to prevent that had not been done, and many of the high-profile compromises of that era trace back to that gap. The exchanges that survived learned to put boundaries between the wallet and the network. That boundary did most of the work. The dragon's teeth around the building did the rest.

When third parties need to rely on external services operating in these environments, they often rely on auditors to attest that management assertions about operational practices are actually being followed. These assessments are usually performed by CPAs, not security specialists, which can limit their value. They also often rely on sampling a small portion of transactions to confirm that the controls being evaluated are being followed. That sample is drawn from evidence provided by the entity being audited, which is also the party paying for the audit.

All of the things discussed above help bring some minimal level of transparency and verifiability, but it is turtles all the way down, layer on layer, none of them reaching the runtime where the actual compromise happens. This is where confidential computing, and solutions like Private Cloud Compute, start to matter.

Policy changes meaning in this model. In the traditional assurance world, policy is a written promise. The CA publishes a CP or CPS, the operator commits to following it, and the auditor samples evidence to decide whether that promise was kept. In a runtime-evidence model, policy becomes part of the mechanism. A measured binary evaluates a specific policy, produces a decision, and the digest of that policy travels with the evidence. The shift is from policy as promise to policy as enforcement, from “trust us, this is what we do” to “this is the policy the measured system actually applied.”

Apple's Private Cloud Compute is the worked example. PCC nodes attest to the binary they are running, refuse to do work for clients that cannot verify that attestation, and publish every production build for public inspection. The user's device, not Apple, decides whether a given node is acceptable. That inversion, the relying party verifying the service rather than the service asserting to the relying party, is the part of the pattern that matters. The pieces are not new individually. The combination, at the scale Apple shipped it, proves the pattern is real. The third-party security reviews prove the architecture is serious. Attacks on confidential computing do not refute that point. They prove there is now a boundary worth attacking, measuring, and improving.

Apple is not the only proof point. Signal used SGX remote attestation for private contact discovery in 2017, with clients verifying that the enclave was running the expected open-source code. WhatsApp's end-to-end encrypted backups use an HSM-based Backup Key Vault to keep recovery keys out of the ordinary service path, and that design was publicly reviewed by NCC Group. Microsoft's Confidential Consortium Framework powers Azure Confidential Ledger. Different systems, different threat models, same direction of travel. High-assurance services are moving from institutional assurances toward runtime evidence.

What it looks like

Concretely, a CA built on the Private Cloud Compute pattern looks like this.

Issuance is split into two attested components. The first, the registration authority, takes the certificate request, resolves identity from authoritative sources, evaluates the issuance policy, and produces a signed authorization context. The second, the signing oracle, holds the CA private key and produces the signature. Each runs in a separate attested enclave, and each is measured separately. This means policy can evolve without re-measuring the key-custody component, and keys can rotate without re-measuring the policy component.

The policy layer matters here too. Each component is not just running code, it is making a verifiable policy decision before it acts. The RA decides whether the request is authorized and the identity evidence is sufficient. The signing oracle decides whether the RA, the request, and the authorization context are acceptable before it signs. The evidence does not just say which binary ran. It also says which policy that binary evaluated.

The two components do not trust each other because they are on the same network. They trust each other through attestation, mutually verified at every connection, and the signing oracle does not merely accept the RA’s conclusion. Before it signs, it independently verifies the RA’s attestation, checks that the authorization context is fresh, confirms that the request is bound to an allowed profile, and verifies that the policy facts asserted by the RA match the evidence presented to the oracle. A compromised RA, even one with its own signing key, does not get to mint an out-of-profile certificate, bypass attestation, or turn the CA key into a general-purpose signing oracle.

The CA private key is not loaded by the operator. It is held by a custodian, a hardware security module, a cloud KMS, or another enclave, and it is wrapped so that the custodian will release it only to a signing oracle whose attestation matches a published image. The list of acceptable images is small, public, and updated through a documented process. An operator who runs a different binary, however benign the reason, does not get the key. The dragon's teeth around the data center are still there. They no longer have to do the whole job.

Both the RA binary and the oracle binary are built from public source and are reproducibly buildable. Anyone can rebuild from the published sources, compare their measurement to the one in the attestation, and confirm that the two match. This is the part of the model that makes trust mean something specific. Not the operator's word, not the auditor's snapshot, not the CA's policy statement, but the build process and the published source. To verify what a particular issuance was actually done by, you would not need to be admitted to the data center. You would need a compiler.

Each issued certificate is accompanied by a portable evidence bundle, signed by the attested issuance system. The bundle names the binary that produced the signature, the attestation root that vouched for the binary, the RA policy decision, the oracle policy decision, the identity assertion the RA accepted, and the inputs the oracle independently verified before signing. A relying party who trusts the chip vendor's attestation root can determine for themselves whether the issuance was performed by code on the published list, against the policy on the published list, by an RA that accepted the identity claim it claimed to accept. The CA is not asked to be trusted. The CA is asked to produce evidence.

None of this removes the HSM, the auditor, or the operator. The HSM is still excellent at the threat it was built for, and a custodian holding a key wrapped to an attestation policy is still doing HSM work under the hood. The auditor is still needed to attest that the published policy is sensible, that the source matches the binary, that the threat model is honest, and that the runbook is followed in the moments where attestation cannot help. The operator is still needed to run the infrastructure and respond when things break.

What changes is what they are asked to prove.

Today, a relying party mostly gets institutional assurances. The CA says it followed its policy. The auditor samples evidence and says the controls were operating. The operator says the production system was the one described. Those are useful assurances, but they are indirect. They do not let the relying party inspect the actual path between a request, a policy decision, and a signature.

A Private Cloud Compute style CA changes that. It turns the issuance path itself into evidence. The question is no longer only whether the CA says it followed the rules. The question becomes which measured binary evaluated this request, which measured binary signed it, which policy digest was used, which identity evidence was accepted, what validation methods were used during issuance, and whether all of that matches the public commitment the CA made.

When the source is open and reproducibly buildable, that evidence includes a hash of the code that made the decision and signed attestations about the runtime elements that went into that decision. When the code is not open source, third parties can come in and validate the source, the build process, and the correctness of the claims, as Apple did with Private Cloud Compute. The public hashes then let others verify that the code claiming to provide these guarantees is, in fact, the code that ran.

Open source is not magic, and the point is not faith in "many eyes." The point is that this shifts the emphasis from betting on physical security and operational practice audits to secure system design and cryptographic evidence about what code actually ran and what it actually did.

That is the threat model mismatch, and it is not only a CA problem. We built the WebPKI around buildings, cages, ceremonies, HSMs, and audits because those were the tools we had. We did the same thing cryptographically. We built systems around the assumption that factoring large composites and solving discrete logs on elliptic curves were out of reach. Q-day changes that assumption. Runtime compromise changes the operational assumption just as fundamentally.

We apply the same instincts in any environment we want to call high-assurance. They still matter, but most of the failures we care about are not physical failures. They are logical, remote, operational failures in the runtime path. The rate of change makes that gap wider every year. Annual audits are retrospective, and between them systems change thousands of times, so what the auditor described is rarely what is actually running when a relying party sees a certificate.

Cryptography turns security problems into key-management problems. AI turns assurance problems into runtime-evidence problems. Once agents are making decisions, calling tools, and changing state, the question is no longer what policy you wrote or what control an auditor sampled. The question is what actually ran, what it saw, what boundary contained it, what policy constrained it, and what evidence survived execution.

A Private Cloud Compute style CA gives us a way to make that path visible, attestable, and independently verifiable. The same pattern applies wherever the gap between what we say a system does and what it actually does at runtime matters.

The First AI-Built Zero-Day Is Not the Interesting Part

Mon, 11 May 2026 20:23:44 GMT

In the mid 90s I worked at a company called Cybersafe. Today it would get labeled an IAM/SSO vendor. What we actually built was a first-generation security platform: Kerberos, password management, PKI-based MFA, key management, host intrusion detection, and what would now be called zero trust access. The company failed for the usual startup reasons. People. Corporate Politics. Timing. The technology was a decade ahead of its market.

One debate from that period has stayed with me. As we expanded into host intrusion detection, the question of automated response kept surfacing. Could a system safely act on its own to contain an intrusion in progress? Drop a connection. Kill a process. Isolate a host. Nobody on the team could imagine a credible answer. The false positive risk was unbounded. The response itself could be weaponized. The rule sets were not trustworthy enough to delegate authority. We shipped detection and let humans make the call.

That debate has an answer now, and it is not the one we expected. Automation on the offensive side is not new. Worms, exploit kits, credential stuffing, and phishing infrastructure have been automated for decades. What is new is broad delegated judgment at machine speed, in the hands of people who do not have to worry about false positives because the blast radius is somebody else's network.

What the report actually shows

The interesting question is not whether AI helped produce a zero-day. That was inevitable. The interesting questions are operational. What kinds of systems make bad machine judgment cheap enough to deploy at scale. What kinds of defensive systems are still pretending human review is the control boundary.

Google Threat Intelligence Group's latest AI Threat Tracker report documents the first zero-day exploit that GTIG says it has high confidence was developed with AI assistance. The headline framing is technically correct. The specifics tell a more interesting story.

The exploit was a Python script that bypassed 2FA on an open-source web-based system administration tool. It required valid user credentials in the first place. The criminal group planned a mass exploitation campaign, and Google disrupted it through responsible disclosure to the vendor. GTIG identified the artifact as AI-developed because the code carried obvious tells. A hallucinated CVSS score. Textbook Python formatting. Detailed help menus. Educational docstrings characteristic of training data. The artifact still carried the seams of its production.

This is not the LLM failing at the hard part. The vulnerability itself is a real find. GTIG specifically notes that the 2FA flaw stems from a hardcoded trust assumption, a high-level semantic logic flaw of the kind that fuzzers and static analyzers tend to miss but that frontier LLMs can reason about by reading developer intent. The model did discovery work that previously required a competent human auditor. Where the operation broke down was in weaponization. The attacker shipped an artifact that still looked like a tutorial.

This is a familiar failure pattern showing up on the offensive side for the first time. Fluency reads as competence. The attacker trusted an artifact with hallucinated metadata and educational comments still attached because it looked like a real exploit, in the same way over-eager engineering teams hand agents production credentials because the agent sounded like it knew what it was doing. The criminals here got bitten by the same dynamic that has been producing outages and data loss in vibe-coded production systems for the last eighteen months. The substrate is doing some of the work of inviting the misconfiguration.

Hultquist's thread on the report is hedged correctly. The importance is the trajectory, not this specific specimen. Pull the camera back and the rest of the report is more interesting than the lede.

Three things worth surfacing

APT45 sending thousands of repetitive prompts. The North Korean group has been observed using recursive prompting to analyze CVEs and validate proof-of-concept exploits at scale. That is the industrial-scale answer to LLM variance. Solve the quality problem by amortizing across volume, then have humans cherry-pick the outputs that survived validation. The same statistical strategy that makes modern fuzzing work, applied one layer up the stack. The model does not have to be reliable. The pipeline has to be cheap enough that unreliability does not matter.

CANFAIL and LONGSTREAM using LLM-generated decoy code. A Russia-nexus intrusion cluster has been deploying malware that uses LLM-generated code to conceal malicious functionality. GTIG documented LONGSTREAM containing 32 instances of code querying the system's daylight saving status, repetitive benign-looking activity used to camouflage the malicious core. CANFAIL carries similar filler logic with LLM-generated comments self-describing the decoy blocks. The stylistic noise of LLM output is becoming the obfuscation layer. The verbose docstrings. The textbook structure. The over-explained variable names. These used to be tells. They are now camouflage. Any heuristic built on the AI-tell will start producing false negatives.

The wooyun-legacy skill plugin. A specialized GitHub repository is being distributed as a Claude code skill plugin that integrates a distilled knowledge base of over 85,000 real-world vulnerability cases from the Chinese bug bounty platform WooYun (2010 to 2016). This is the supply side of the same market. Skill packs are tooling. Tooling gets distributed. The economic logic for adversarial skill packs is identical to the economic logic for legitimate ones. Any platform hosting them inherits a familiar problem. App stores and package registries have been working through it for two decades. Making trust decisions at distribution scale about code from parties you cannot directly inspect.

Both sides are running on the same substrate

On the defensive side, Google is using Big Sleep to find vulnerabilities and CodeMender (Gemini-driven) to fix them automatically. The criminals are pulling from a model class indistinguishable from the one Google is running its defensive tooling on. Both sides have access to the same substrate. The differential collapses to data quality, harness sophistication, and discipline around permissions.

That last one is the part the 90s HIDS conversation did not anticipate. It is also the part that should be the least surprising. The controls discipline did not get easier because the platform got more capable. If anything the gradient got worse. A confused regex IDS in 1999 had a bounded action space. The rule set was enumerable. You could write down what it would do wrong. A confused agent in 2026 has whatever action space its credentials grant it, which in most deployments is more than it should. The fluency that made it easy to give the agent broad permissions in the first place is exactly the property that makes its failures look reasonable in the moment.

The race Hultquist refers to is real, and it has started. The race is not about model capability. Both sides are running models from the same vendors, often the same model. The race is about who has better-curated data feeding their harnesses. Who has stricter discipline around what their automation can touch. Who has the institutional memory of what happens when you delegate authority to a system whose judgment you cannot audit in advance.

The HIDS debate from the mid-90s got an answer. It came from the other side of the wire. Not because defenders learned how to trust autonomous judgment, but because attackers learned they did not need to. They could delegate broadly, externalize the blast radius, and let volume compensate for judgment. The defensive answer cannot be more vibes, broader credentials, and better prompts. It has to be the inverse. Narrower authority. Better harnesses. Replayable decisions. And institutional memory about what happens when fluent systems get mistaken for trustworthy ones.

AI Is Not Why They Are Cutting (Yet)

Fri, 08 May 2026 18:38:35 GMT

Back in 2000, the rule of thumb at Microsoft was that each employee needed to average roughly $600K in top-line revenue. Inflation adjusted, that is about $1.1M to $1.2M today. Microsoft was a high-margin software monopoly at peak, so it is not a universal benchmark, but it gives a sense of what disciplined operating leverage looked like even at a company printing money.

Over the last decade, and especially during the COVID-era zero-rate and QE environment, many companies responded to dysfunction by hiring around it instead of fixing it. Cheap capital reduced the pressure to make hard operating decisions. Necessity is the mother of invention, but cheap money suppressed that necessity for a long time.

Then two things changed at roughly the same time. Rates went from zero to five, and Section 174 of the tax code stopped letting companies expense software developer salaries in the year incurred. The R&D amortization rule from TCJA kicked in for the 2022 tax year, forcing five-year amortization domestically and fifteen years for work done offshore. At the exact moment capital got expensive, a major software-company cost center became less friendly from a cash-tax and after-tax economics perspective.

Now AI has added a new pressure. Companies are adopting AI quickly, but we are still early. Much of what is happening inside enterprises is still R&D, experimentation, platform buildout, workflow redesign, and internal tooling. That work is not free. It comes with token costs, infrastructure commitments, GPU capacity, vendor contracts, and a lot of expensive trial and error.

Jensen Huang has made the point, in characteristically aggressive form, that if he pays someone $500K, he expects them to use a meaningful amount of compute to become more productive. Whether or not you take the specific numbers literally, and you probably should not since Nvidia sells the machines that consume those tokens, the economic point matters. AI spend has to come from somewhere.

That is the part many layoff narratives miss. Companies are not simply replacing workers with AI. They are also reallocating budget toward AI. Token budgets, model access, inference costs, internal AI platforms, data infrastructure, and R&D commitments are becoming real line items. To fund them, companies are looking at the headcount they accumulated under different interest-rate assumptions, different tax assumptions, and a different view of software demand.

There is also a demand-side story. COVID pulled years of enterprise software adoption into eighteen months, and a lot of what gets reported as growth now is ARR rotating through M&A rather than new logos landing. In parts of the market, revenue is moving around as much as it is expanding.

That is the real backdrop for the wave of layoffs. AI is the story being told on earnings calls. The reality is accumulated management debt finally meeting a cost of capital that punishes it. Layers of process. Unclear ownership. Duplicated work. Headcount that grew faster than execution improved. And now, on top of that, companies need to make room for a new class of AI-related spend.

The pressure also lands hard on old farts like me. We are expensive. And to be honest, some of us (not all) do not want to change how we work or keep up with how the technology is evolving. That makes us easy targets when finance needs to hit a cost number. AI gives the story a forward-looking sheen, but the underlying move is simpler: reduce expensive headcount, flatten layers, correct years of operational laziness, and redirect budget toward the new thing everyone believes they must fund.

AI is real. The layoff narrative around it usually is not. When you read a layoff announcement blaming AI, you are mostly reading a press release about cost of capital, tax policy, demand pull-forward, AI infrastructure spend, and an org chart that finally got too expensive to defend.

Read the 10-Qs, not the blog posts.

Smaller, Provable, and on Hardware You Own and Operate

Fri, 08 May 2026 17:49:58 GMT

Dino Dai Zovi made an argument recently that I want to build on.

"If you agree that AI will help attackers discover and exploit vulnerabilities 10-100x more easily, then your excess attack surface has also just become 10-100x more of a liability. The right defensive strategy is to prioritize reducing attack surface and trusted computing bases."

The argument is right. It is also not new.

We have been working on this problem for fifty years

Operating system designers gave this set of principles a name in 1975. Saltzer and Schroeder published The Protection of Information in Computer Systems and laid out economy of mechanism, least privilege, separation of privilege, complete mediation, fail-safe defaults, and open design. The Orange Book formalized "trusted computing base" a few years later, with the central observation that the security of a system depends on what is inside the TCB, and that smaller TCBs are easier to make trustworthy than stronger ones. The microkernel debate that ran from Mach through L4 was an argument about how aggressively to apply these principles to commodity systems. seL4 went further and produced a formally verified microkernel in 2009, demonstrating that the principles could be pushed all the way to mathematical proof.

The same ideas show up everywhere once you look. Chrome's site isolation is privilege separation applied to the browser. OpenBSD pledge and unveil are least privilege applied to userland. Linux namespaces, capabilities, and seccomp are mediation primitives. CHERI takes the same intuitions down into the instruction set. GlobalPlatform Security Domains are the smart-card-world version of compartmentalized trust, with separate keysets, separate trust roots, and isolation between issuers, verifiers, and applications on the same chip.

None of this is new vocabulary. Security domains. Privilege separation. Attack surface reduction. Trusted computing bases. We have known the names of these things for decades, and we have known what to do about them.

What AI changes is the math, not the principles. Excess privilege has always been a liability. The probability of it mattering on any given day was low enough, and the timescale on which it mattered was long enough, that organizations could carry oversized TCBs and broad blast radii in the backlog as "things we should clean up someday." AI compresses the timescale and raises the probability. The slack that was tolerable on a five-year cleanup horizon is not tolerable on a six-month one. Dai Zovi's 10-100x is a multiplier on the cost of carrying slack, not a discovery about whether slack should be carried.

The OS tradition assumed you owned the layer below the boundary

There is one place where the classical OS framework needs an extension before it covers the world we are actually deploying into.

The kernel could enforce process isolation because the kernel was below the processes. The hypervisor could enforce VM isolation because the hypervisor was below the VMs. The trust property was "I control the layer below the boundary, so the boundary is meaningful to me." Every classical OS-level guarantee depends on that.

Cloud broke that assumption. AI workloads, which run on cloud GPUs and orchestration infrastructure that almost nobody owns, intensify the break. The layer below your workload is operated by someone else. Their hypervisor, their firmware, their physical facility, their scheduling. The classical principles still apply, but their enforcement mechanism is gone.

Reduction is necessary. Reduction is not sufficient. Once you have shrunk the attack surface and the TCB to something defensible, you still have to prove that the small thing you reduced to is the small thing actually running, and that what it just did is what you said it would do. Without that proof, the small thing is functionally indistinguishable from the large thing. An attacker who replaces your tiny attested signing service with a tiny lookalike has bought themselves all the same access at a lower cost.

The defensive posture in an AI-leverage world is not just smaller. It is smaller and provable.

Law #3 did not go away

There is also one law older than the OS-design principles that the cloud security pitch of the last decade has spent a lot of energy pretending to repeal.

Microsoft's Ten Immutable Laws of Security were published by Scott Culp in 2000. Law #3 is the relevant one here. If a bad actor has unrestricted physical access to your computer, it's not your computer anymore. The marketing for confidential computing has, in effect, been an extended argument that hardware-encrypted memory and remote attestation make Law #3 obsolete on cloud infrastructure. They do not, and the research record is clear that they will not.

Cloud TEEs share microarchitectural resources with the hypervisor and with co-tenants. That is what produces the side-channel catalog. Cloud providers have physical access to every server they operate. That is what produced TEE.Fail. Hardware roots of trust have a shelf life because they live on the same silicon as everything else, and that silicon is in the operator's possession. None of these properties are bugs. They are what "running on hardware somebody else owns" means.

Server-side cloud TEEs are useful for narrow, bounded properties. They are not useful for repealing Law #3 against a determined operator, and they will not meaningfully defeat multi-tenant side channels at the scale at which they are deployed. Selling them as if they would is what produces the gap between marketing and engineering that I have been writing about for the last year in Confidential Computing's Inconvenient Truth, What Is Confidential Computing, What It Isn't, and How to Think About It, and TPMs, TEEs, and Everything In Between.

The criticism in those pieces is specific. It is about the gap between what cloud TEEs are sold as doing (defeating the operator) and what they actually do (making narrow verifiable claims to relying parties about specific operations). The criticism is not that the underlying assurance technology is useless. The technology delivers exactly what it was originally designed to deliver, in the contexts where the original threat model holds. The marketing has been run over those contexts.

Where the assurance property actually delivers

The assurance property does deliver, where the model fits. The model fits when the hardware is in the user's possession, when the device is discrete and tamper-resistant, and when attestation is used to prove "the key in this request lives on this specific device and has never left it" rather than to prove "the operator of the rack cannot read your memory." That is the threat model the technology was designed for, and it has been working in production for a long time.

A few examples of the pattern done honestly.

YubiKey PIV attestation. The YubiKey can produce an attestation certificate, signed by Yubico's manufacturer key, asserting that a private key was generated on this YubiKey, has the slot and policy attributes you expect, and is non-exportable. Yubico documents the protocol clearly. The trust property is sharp because the device is sharp. Discrete silicon, tamper-resistant package, manufacturer chain you can pin against. Law #3 still applies, and it cuts the right way: the user has unrestricted physical access to the YubiKey, and the YubiKey is the user's computer.

Apple Secure Enclave for SSH agents. Paprika and Secretive are SSH agents that store the private key in the Mac's Secure Enclave Processor. The application processor never sees the key, and even root on the Mac cannot extract the key material. Root can still cause the key to be used through the legitimate signing API, modulo whatever consent prompts apply, but extraction itself is what the SEP boundary is built to defeat. The user owns the laptop, the key is on a physically separated processor on the same SoC, and the threat model (other applications on the same device, or malware that compromises the application processor) matches what the SEP was built for.

Smart cards and HSMs. GlobalPlatform Security Domains, the Yubico PIV applet, hardware-backed PKCS#11 tokens, FIPS 140-3 Level 3 modules. Discrete silicon, tamper-resistant packaging, attestation chains rooted in manufacturer keys. The model that worked in the late 1990s and that still works today, because the threat model has not drifted.

PeculiarVentures/attestation is the verification side of all of this. Parsing, validating, and reasoning about attestation evidence from these various sources. Attestation without a verifier is a claim. Attestation with a verifier is something the relying party can act on.

The common shape across all of these is that the user owns the hardware, the boundary is physical, and the attestation chain anchors in a manufacturer key whose threat model the user can actually evaluate. Law #3 is honored rather than denied.

Transparency is the other cross-machine extension

There is a second extension of the classical OS-design tradition that matters for the AI-leverage world, and that composes with attestation in important ways.

Saltzer and Schroeder's open design principle says the security of a system should not depend on the secrecy of its mechanism. The cryptography community has applied this rule to algorithms for decades. The systems community has been slower to apply it to operations. What is the rack actually doing right now? and what has it done in the past? are operational questions, and historically the answer was "trust the operator's audit logs."

Transparency logs are the operational extension of open design. The idea is to publish what a system is doing to an append-only public log, with cryptographic proofs that the log cannot be retroactively modified, and to design the relying party to require evidence from the log before trusting any operation. Multiple independent witnesses cosign the log so that no single party can serve different views of reality to different relying parties.

The pattern is in production at scale. Certificate Transparency requires every WebPKI certificate to be logged publicly before browsers will trust it, which converts CA misissuance from "discovered by accident, sometimes" into "discovered by anyone watching the log." Sigstore applies the same model to software signing, with every signature published to Rekor and consumers able to require log inclusion before accepting a binary. Google DeepMind's Verifiable Data Audit was an early attempt to apply the same model to data access in healthcare. The infrastructure is consolidating at transparency.dev, and C2SP standardizes the interoperability primitives: tlog-tiles, the witness and cosignature protocols, signed-note, and static-ct-api.

Attestation tells a relying party "this code is running right now." Transparency tells a relying party "this code has been published, reproduced, and witnessed by parties whose collusion would be visible." The two compose. Apple's Private Cloud Compute is the most prominent recent example. Every production build is published to a transparency log, user devices will only communicate with nodes whose attested measurement matches the log, and Apple released a virtual research environment so anyone can verify the build claims independently. Google's Project Oak was an earlier expression of the same combination, building remote attestation against publicly-published binaries as the foundation of trust. The Merkle Tree Certificates draft, now a working group document in the IETF's new PLANTS working group, extends the same logic to TLS at scale, replacing traditional X.509 issuance with batched, transparency-native cert formats designed for the shorter lifetimes the WebPKI is moving toward.

The relevant property for the AI conversation is that transparency reduces the number of parties you have to trust to one less than would otherwise be required. With attestation alone, you trust the manufacturer of the silicon. With transparency, you trust any of the witnesses to be honest, plus the manufacturer of the silicon. That asymmetry is what makes transparency the right tool for environments where the operator might be the adversary.

What this leaves for server-side TEEs

Bounded usefulness, designed honestly.

Server-side cloud TEEs do not defeat the operator. They produce narrow verifiable claims that a relying party can check against their own trust anchors. This signing service ran this image at this measurement. This certificate was produced by this enclave for this RA. This policy was applied. This key was attested as non-exportable by the HSM that signed. Each of those is a useful property. None of them is "the operator cannot see your data." Building an architecture that pretends otherwise is how organizations end up with a single point of failure they did not know they had.

I have been building GoodKey CA as a worked example of the bounded-usefulness pattern. A certificate authority is a useful test case for this kind of architecture, because the trust property is sharp and the threat model is well understood. The shape of the answer is mostly classical OS design pulled across machine boundaries, with hardware-anchored trust at the endpoints and a deliberately bounded intermediary in the middle.

Each enclave is a security domain. RA, CA, and HSM are independent compartments. Each has its own measured image, its own keys, and its own attested boundary. Compromising one does not compromise the others. Privilege is separated by design rather than by policy.

The TCB inside each domain is small enough to characterize. Each enclave runs a single-purpose deterministic image. The measurement is one number. The image is reproducible from source. There is no general-purpose runtime to subvert and no orchestration sidecar to gain a foothold from. AWS Nitro Enclaves were the deliberate choice over SGX or TDX. The architecture uses VM-level isolation with dedicated CPU and memory rather than carving enclaves out of shared-cache, shared-core silicon, which reduces a large class of the microarchitectural side-channel exposure that the SGX and TDX families have to grapple with. Dedicated resources, minimal hypervisor, deterministic measurement.

Mediation is complete and inside the boundary. Every signing operation goes through the policy evaluator (Cedar) inside the enclave. Authorization is part of what is attested, not external to it. A compromised RA cannot lie about what policy was applied, because the policy evaluation was inside the measurement.

Trust is not transitive. When the RA tells the CA that a client attestation passed, the CA does not believe it. The CA re-runs the verification itself, against its own registered verifier, before signing anything. This is the cross-machine version of "the kernel does not trust userland's claim that a syscall is authorized." The CA does the check itself, every time.

Per-operation attestation, not per-boot attestation. The CA produces a fresh Nitro attestation for every certificate it signs, with user_data set to SHA-256(certDER || raKeyFingerprint). That binds this specific certificate to this specific enclave with this specific RA on the other end of the conversation. A boot-time attestation tells you the box looked right when it started. A per-operation attestation tells you the box looked right when it did the thing you actually care about.

Hardware-anchored trust at the endpoints. The signing keys themselves live in a hardware HSM with discrete-silicon attestation rooted in the Marvell manufacturer chain. The clients prove they hold hardware-protected keys via TPM or device attestation. The Nitro layer in the middle does not have to defeat AWS to be useful, because the actual key material is protected by a different boundary that AWS does not own, and the evidence on the wire is anchored in trust roots the relying party already trusts.

Operations published to a transparency log. The CA's attested measurements, policy versions, and issuance records get logged to an append-only structure with multi-witness cosigning. The operator still chooses what to submit. What the operator does not get is the ability to retract entries after the fact, modify history, or serve a different version of the log to a different relying party without those parties detecting the divergence. A relying party's confidence that the system has been running honestly over time stops being a function of trust in the operator's audit logs and starts being a function of properties that hold against the operator. This is the same shape Certificate Transparency gives the WebPKI, applied to the CA's own operational claims about itself.

Failure modes are bounded by design. Certificates are seven days. ACME Renewal Information lets the CA shorten renewal windows targeted at specific machines or specific profiles, and goodenroll polls for those signals on its own schedule. The fleet rotates without an emergency window and without anyone touching a machine. The exposure window becomes a configuration choice rather than a function of certificate lifetime, and revocation infrastructure stays out of the critical path of the threat model.

Post-quantum where it counts. ML-DSA-65 (FIPS 204) for certificate signing, ML-KEM-768 (FIPS 203) as the subject key for TLS key-exchange certificates. ARI is what makes the migration tractable on the deployed fleet, because you do not have to wait for natural expiry to do the work.

Nitro is a bounded-trust intermediary. AWS still owns the silicon it runs on. What the architecture buys you is that the property the relying party has to verify is narrow and concrete, and that the actual long-lived secrets are protected by hardware that AWS does not own. Against an AWS-internal threat with full physical access and unbounded effort, Law #3 still applies. Against the attacks the architecture is actually defending against (software compromise of the CA pipeline, a rogue admin pulling secrets through the management plane, a tampered build reaching production), the bounded property is exactly the property you need.

The substrate

An architecture like this only works if the underlying primitives are right. Three pieces of infrastructure I have been spending time on are upstream of GoodKey CA.

PeculiarVentures/scp is GlobalPlatform Security Domain key management in Go. The name is not a coincidence. Smart cards and HSMs have been doing security domains in hardware for two decades, with separate keysets, separate trust roots, and isolation between issuer, verifier, and application code on the same chip. The library implements SCP03 and SCP11 and a typed Security Domain management layer for key lifecycle, certificate provisioning, and trust validation, against verified profiles with byte-exact validation against independent reference implementations. This is the unglamorous work of "make sure the keys you are putting on hardware are actually being put on hardware in the way you think they are." If the key on the device is not where you think it is, every downstream signature is asserting something false.

draft-ietf-acme-device-attest, which I am a co-author on, is the cross-machine extension on the client side. It standardizes how a device proves to an ACME server that the key in a certificate request lives in attested hardware on a specific device. The recent revisions resolved several interoperability gaps that had blocked broad implementation, including the Apple-specific attToBeSigned semantics around sha256(token) versus sha256(keyAuth), an explicit identifier-verification step, the badAttestationStatement error type, and a hardware-module identifier type. The point of the work is to make the client side of the trust chain as verifiable as the CA side. An attested signing service that issues credentials to anyone who asks is not solving the problem, it is moving it.

PeculiarVentures/attestation closes the loop. It is the verifier side that consumes attestation evidence from these various sources (TPMs, YubiKeys, Apple devices, Nitro Enclaves) and reduces it to claims a relying party can act on. Without a verifier, attestation is marketing. With a verifier, it is engineering.

These are not separate efforts. They are what makes hardware-anchored cross-machine trust mean anything in the wild. The transparency-log side of the same problem is being standardized in parallel through transparency.dev, C2SP, and the Merkle Tree Certificates draft, which together extend the same model to issuance auditability at WebPKI scale.

What this asks builders to do

The Dai Zovi prescription is operating-systems hygiene applied to the whole stack. The verifiability corollary is the same hygiene extended across machines you do not own. Both are old. AI is what is making them mandatory.

Pick small. Compartmentalize. Strip privilege to what each component genuinely needs. Make each component's TCB small enough that one person can characterize it in a sitting. Single-purpose services, deterministic builds, dedicated resources rather than shared microarchitectural state, single-image enclaves rather than orchestrated runtimes.

Make it provable across machines. Per-operation attestation rather than per-boot. Independent re-verification at every hop, not transitive trust. Authorization decisions inside the attested boundary. Evidence bundles the relying party can run a verifier against, with their own trust anchors. Short lifetimes with active rotation rather than long-lived credentials backstopped by revocation. And publish the operations themselves to a transparency log with independent witnesses, so the proofs survive disagreement about who saw what when, and so a single dishonest operator cannot serve different versions of reality to different relying parties.

Anchor trust in hardware whose threat model you can actually evaluate. Where you can put the long-lived secret on hardware the user owns, do that. YubiKey, Apple Secure Enclave, TPM in the laptop on the engineer's desk, smart card in the operator's pocket. Where you cannot, use a cloud TEE as a bounded-trust intermediary that produces narrow verifiable claims, and design the architecture so the long-lived material lives in a different boundary that the cloud operator does not own.

And know what your assurance is buying you. Cloud TEEs are not how you defeat the operator. They are how you make narrow operations verifiable to relying parties while accepting that absolute properties against the operator are not on offer. The places where attestation delivers what it advertises are the places where the user owns the silicon. Law #3 has not been repealed, and AI has only raised the cost of pretending otherwise.

Smaller is the easy half. Provable is most of the engineering. On hardware you own is where the property actually holds.

The Illusion of Constant Acceleration

Mon, 06 Apr 2026 03:37:35 GMT

Spend enough time around AI right now and you start to get the feeling that everything is speeding up, all the time.

Every week there is a new model, a new capability, a new claim that some industry is about to be remade. It starts to feel like rapid change is just the new baseline. Like history has bent into a permanently steeper slope.

I do not think that is right.

What I think is closer to the truth is that we have gotten used to confusing motion with progress, and delay with inevitability. Some things are moving very quickly. Others are barely moving at all. We treat the former as inevitable and the latter as unavoidable.

Neither is true.

My father was born in 1942. That is not ancient history. When he was born, there was still a lot of basic infrastructure left to build.

Within a little more than a decade, nonstop transcontinental passenger air service became viable. Less than eight years after that, a human entered space. Eight years later, people were walking on the Moon.

That is a staggering amount of change in a very short period of time.

In one person’s early life, we went from making coast-to-coast air travel practical to landing human beings on another celestial body. Not as a thought experiment. Not as a roadmap. We just did it.

And it was not only aerospace. The Golden Gate Bridge was built in about four years. The first transcontinental railroad was completed in about six. These were massive physical undertakings that reshaped how people moved and how economies functioned, delivered on timelines that would feel almost implausible now.

The easy way to dismiss this is to say that software is fast and physical infrastructure is slow. That if AI looks fast and transit looks slow, that is just how the world works.

But that does not really hold up.

Ukraine did not build its drone ecosystem on leisurely timelines. Tesla compressed what many assumed would be a slow industrial transition into something the rest of the auto industry had to react to. When something actually matters, physical systems move. Supply chains get reorganized. Tradeoffs get made. Bureaucracies get bent. Talent concentrates. People stop explaining why something is hard and start figuring out how to get it done.

That is part of what makes Artemis interesting.

This is not a criticism of Artemis. It is an ambitious and serious effort. But it is also a reminder that progress is not self-sustaining. Apollo is often remembered as a triumph of technology, but it was just as much a triumph of focus, alignment, and urgency. Artemis reminds us that those things matter just as much as the rockets do.

There is another force that shows up in systems like this.

At Google, there was a name for it: slime mold.

It is what happens when layers of process, approvals, coordination costs, and local incentives build up over time until forward motion gets harder even when nobody involved is being unreasonable. Everything makes sense on its own. The system just moves more slowly.

Technology policy has its own versions of slime mold.

We saw it in the crypto wars, when policymakers convinced themselves that math could be slowed down with policy, as if cryptographic reality were open to negotiation. It was not. What that produced was not real control. It produced friction, workarounds, and the illusion of governance.

You can see the same instinct showing up again in parts of the conversation around AI. When institutions feel outpaced, they respond with process. That instinct is understandable, but it rarely solves the problem. You do not make systems safer by pretending inevitabilities are optional. You make them safer by building the infrastructure, incentives, and accountability needed to deal with what is actually happening.

But that is not how we tend to think about progress.

We talk about technological achievement as if it were mostly about invention, as if once something has been demonstrated it remains latent in society, ready to be called back into service whenever we need it.

That is not how any of this works.

The ability to do ambitious things quickly depends on organizational memory, industrial capacity, political alignment, tolerance for risk, and a culture that still expects big things to happen on human timescales.

Lose enough of that, and even getting back to where you once were becomes hard.

You can see it in infrastructure. Projects that once would have been treated as urgent now take decades, often in fragments so small that earlier generations would have treated them as preliminary milestones. Over time, that changes expectations. Slowness starts to look like responsibility. Ambition starts to sound naive.

That is the trap.

The problem is not just that progress slows. It is that people get used to it. What would once have looked like drift starts to look like process. What would once have sounded like an excuse starts to sound like maturity.

Meanwhile, in domains where urgency and incentives line up, things still move very quickly. ChatGPT was released publicly in late 2022. In a few years, AI went from something most people associated with research labs to something embedded in everyday workflows, products, and policy debates.

AI did not prove that everything is accelerating.

It proved that when enough capability, capital, and attention line up, rapid change is still possible.

That is the point.

The world is not uniformly speeding up. Some parts of it are. Others are not. And the difference has less to do with atoms versus bits than with whether we have decided something actually matters.

That ought to make us a little less complacent.

People like to tell themselves that once a technology is important enough, the rest somehow sorts itself out. The problems get solved. The risks get managed. The surrounding systems catch up.

History does not really support that.

Things were only all right in the past because people worked very hard to make them all right. The systems that made aviation safe, that made infrastructure dependable, that made computing usable in high-trust environments, none of that appeared on its own.

The same will be true here.

If we want AI to be safe, trustworthy, and broadly useful, that will not happen as a side effect of capability gains. Security will not emerge on its own. Governance will not emerge on its own. The infrastructure needed to make these systems worthy of dependence will not emerge on its own.

Those things only happen when people decide they matter.

That is the real problem with the idea that everything is accelerating. It makes it easy to believe that progress takes care of itself.

It does not.

Progress happens when people decide it needs to, and then do the work.

Confidential Computing's Inconvenient Truth

Fri, 03 Apr 2026 19:35:42 GMT

This is part of a series on confidential computing. See also: Confidential Computing: What It Is, What It Isn't, and How to Think About It for practical deployment guidance, and Why Nobody Can Verify What Booted Your Server for the attestation infrastructure gap. Two companion reference documents provide the evidence base: the TEE Vulnerability Taxonomy and TPM Attestation and PCR Verification: The Infrastructure Gap.

Confidential computing has a vulnerability record that grows every year, an attestation infrastructure that does not work at scale, and a hardware root of trust with a demonstrated shelf life. This piece explains why.

I want to be clear about where I stand before cataloging problems. I believe in this technology. What Signal has done with Private Contact Discovery and Sealed Sender using SGX enclaves, building systems where even Signal's own servers cannot see who is contacting whom, is exactly the kind of architecture that confidential computing makes possible. Apple's Private Cloud Compute takes the model further. Every production build is published to a transparency log, user devices will only communicate with nodes whose attested measurements match the log, and Apple released a virtual research environment so anyone can verify the claims independently. Moxie Marlinspike's Confer applies the same idea to AI inference, with all processing inside a TEE and remote attestation so the service provider never has access to your conversations. These are real systems delivering real privacy guarantees that would be hard to achieve any other way.

More broadly, TEEs make systems more verifiable. Instead of asking users to take on faith that a service handles their data correctly, the service can prove it through attestation. I wrote earlier about attestation as the MFA for machines and workloads, and I explored the same idea in 2022 in the context of certificate authorities. If the CA runs open-source software on attesting hardware with reproducible builds, you can verify its behavior rather than trusting an annual audit. That shift, from asserted trust to verifiable trust, is genuinely important, and confidential computing is what makes it possible.

But "the direction is right" is not the same as "the current state is adequate." We should not make perfection the enemy of good. This technology delivers real value today. But we also cannot afford to mistake the current state for the desired end state. Getting to where this technology needs to be requires seeing clearly where it actually is. That is what this piece is about.

The answer is not "the implementations are buggy." The answer is structural. These technologies were designed for threat models that do not match how they are being deployed. Smart cards and HSMs were physically discrete devices with clear trust boundaries. TPMs were designed for boot integrity on enterprise desktops. Intel SGX was designed for desktop DRM. Each was repurposed for the cloud because the technology existed and the market needed something now. The repurposing created systematic security gaps that the research community has spent a decade documenting and the market has spent a decade deploying through.

In March 2025, I published a technical reference on security hardware and an in-depth companion document that categorized how these technologies fail. One of those failure categories was "Misuse Issues": vulnerabilities that occur when security technology is adopted beyond its original design. A year later, with TDXRay reconstructing LLM prompts from inside encrypted VMs, TEE.Fail extracting attestation keys with a $1,000 device, and the SGX Global Wrapping Key extracted from hardware fuses, that observation warrants a much fuller treatment.

Timeline

Year	Event	Category
1968	Smart card patents (Dethloff, Moreno). Special-purpose computers in tamper-resistant packages. The original TEE.	Hardware TEE
1980s	IBM secure coprocessors for banking. US government funds kernelized secure OS research.	Hardware TEE
1996	nCipher founded. nShield HSMs with CodeSafe: custom application code inside tamper-resistant hardware.	Hardware TEE
1998	IBM 4758 commercially available. Arbitrary code execution inside tamper-responding enclosure. FIPS 140-1 Level 4.	Hardware TEE
2003	TCG founded, TPM standardized. Designed for boot integrity from ring -x. Hardware root of trust, measurement chains, attestation concepts established.	Institutional
2006	AWS launches EC2. Public cloud computing begins. Workloads move to shared infrastructure owned by someone else.	Cloud
2006	BitLocker ships with TPM support. TPMs reach millions of enterprise devices. Reference value infrastructure never materializes.	Hardware TEE
2008-2010	Cloud goes mainstream. Azure (2010), GCP (2008), OpenStack (2010). Multi-tenant shared infrastructure becomes the default enterprise compute model.	Cloud
2012	AlexNet wins ImageNet. Deep learning proven at scale on GPUs. AI workloads begin moving to cloud GPU infrastructure.	AI
2013	Apple Secure Enclave Processor (iPhone 5s). Physically separate processor on SoC. First mass-market TEE. Invisible to users.	Hardware TEE
2015	Intel SGX (Skylake). Enclaves inside the CPU. Designed for desktop DRM: single-tenant threat model. Cloud providers begin evaluating for multi-tenant use.	CPU TEE
2016	AMD SEV. VM-level memory encryption. First CPU TEE designed with virtualization in mind.	CPU TEE
2017	Transformer architecture published ("Attention Is All You Need"). Foundation for the model scale that will drive confidential computing demand.	AI
2017	First SGX side-channel attacks. Cache-timing, Spectre adaptation. Desktop design meets multi-tenant reality.	Vulnerability
2018	Foreshadow (L1TF) reads arbitrary SGX memory. SEVered remaps SEV guest pages. Desktop-to-cloud threat model gap exploited.	Vulnerability
2019	Confidential Computing Consortium founded (Google, Microsoft, IBM, Intel, Linux Foundation). Repurposing becomes official strategy.	Institutional
2019	Plundervolt, ZombieLoad, RIDL. Three distinct attack classes against SGX in one year.	Vulnerability
2020	GPT-3 (175B parameters). Model weights become billion-dollar assets. Protecting weights on shared infrastructure becomes a business requirement.	AI
2020	AWS Nitro Enclaves. Purpose-built for cloud, not repurposed from desktop. The exception to the pattern.	Cloud
2020	AMD SEV-SNP, Intel TDX announced. VM-level TEEs designed for cloud but still sharing microarchitectural resources. Azure/GCP ship confidential VMs with vTPMs.	Cloud
2021	Intel deprecates SGX on consumer CPUs (11th/12th gen Core). Desktop DRM cannot sustain the technology alone.	CPU TEE
2022	ChatGPT launches (Nov). AI goes mainstream. Every enterprise begins evaluating LLM deployment on cloud infrastructure.	AI
2022	ÆPIC Leak, SGX.Fail. Vulnerable platforms remain in TRUSTED attestation state months after disclosure.	Vulnerability
2023	GPT-4, Llama 2, Claude 2. Foundation model race accelerates. EU AI Act passed.	AI
2023	Downfall (SGX), CacheWarp (SEV-SNP). CacheWarp is first software-based attack defeating SEV-SNP integrity. NVIDIA H100 confidential GPU ships.	Vulnerability
2024	Confidential AI goes mainstream. Azure, GCP, AWS all position confidential computing for AI. TDXdown and Heckler attacks hit TDX. HyperTheft extracts model weights via ciphertext side channels.	AI / Vulnerability
2025 Feb	Google finds insecure hash in AMD microcode signature validation (CVE-2024-56161). Malicious microcode loadable under SEV-SNP.	Vulnerability
2025 May	Google announces confidential GKE nodes with NVIDIA H100 GPUs. Confidential AI training and inference on GPU clusters.	AI
2025 Oct	TEE.Fail. $1K DDR5 bus interposer extracts attestation keys from Intel TDX and AMD SEV-SNP. Attestation forgery demonstrated.	Vulnerability
2025 Dec	IDC survey: 75% of organizations adopting confidential computing, 84% cite attestation validation as top challenge. Gartner predicts 75% of untrusted-infra processing uses CC by 2029.	Institutional
2025 Dec	IETF RATS CoRIM reaches draft-09. Reference value format standards mature. Vendor adoption of publishing measurements remains minimal.	Institutional
2026 Jan	StackWarp (CVE-2025-29943). Stack Engine synchronization bug enables deterministic stack pointer manipulation inside SEV-SNP guest via MSR toggling. Affects AMD Zen 1 through Zen 5. USENIX Security 2026.	Vulnerability
2026	TDXRay (IEEE S&P 2026). Reconstructs LLM user prompts word-for-word from encrypted TDX VMs by monitoring tokenizer cache access patterns. No crypto broken. UC San Diego, CISPA, Google.	AI / Vulnerability
2026 Mar	NVIDIA publishes zero-trust AI factory reference architecture. CPU TEE + confidential GPU + CoCo + KBS. Model weights encrypted until attestation passes.	AI
2026 Mar 31	Ermolov extracts SGX Global Wrapping Key from Intel Gemini Lake. Root key extraction via arbitrary microcode. Unpatchable (hardware fuses).	Vulnerability

Trusted Platform Modules: Boot Integrity and System State

The idea that hardware should measure and attest to software integrity goes back to the late 1990s. The Trusted Computing Group, formed in 2003, standardized the Trusted Platform Module, a discrete chip that stores cryptographic keys and maintains Platform Configuration Registers recording the boot chain as a sequence of hash measurements.

The TPM was designed to solve a specific problem: bootloader-level attacks. Rootkits and bootkits that compromised the system before the OS loaded were invisible to any software-based security tool. The TPM sat below the OS, measuring each boot stage before execution. It could answer a question that no operating system could answer about itself: did this machine boot the software it was supposed to boot?

Each boot stage measures the next before handing off execution. The measurements are extended into PCRs using a one-way hash chain: PCR_new = Hash(PCR_old || measurement). The TPM can produce a signed quote of its PCR values, and a remote verifier can check whether the system booted the expected software stack.

TPMs shipped in millions of enterprise laptops and servers. BitLocker used TPM-sealed keys for disk encryption. Linux distributions added measured boot support. But TPMs never achieved the broad security impact their designers envisioned. The problem was practical: to verify a TPM quote, you need to know what the correct PCR values should be, and nobody built the infrastructure to distribute and maintain those reference values at scale.

The TPM could tell you what booted. It could not tell you whether what booted was good.

What TPMs did accomplish was laying the conceptual groundwork for everything that followed. Hardware root of trust, measurement chains, remote attestation, platform state quotes. All of this vocabulary originated in the TPM ecosystem. Modern CPU TEEs inherited these concepts even as their architectures diverged significantly from the TPM model.

Hardware-Isolated Execution: Older Than You Think

Running code inside a tamper-resistant hardware boundary did not start with Intel or Apple. It started with smart cards.

Smart cards emerged in the late 1960s as special-purpose computers embedded in plastic cards. By the 1980s, they were executing cryptographic operations in banking, telecommunications, and government ID. A smart card is a tiny computer with its own processor, memory, and operating system, running inside a tamper-resistant package. That is a trusted execution environment by any reasonable definition, even if nobody called it that at the time.

HSMs extended the same concept to server-class computing. IBM's 4758, commercially available in the late 1990s, provided a tamper-responding enclosure with its own processor, battery-backed memory, and secure boot chain. If someone tried to open the case, drill through it, or expose it to extreme temperatures, the device would zeroize its keys. The 4758 ran arbitrary code inside the boundary.

nCipher (founded 1996, later acquired by Thales) took this further with CodeSafe on the nShield HSM line, a development framework for deploying custom applications inside the HSM. This was general-purpose computation inside a hardware trust boundary, exactly the model that SGX would later attempt to replicate in silicon without a separate physical device. I spent years working with these HSMs. They ran custom signing logic, policy engines, tokenization routines, and key derivation functions, all inside the tamper-resistant module where the host OS could not observe or interfere.

The difference between these earlier systems and modern confidential computing is not the concept. It is the integration point. Smart cards and HSMs are discrete devices with well-defined physical boundaries. You can see the trust boundary. You can hold it in your hand. SGX, TDX, and SEV moved the trust boundary inside the CPU itself, eliminating the separate device but also eliminating the physical clarity. When the trust boundary is a set of microarchitectural state bits inside a processor with billions of transistors and a microcode layer updated quarterly, the attack surface becomes much larger.

Apple's Secure Enclave Processor, introduced with the iPhone 5s in 2013, sat between these two models. It was a physically separate processor on the SoC with its own encrypted memory, dedicated to protecting biometric data and cryptographic keys. Even a fully compromised application processor with root privileges could not reach the Secure Enclave's memory.

The SEP succeeded where HSMs had stayed confined to data centers for two reasons. It was invisible to users. Nobody configured it or provisioned it. And it protected something users cared about: their fingerprints and their money. The security was a means to a consumer feature, not a product in itself.

Intel SGX: Designed for the Desktop

Intel SGX, introduced with Skylake processors in 2015, brought the enclave concept to general-purpose computing. Instead of a separate processor, SGX created isolated memory regions within the main CPU. Code and data inside an enclave are encrypted in memory and protected from all other software on the system. The enclave's measurement (MRENCLAVE) is a hash of exactly what was loaded, making attestation straightforward. One binary, one deterministic hash.

SGX was designed for the desktop. Its primary use cases were single-tenant scenarios like content protection, DRM key management, and Ultra HD Blu-ray playback. The threat model is clear. One machine, one user, and the enclave protects the content owner's code from that user.

This is a single-tenant threat model. The attacker is the machine owner. There is no hypervisor. There are no co-tenant workloads competing for shared microarchitectural resources. The side-channel attack surface exists, but the economic incentive is limited. The attacker gains access to one DRM key or one media stream.

Enterprise adoption beyond DRM was limited. SGX enclaves had severe memory constraints (initially 128MB). Programming for SGX required partitioning applications into trusted and untrusted components. Intel deprecated SGX from consumer processors in 2021. The desktop DRM use case was not enough to sustain the technology.

Cloud Adoption and the Threat Model Mismatch

The cloud introduced a fundamentally different threat model, and this is where the problems began.

In the desktop DRM model, you protect your code from one user on one machine. In the cloud, you protect your code and data from the infrastructure provider, co-tenant workloads, the hypervisor, firmware, and anyone with physical access to a shared data center. The provider controls the hardware, the hypervisor, the firmware, the physical facility, and the scheduling of workloads across shared CPU cores.

The industry took technologies designed for the desktop single-tenant model and applied them to this multi-tenant cloud model. The architectural mismatch opened attack surfaces that the original designs did not anticipate.

SGX on a desktop shares caches, branch predictors, execution ports, and power delivery with the enclave owner's own code. On a cloud server, those same resources are shared with co-tenant workloads controlled by different parties, each potentially adversarial. Cache-timing attacks that were theoretical on a desktop became practical in the cloud because the attacker could run arbitrary code on the same physical core. The side-channel catalog that accumulated against SGX from 2017 onward was not a series of implementation bugs. It was a consequence of deploying a single-tenant design in a multi-tenant environment.

AMD SEV and Intel TDX were designed with the cloud threat model more explicitly in mind, protecting entire virtual machines rather than individual enclaves. But they still share fundamental hardware resources with the hypervisor and co-tenants. CPU caches, memory buses, power delivery, and microarchitectural scheduling state. CacheWarp, StackWarp, WeSee, and Heckler all exploit the interfaces between the confidential VM and the hypervisor that manages it.

Virtual TPMs are another instance of the same pattern. Physical TPMs provide hardware-rooted trust because they are discrete chips with their own silicon. A vTPM is software running inside the hypervisor or a confidential VM. Cloud providers adopted vTPMs because provisioning hardware TPMs per VM is impractical at scale. The vTPM's trust root is the software stack that hosts it. If the hypervisor is compromised, the vTPM is compromised.

The Repurposing Pattern

This is a recurring pattern in security technology, and it is one I have watched play out multiple times in my career. Build X for threat model Y, then repurpose X for threat model Z because X already exists and deploying it is cheaper than building something new.

SMS was designed for person-to-person messaging. It was repurposed for two-factor authentication because every phone could receive an SMS. The threat model assumed the cellular network was trusted. SIM swapping, SS7 interception, and malware-based SMS capture exploited the gap between "messaging channel" and "authentication channel." NIST deprecated SMS-based 2FA. SMS OTP is still everywhere because deployment inertia exceeds the security community's ability to move the market.

SSL was designed for securing web browsing sessions. It was repurposed for API authentication, IoT device communication, email encryption, and VPN tunneling. Each repurposing exposed assumptions in the original design that did not hold in the new context. The ecosystem spent two decades fixing the gaps through Certificate Transparency, HSTS, and progressively stricter CA/Browser Forum requirements. I was part of that ecosystem. The fixes were not inevitable. They required sustained institutional effort.

TPMs were designed for boot integrity on enterprise desktops. They were repurposed as vTPMs for cloud VM attestation, trading hardware isolation for scalability. SGX was designed for desktop DRM. It was repurposed for cloud confidential computing, trading single-tenant simplicity for multi-tenant attack surface. Each repurposing followed the same logic. The technology existed, the market needed something, and "available now with known limitations" beat "purpose-built but years away."

The repurposed technology works well enough to create adoption. The adoption creates dependency. The dependency makes it difficult to replace even after the threat model gap is well understood. And the security research community spends years documenting the consequences while the market continues deploying.

AWS took a different path with Nitro Enclaves. Rather than building on CPU instruction extensions designed for desktops, Nitro Enclaves are isolated virtual machines on a purpose-built hypervisor with no persistent storage, no network access, and no access from the host. The Nitro model sidestepped many of the shared-resource problems because the hypervisor is minimal and the enclave has dedicated resources. The measurement model is clean. One image, one deterministic measurement.

Azure and GCP followed with confidential VM offerings on AMD SEV-SNP and Intel TDX. Google has positioned confidential computing as foundational to AI, expanding support across Confidential VMs, Confidential GKE Nodes, and Confidential Space with Intel TDX and NVIDIA H100 GPUs.

NVIDIA entered with confidential GPU support on H100 and Blackwell architectures. Their reference architecture for "zero-trust AI factories" combines CPU TEEs with confidential GPUs, Confidential Containers via Kata, and a Key Broker Service that releases model decryption keys only after remote attestation succeeds. Model weights remain encrypted until the hardware proves the enclave is genuine. This positions confidential computing as IP protection for model owners deploying on infrastructure they do not control.

Intel launched Trust Authority as a SaaS attestation service independent of the cloud provider. If the cloud provider both runs your TEE and verifies its attestation, you are still trusting the provider. An independent verifier breaks that circularity.

By 2025, every major hardware vendor and every major cloud provider had a confidential computing offering. The question was no longer whether the technology existed. It was whether anyone could make it work at scale.

Why It Never Hit Mass Adoption

Despite the investment, confidential computing did not achieve mass adoption through the SGX era or the first wave of confidential VMs. Several problems compounded.

Attestation is hard to operationalize. The verification step requires infrastructure that most organizations do not have and that the ecosystem has not built. I wrote about this problem in detail in Why Nobody Can Verify What Booted Your Server. The short version: 84% of IT leaders cite attestation validation as their top adoption challenge.

The performance overhead was non-trivial in early implementations. SGX had significant costs from enclave transitions and limited memory. Confidential VMs with SEV-SNP and TDX reduced this to single-digit percentage overhead for most workloads, but the perception of "secure means slow" persisted.

The developer experience was poor. SGX required application partitioning and a specialized SDK. Confidential VMs improved this by running unmodified applications, but attestation integration, key management, and secret provisioning still required specialized knowledge. As of early 2026, deploying a confidential workload still requires expertise that most teams do not have.

The vulnerability narrative undermined confidence. The side-channel attacks against SGX were not random bugs. They were a predictable consequence of deploying a single-tenant design in a multi-tenant environment. Each new attack generated press coverage and reinforced the perception that the technology could not deliver. Security teams found a long list of CVEs, academic attacks, and "known limitations" that made the risk-benefit calculus uncertain.

And without AI, the use cases were niche. DRM, financial services MPC, healthcare analytics, sovereign cloud compliance. Real markets, but not mass markets. Not enough volume to drive the ecosystem maturity needed for broad adoption.

The Vulnerability Record

The side-channel attacks did not stop with SGX's partial deprecation. They followed the technology into the cloud.

Intel TDX still shares microarchitectural resources with the hypervisor. TDXdown demonstrated single-stepping and instruction counting against TDX trust domains. PortPrint showed that CPU port contention reveals distinctive execution signatures across SGX, TDX, and SEV alike, and because it exploits instruction-level parallelism rather than thread-level parallelism, disabling SMT does not help.

The attack that most directly undermines the "Private AI" narrative is TDXRay (IEEE S&P 2026, UC San Diego, CISPA, Google). TDXRay produces cache-line-granular memory access traces of unmodified, encrypted TDX VMs. The researchers reconstructed user prompts word-for-word from a confidential LLM inference session. No cryptography was broken. The attack works because standard LLM tokenizers traverse a hash map to find token IDs, and that traversal creates a memory access pattern observable at 64-byte cache-line resolution. The host watches which hash map nodes the tokenizer visits and stitches the prompt back together. The encryption protects the data in memory. The computation pattern leaks it through the cache.

TEE.Fail (ACM CCS 2025) is the most dramatic recent finding. Researchers built a $1,000 physical interposer that monitors the DDR5 memory bus and extracted ECDSA attestation keys from Intel's Provisioning Certification Enclave, the keys that underpin the entire SGX and TDX attestation chain. Attestation can be forged. The attack requires physical access, which limits applicability. But cloud providers have physical access to every server they operate.

On March 31, 2026, Mark Ermolov announced the extraction of the SGX Global Wrapping Key from Intel Gemini Lake. This is not a side-channel leak. It is extraction of the root cryptographic key that protects SGX sealing operations. The key wraps Fuse Key 0, which means the entire key hierarchy rooted in hardware fuses is compromised for that platform generation. No microcode update can change fuses. Ermolov's assessment: "its fundamental break means that the HW Root of Trust approach is not unshakable."

Gemini Lake is a low-power consumer chip, not a Xeon server processor. The same attack has not been demonstrated on current server-class implementations. But the research trajectory is clear. Each generation of hardware trust primitives has been broken by the next generation of hardware security research.

Why the Pattern Persists: Five Broken Design Assumptions

The vulnerability record is not a collection of unrelated bugs. It is the predictable result of specific design assumptions that held in the original use cases but fail in the cloud and AI contexts where the technology is now deployed.

The attacker does not share physical hardware with the victim. SGX was designed for a desktop where one user runs one workload. In the cloud, co-tenants share CPU cores, caches, branch predictors, TLBs, execution ports, memory controllers, and power delivery. CacheWarp, StackWarp, and TDXRay all exploit resources that remain shared because complete resource partitioning would make the hardware unusable for general-purpose computing.

The platform owner is not the adversary. TPMs and early SGX assumed the platform owner was the user or a trusted IT department. In the cloud, the provider controls the hypervisor, firmware, BMC, physical facility, and scheduling. The interfaces between the TEE and the provider-controlled environment become the attack surface. WeSee, Heckler, and SEVered exploit these interfaces. TEE.Fail exploits the provider's physical access to the memory bus.

The hardware root of trust is immutable. The attestation model depends on root keys being beyond the reach of software attacks. This assumption has been violated repeatedly. Ermolov reached fuse-based keys through microcode. Google's CVE-2024-56161 found an insecure hash in AMD's microcode signature validation. Sinkclose provided universal Ring-2 escalation on AMD CPUs back to 2006.

Attestation verification is someone else's problem. The specifications define how to produce attestation evidence but not how to verify it at scale. In the desktop DRM case, one binary produced one hash. In the cloud, PCR values are combinatorial across firmware, bootloader, kernel, and boot configuration.

Performance and security tradeoffs are invisible. On a desktop running DRM playback, a 5% performance hit is imperceptible. On a cloud server running AI inference at scale, every percentage point is cost. Disabling SMT, applying Downfall mitigations, and enabling inline encryption all have measurable overhead. Organizations are pressured to disable countermeasures for performance, reopening the attack surface.

These assumptions compound. The attacker shares hardware with a platform owner who is the adversary, exploiting a hardware root of trust that has a shelf life, verified through attestation infrastructure that does not exist at scale, with mitigations that carry performance costs the deployment context cannot absorb. No single patch addresses the compound effect. The assumptions are architectural, not implementational, which is why the vulnerability catalog grows despite continuous investment in mitigations.

The full root cause analysis with specific attack mappings for each assumption is in the companion TEE Vulnerability Taxonomy.

AI Changes the Calculus

All of the problems described above are real and unresolved. None of them are stopping adoption, because AI changed the calculus.

Model weights represent billions of dollars in training investment. A leaked foundation model is a competitive catastrophe. Running inference on shared cloud infrastructure means trusting the cloud provider not to inspect memory, which is the exact problem TEEs solve.

Training data includes regulated information across healthcare, financial services, and government. The EU AI Act, DORA, CCPA, and evolving federal privacy frameworks create compliance pressure that confidential computing directly addresses.

Multi-party AI scenarios (federated learning, collaborative training, secure inference on third-party data) require environments where no single party sees the complete dataset. TEEs provide the isolation boundary. This is why every major hyperscaler is building on confidential computing despite its known limitations.

But AI workloads amplify every weakness. GPU TEEs are new and their attestation models are immature. The attestation chain now spans CPU TEE, GPU TEE, and potentially TPM, each with different measurement schemes. AI workloads run on heterogeneous infrastructure across multiple cloud providers. And AI workloads are the most valuable targets for the attacks TEEs are vulnerable to. An attacker who extracts model weights via a side channel gets a multi-billion-dollar asset.

The market treats the different TEE designs (SGX, SEV, TDX, Nitro, NVIDIA confidential GPU) as interchangeable. They are not. Each has different properties and different security guarantees. Pretending otherwise is how organizations end up deploying against a threat model their chosen TEE was not designed to address.

The Trust Model Gap

The deeper issue is the gap between what is marketed and what is engineered.

Confidential computing marketing says "even the infrastructure provider cannot access your data."

The engineering reality is different. The infrastructure provider cannot access your data through the software stack, but the hardware has known side-channel leakages that a sufficiently motivated attacker with privileged access can exploit. The attestation infrastructure that proves the TEE is genuine has structural limitations that make verification at scale dependent on each organization building its own reference value databases. And the hardware root of trust that anchors the entire system has a demonstrated shelf life.

This is a reasonable tradeoff for many threat models. Most organizations are defending against curious administrators, software-level compromise, and regulatory compliance requirements. Side-channel attacks require significant expertise and often physical access. But the market does not present it as a tradeoff.

What Needs to Happen

Closing the gap between the market narrative and the engineering reality requires work that is less exciting than launching new AI services.

Firmware and OS vendors need to publish reference measurements. The standards exist. CoRIM provides the format. RFC 9683 provides the framework. What is missing is the operational commitment to publish signed measurement values for every release. I wrote about the infrastructure that would need to exist and why none of it does yet.

The industry needs honest threat modeling that acknowledges what TEEs protect against and what they do not. TEE.Fail requires physical access, but cloud providers have physical access to every server. TDXdown requires a malicious hypervisor, which is precisely the threat TDX is designed to defend against. These are not edge cases. They are the threat model.

Attestation verification needs to become a commodity. Organizations should not need to build their own reference value databases, write their own event log parsers, and maintain their own golden image registries. This infrastructure should be as standardized and available as Certificate Transparency logs are for the web PKI.

And the security research community's findings need to be incorporated into the market narrative rather than treated as exceptions. The pattern of continuous vulnerability discovery and mitigation is the normal state of the technology, not an aberration.

Confidential computing is directionally correct. The ability to verify what code is running on hardware you do not control, rather than simply trusting the operator, is a fundamental improvement in how we build systems. Signal proved the model works. The challenge is closing the gap between that promise and the current engineering reality.

The organizations deploying confidential computing for AI workloads today should understand what they are buying. Against the threats they are most likely to face, curious administrators, software-level compromise, regulatory compliance gaps, and unauthorized data access by the infrastructure operator, confidential computing is a significant improvement. Against a well-resourced attacker with physical access to the hardware, side-channel expertise, or the ability to exploit a hardware root-of-trust vulnerability, it is a partial mitigation, not an absolute guarantee.

That is a defensible position. It is just not the one being marketed.

For practical guidance on deployment, see Confidential Computing: What It Is, What It Isn't, and How to Think About It.

For the full vulnerability catalog and root cause framework, see the TEE Vulnerability Taxonomy and TPM Attestation and PCR Verification .

Previously: TPMs, TEEs, and Everything In Between (March 2025). See also: Why Nobody Can Verify What Booted Your Server.

What Is Confidential Computing, What It Isn't, and How to Think About It

Fri, 03 Apr 2026 19:03:52 GMT

Confidential computing is the most important security technology that most organizations deploying it do not fully understand.

Last March, I wrote about the terminology confusion in security hardware — how terms like TEE, TPM, secure enclave, and confidential computing get used interchangeably in ways that obscure what these technologies actually do. The accompanying technical reference laid out the foundational concepts and the ways these technologies fail.

A year later, confidential computing is no longer a niche technology. AI has made it urgent. When you run inference on a model worth hundreds of millions in training compute, on hardware you don't own, in a data center you've never visited, the question of what the infrastructure operator can see becomes a business-critical concern. Confidential computing is the industry's answer.

It is also a technology whose security properties are routinely overstated by the vendors selling it and the cloud providers deploying it. Marketing language like "even the infrastructure provider cannot access your data" appears in product pages from every major hyperscaler. The engineering reality is more constrained than that, and the gap between the marketing and the engineering is where organizations get hurt. None of that means you shouldn't use it. I use it extensively. It means you need to understand what it actually gives you so you can build architectures that account for what it doesn't.

What Confidential Computing Actually Does

Confidential computing protects data while it is being processed. Traditional encryption covers data at rest and data in transit. Confidential computing addresses the third state: data in use, the window when data must be decrypted for computation and is therefore exposed in memory.

The mechanism is hardware-based isolation. The CPU (or GPU, in newer implementations) creates an environment where code and data are encrypted in memory and protected from all other software on the system, including the operating system and hypervisor. The cloud provider's administrators cannot read your data even though it is running on their hardware.

The technology comes in several forms. AMD SEV-SNP and Intel TDX protect entire virtual machines. AWS Nitro Enclaves provide isolated execution environments on Amazon's custom hardware. NVIDIA's H100 and Blackwell GPUs add hardware-encrypted GPU memory with GPU-specific attestation. Apple's Secure Enclave protects biometric data and cryptographic keys on a physically separate processor. The implementations differ significantly, but they share a common principle: hardware-enforced boundaries that the software stack cannot cross.

What It Does Not Do

Confidential computing does not make your workload invulnerable. It changes the threat model. Understanding what it does not protect against matters as much as understanding what it does.

Side-channel attacks remain viable. The CPU still shares caches, branch predictors, execution ports, and power delivery with other workloads. Researchers have demonstrated attacks that extract data from inside TEEs without breaking any cryptography. The TDXRay attack (IEEE S&P 2026) reconstructed user prompts word-for-word from an encrypted Intel TDX VM by watching which cache lines the LLM tokenizer accessed. The data was encrypted in memory. The computation pattern leaked it through the cache.

Physical access defeats memory encryption. The TEE.Fail attack (ACM CCS 2025) used a $1,000 device soldered to the DDR5 memory bus to extract attestation keys from Intel TDX and AMD SEV-SNP. Cloud providers have physical access to every server they operate. That is the threat model confidential computing claims to address.

Attestation depends on hardware roots of trust that have a shelf life. Attestation is how a TEE proves to a remote party that it is running expected code on genuine hardware. The proof depends on cryptographic keys embedded in the processor. Those keys have been extracted. The March 2026 extraction of the SGX Global Wrapping Key from Intel Gemini Lake reached root keys burned into hardware fuses. Google's discovery of an insecure hash in AMD's microcode signature validation (CVE-2024-56161) allowed loading malicious microcode that could subvert SEV-SNP. When root keys are compromised, attestation can be forged.

Attestation verification infrastructure barely exists. Even if the TEE hardware is sound, verifying attestation at scale requires knowing what the correct measurements should be. For TPM-based attestation, this means maintaining reference values for every combination of firmware, bootloader, kernel, and boot configuration across a heterogeneous fleet. That infrastructure largely does not exist. An IDC survey found that 84% of IT leaders cite attestation validation as their top adoption challenge.

The Vulnerability Record in Context

The security research community has published over 50 distinct attacks against TEE platforms since 2017. The companion TEE Vulnerability Taxonomy catalogs these in detail.

The number is large. It does not mean confidential computing is broken. It means the technology has been subjected to intense scrutiny by some of the best hardware security researchers in the world, and they have found weaknesses. The question is whether the risk after deploying the technology is lower than the risk without it.

For most deployments, the answer is yes. Confidential computing raises the bar significantly. An attacker who could previously read VM memory through a compromised hypervisor now needs a side-channel attack, a physical interposition device, or a root-of-trust compromise. Each of these is substantially harder than the baseline attack.

The vulnerability record matters most when the attacker is well-resourced and has privileged access to the infrastructure — which is exactly the cloud provider threat model that confidential computing is designed to address. The threat model it targets is the one where its limitations are most relevant. That tension is real, and pretending it does not exist does not help anyone making deployment decisions.

The Gartner prediction that 75% of processing in untrusted infrastructure will use confidential computing by 2029 assumes a maturity the technology has not achieved. Treating a bounded isolation primitive as a general trust solution is how organizations end up surprised.

How to Think About Deployment

Confidential computing is one layer in a defense-in-depth architecture. It is not a substitute for the other layers. I have deployed confidential computing in production and these are the principles I have found matter most.

Use it, but don't rely on it alone. Encrypt data at rest and in transit independently of the TEE. Use application-level encryption for the most sensitive data so that even a TEE compromise does not expose plaintext. The TEE is a defense-in-depth layer, not your sole protection.

Verify attestation, and understand what verification actually proves. A TPM quote or attestation report proves the state of the machine at the time the quote was generated. It does not prove the machine is still in that state five minutes later. It does not prove the machine's physical location or who has physical access. Build your verification flow with these limitations in mind.

Know your TEE's specific threat model. AMD SEV-SNP, Intel TDX, AWS Nitro Enclaves, and NVIDIA GPU CC have different architectures, different shared resource boundaries, and different attestation mechanisms. They are not interchangeable. A TDX trust domain sharing microarchitectural state with a hypervisor has a different side-channel surface than a Nitro Enclave running on a purpose-built hypervisor with dedicated resources.

Plan for the hardware root of trust to eventually fail. The research trajectory is clear: each generation of hardware trust primitives has been broken by the next generation of hardware security research. Build your key management and secret rotation so that a root-of-trust compromise on one platform generation does not expose secrets that have already been rotated.

Ask whether your workload actually needs multi-tenant cloud TEEs. For some use cases, a physically discrete device — an HSM, a USB Armory, a Nitro Enclave with dedicated resources — provides stronger isolation than a confidential VM sharing silicon with co-tenants. The multi-tenancy problem is where most of the vulnerability surface lives. If your workload does not require multi-tenant shared infrastructure, you can sidestep the largest attack class entirely.

What a Practical Architecture Looks Like

Consider a Certificate Authority that runs its signing operations inside AWS Nitro Enclaves. The enclave has no persistent storage, no network access, and no access from the host instance. The signing key never leaves the enclave. Attestation is verified through the Nitro Attestation PKI, which produces deterministic measurements of the enclave image.

Nitro Enclaves were chosen because their architecture sidesteps the shared-resource side-channel problems that affect SGX and TDX. The Nitro hypervisor is purpose-built and minimal. The enclave gets dedicated resources. The measurement model is clean: one image, one deterministic measurement, no combinatorial PCR explosion.

But the enclave is not the only security layer. The signing keys are backed by a hardware root of trust. The enclave image is built from reproducible builds so the expected measurements are verifiable from source. Access to the host instance is controlled through IAM policies that are themselves audited. The architecture is designed so that compromising any single layer does not compromise the signing keys.

Use confidential computing as a meaningful security improvement, understand its specific limitations, and build the rest of your architecture so that the limitations do not become single points of failure.

Where This Is Heading

Confidential computing is not going away. The economic pressure to deploy AI workloads on shared infrastructure guarantees continued investment. NVIDIA's Blackwell architecture extends confidential GPU support. ARM CCA adds Realm World isolation. The Confidential Computing Consortium continues to drive standardization.

The technology will improve. Side-channel mitigations will get better. Attestation infrastructure will mature — the IETF RATS standards are ready, and what is missing is vendor adoption of publishing reference values. Performance overhead will continue to decrease.

But the fundamental constraints — shared microarchitectural resources, physically accessible memory buses, the shelf life of hardware roots of trust — are properties of how CPUs and memory work. They will not be eliminated by the next generation of silicon. They will be reduced, mitigated, and worked around.

Confidential computing is a significant improvement in security posture. It is not an absolute guarantee. That is a defensible position to sell. It is just not the one being sold.

For the deeper analysis of why the vulnerability record looks the way it does, see Confidential Computing's Inconvenient Truth.

For the full vulnerability catalog, attestation gap analysis, and root cause framework, see the TEE Vulnerability Taxonomy and TPM Attestation and PCR Verification .

Previously: TPMs, TEEs, and Everything In Between: What You Actually Need to Know (March 2025). See also: Why Nobody Can Verify What Booted Your Server.

Why Nobody Can Verify What Booted Your Server

Fri, 03 Apr 2026 18:29:28 GMT

There is no public database of known-good TPM measurements. There never has been.

The Trusted Platform Module, a security chip that measures and attests to system integrity, has been a standard for twenty years. TPMs ship in virtually every enterprise laptop and server. Software-emulated versions are provisioned for every cloud VM on Azure, GCP, and AWS. Measured boot is a checkbox in every compliance framework that touches system integrity. The hardware that produces platform measurements is everywhere. The infrastructure to verify those measurements is not.

If you have deployed measured boot at scale, you have hit this wall. I have, more than once. If you haven't yet, you will.

I wrote about the foundational concepts behind these technologies last year, covering how TPMs, TEEs, HSMs, and secure enclaves differ and where they fail. This post goes deeper on one specific problem that anyone deploying measured boot or confidential VMs hits immediately: the verification gap for PCR values.

What PCRs Are and Why They Exist

A TPM contains a set of Platform Configuration Registers, special-purpose storage locations that record the boot chain as a sequence of cryptographic measurements. Each boot stage measures the next before handing off execution. The measurements are extended into PCRs using a one-way hash chain: the old value is concatenated with the new measurement and hashed to produce the new value. This is irreversible. Given a final PCR value, you cannot determine the individual measurements without replaying the full sequence.

A TPM quote is a signed snapshot of these PCR values, which lets a remote verifier assess what software actually booted on the machine. This is remote attestation, and it answers a question no operating system can answer about itself: did this machine boot what it was supposed to boot?

This works fine for a single machine. The problem is fleets.

Why There Is No PCR Registry

You would think someone would have built a public database of known-good PCR values by now, something like CCADB for certificate trust or VirusTotal for malware hashes. Nobody has, and it is not because nobody thought of it. The reasons are structural.

PCR values are combinatorial. A single PCR accumulates measurements from multiple software components. PCR 0 reflects the firmware version, CPU microcode patches, and the UEFI configuration that controls early boot behavior. PCR 4 reflects the bootloader and the shim that validates Secure Boot signatures. On modern Linux distributions using Unified Kernel Images, which bundle the kernel and initial RAM disk into a single signed binary, measurements fragment across PCRs 8, 9, 11, and 12 depending on the distribution and boot configuration. This is messier than the traditional GRUB boot path, and it was already messy.

Any component update produces a completely different PCR value for the affected register. A fleet with 3 firmware versions, 2 bootloaders, 4 kernels, and 3 initrd configurations has 72 valid PCR value combinations for a single hardware model. Five hardware models is 360. Add boot parameters and the number becomes effectively unbounded.

Measurement ordering matters. The hash chain is order-dependent. Extending measurement A then B produces a different result than B then A. Boot is not fully deterministic. Driver initialization order, ACPI table enumeration, and peripheral probe sequences can vary between boots of identical software on identical hardware. The TCG's own specification acknowledges this directly: operating system boot code is "usually non-deterministic, meaning that there may never be a single 'known good' PCR value."

Firmware measurements are opaque. The UEFI event log is the detailed record behind those PCR values, and in practice it is often more useful than the final values themselves. But the event data for firmware blobs is often just a physical memory address and size. No indication of format or purpose. Intel Boot Guard measurements use methods that are under NDA. Dell extends proprietary configuration data into PCR 6 in undocumented formats. A verifier cannot independently reconstruct many of these measurements without vendor-specific knowledge that is not publicly available.

Nobody is obligated to publish reference values. The standards for publishing expected measurements exist. The TCG Reference Integrity Manifest specification defines the formats. The IETF RATS working group developed CoRIM, a compact machine-readable format for publishing reference measurements. RFC 9683, which covers remote integrity verification of network devices containing TPMs, specifies that software suppliers MUST make reference values available as signed tags. The standards are there. Manufacturers are not obligated to follow through, and most do not.

What Everyone Actually Does Instead

PCR value matching fails at scale, so the industry has quietly converged on something else: event log verification.

The TPM does not just produce final PCR values. It also maintains an event log, a sequential record of every individual measurement extended into each PCR during boot. Each entry contains the PCR index, the hash of what was measured, and a description of the event — "loaded bootloader from partition 1" or "Secure Boot certificate db contained these entries."

The event log is what makes attestation workable in practice. The verifier replays the log by re-computing the hash chain from the individual entries. If the replayed chain produces the same PCR values that the TPM signed in its quote, the log has not been tampered with. The events it describes are the actual events that produced those values. The verifier then evaluates individual events against a policy: is this firmware version on the approved list? Is Secure Boot enabled? Is the kernel signed by a trusted key? Was anything unexpected loaded?

This is more flexible than PCR matching. A firmware update changes one event in the log, not the entire composite hash, so the policy absorbs the change without requiring new reference values.

But event log verification has its own problems. Event data is often insufficient for independent verification. Vendor-specific formats are undocumented. Event types and descriptions are not part of the hash, so they can be manipulated without affecting the signed PCR value. Intel's CSME subsystem extends measurements that verifiers cannot evaluate without access to Intel's proprietary documentation.

Keylime, the most mature open-source attestation framework, says it plainly: direct PCR value matching is "only useful when the boot chain does not change often." Intel Trust Authority, Google Cloud Attestation, and Azure Attestation all verify event log properties rather than matching literal PCR values.

So every organization deploying TPM attestation at scale ends up building their own reference values by capturing measurements from known-good environments. The "registry" is whatever you build from your own golden images. This is not a sustainable state of affairs, but it is the state of affairs.

vTPMs Add Another Layer

Virtual TPMs make the verification problem worse. A physical TPM's trust comes from being a discrete chip with its own silicon. A vTPM is software running inside the hypervisor or a confidential VM. Cloud providers adopted vTPMs because provisioning physical TPMs per VM is impractical at cloud scale.

The vTPM's trust root is the software and hardware stack that hosts it. If the hypervisor is compromised, the vTPM is compromised. If the CPU's hardware isolation (the TEE that protects the confidential VM) has a side-channel vulnerability, the vTPM's keys are exposed through that side channel. Verifying vTPM evidence requires also verifying the TEE evidence, because the trust chains through.

Each layer's trust depends on the layer below, and the bottom layer has a demonstrated shelf life. The March 2026 extraction of the SGX Global Wrapping Key from Intel Gemini Lake and Google's discovery of an insecure hash in AMD's microcode signature validation (CVE-2024-56161) are the latest demonstrations that hardware roots of trust are not permanent.

A Practical Approach

The reference value infrastructure does not exist. So what do you actually do?

Pick the verification approach that matches what your deployment can support, and accept the tradeoff. I have listed these from strongest assurance to weakest, which is also from highest operational cost to lowest.

Exact PCR match compares values against a fixed allowlist. Strongest when reference values are correct. Breaks on any component update. Only practical for enclave-style deployments like AWS Nitro Enclaves or Intel SGX, where one image produces one deterministic measurement. If you control the entire image and the measurement is deterministic, this is the easy case.

Event log policy replays the event log and evaluates individual events against policy. Flexible to component updates. Requires an event log parser and per-vendor knowledge of event formats.

Signed baseline accepts any PCR values covered by a signature from a trusted key. The signing key becomes the trust anchor rather than a registry of literal values. When software updates change PCR values, the security team signs a new baseline. This is the PolicyAuthorize pattern that System Transparency documents and pcr-oracle supports: seal secrets to a signing key rather than to specific PCR values, so that software updates do not lock you out of your own data.

Node identity only verifies the TPM's Endorsement Key identity without PCR verification. Proves hardware identity, not software state. Weakest assurance, lowest operational cost.

Most real-world deployments will use different approaches for different parts of their architecture. Exact match for the most sensitive operations. Event log policy for managed servers. Signed baselines for fleet environments where the security team controls the update cycle. The right answer is almost never one approach for everything.

What Would Need to Exist, and Why It Matters

The gap between what TPM attestation promises and what it delivers at scale comes down to five missing pieces of infrastructure. None of them are technically novel. All of them require cross-vendor coordination, which is the hard part.

Firmware vendors publishing signed reference measurements for every release. If Dell, HP, Lenovo, Supermicro, and Intel published signed CoRIM measurement bundles alongside firmware updates, verifiers could check boot measurements against vendor-provided values instead of building golden image databases. The thousands of organizations currently maintaining their own reference values stop doing that redundant, error-prone work. A firmware update becomes verifiable by any attestation service, not just by organizations that happened to capture the right measurements before deploying. This is the single highest-impact change.

OS vendors publishing signed reference measurements for kernels, bootloaders, and initrd images. Red Hat, Canonical, and SUSE would publish expected measurement values for each package version. The cost of operating measured boot drops from "dedicated team" to "configuration."

A transparency log for reference measurements. Analogous to Certificate Transparency for the web PKI. Reference value providers submit signed measurements to a log. Verifiers check the log. Monitors detect inconsistencies. The incentive structure shifts from "trust the vendor" to "verify the vendor," which is the entire point of attestation in the first place.

This is not hypothetical. I worked on firmware transparency at Google, including work with Andrea Barisani to integrate it into the Armored Witness, a tamper-evident signing device built on TamaGo and the USB Armory platform. Google publishes a transparency log for Pixel factory images. The broader Binary Transparency framework has production deployments across Go modules, sigstore, and firmware update pipelines. Researchers are extending the approach to server firmware signing. The pattern works. What is missing is adoption by the server firmware vendors whose measurements actually need verifying.

Cross-vendor event log normalization. A library that translates vendor-specific event log formats into a common representation, abstracting away the differences between Dell, HP, Lenovo, and Intel firmware event structures.

Attestation verification as a commodity service. Not vendor-specific, not requiring deep expertise, but as simple as an OCSP responder for certificate revocation: send a TPM quote and event log, get back a signed attestation result.

None of these exist at scale as of April 2026. The standards are ready. The hardware is deployed. The market is adopting confidential computing at a pace that assumes this infrastructure is coming. It is not here yet.

None of this fixes the side-channel vulnerabilities in the TEE hardware itself. None of it extends the shelf life of hardware roots of trust. Those are silicon problems that require silicon solutions. But the attestation infrastructure gap is not a silicon problem. It is a coordination and incentive problem, and those are solvable.

The web PKI went through a similar transition, and I watched it happen from the inside. Certificate mis-issuance was undetectable until Certificate Transparency made it visible. Certificate authorities operated without enforceable standards until the CA/Browser Forum Baseline Requirements created them. There was no shared database of trusted roots until CCADB built one. Each of those required cross-vendor coordination that looked unlikely right up until it shipped. The result is an ecosystem that is not perfect but is dramatically more trustworthy than it was fifteen years ago.

The attestation infrastructure could follow the same path. The standards work is done. What remains is the operational commitment from the vendors who manufacture the hardware and the organizations that rely on it.

Every organization deploying measured boot today is independently solving the same problem with their own golden images, their own event log parsers, and their own reference value databases. I have built some of these myself. The standards are ready, the hardware is deployed, and the economic incentive is growing. What is missing is the willingness to coordinate. That is a solvable problem.

This post is the first in a series on confidential computing. The next two posts, What Is Confidential Computing, What It Isn’t, and How to Think About It, and Confidential Computing's Inconvenient Truth. Two companion reference documents provide the full evidence base: the TEE Vulnerability Taxonomy and TPM Attestation and PCR Verification

Previously: TPMs, TEEs, and Everything In Between: What You Actually Need to Know (March 2025)

We Built It With Slide Rules. Then We Forgot How.

Sun, 29 Mar 2026 23:39:10 GMT

My father grew up on a subsistence farm, the kind that raised chickens and grew just enough to get by. Farmers were the original hackers. You couldn't wait for the right tool or the right expert. You fixed what was broken with what you had, because the alternative was worse.

As a kid he taught himself rocket chemistry. Not from a kit. From whatever he could source locally. He was trying to make things burn hotter and fly farther, adjusting mixtures through trial and error long before he had words like specific impulse or oxidizer ratio for what he was doing.

The materials weren't exotic. Potassium nitrate sold as stump remover. Sulfur and charcoal. Mix them correctly and you have black powder, the same oxidizer-fuel logic underlying every solid rocket motor ever built. More ambitious builders used potassium perchlorate from chemical suppliers, mixed with aluminum powder or sugar to control burn rate and energy density. All of it over the counter. All of it accessible to someone willing to read carefully and try things until they worked.

He wasn't following a plan. He was just that kind of person.

Most people have forgotten that the Air Force had its own space program before NASA existed. NASA was carved out of NACA in 1958, but the Air Force had been running parallel efforts since the mid-1950s. That generation had grown up on science fiction and wanted to see it happen. When Sputnik launched in October 1957 the country went into a low-grade panic about whether it understood physics well enough to survive, and suddenly the kids who had been dreaming about space since they could read had somewhere to go with it. What followed was one of the rare moments in American history when technical aptitude was a genuine class elevator. The government needed people who understood this stuff badly enough to find them wherever they were.

He enlisted in his early twenties, aerospace degree in hand. The Air Force space program was what he was aiming at. He ended up working on attitude control thrusters for reconnaissance satellites, the kind that could resolve fine surface detail on Earth from hundreds of miles up. For that mission attitude control wasn't a secondary problem. It was the central one. A camera that can't hold still is useless. The thrusters are what made the intelligence possible. The underlying engineering was the same problem he had been teaching himself: oxidizer, fuel, combustion geometry, now controlled to tolerances that left no margin.

I remember him watching a satellite reenter on the cable news when I was young. I don't know which one or exactly what year. What I remember is that he cried. He told me later there was a plate on that satellite with his name engraved on it. Work he had done, hardware he had touched, in orbit for years and now gone. Grief with no adequate audience, because the context was secret and the people who would have understood were scattered across programs that didn't officially exist.

Years later my father was excited watching Iridium launch, Motorola's commercial satellite constellation, first launches 1997. The same fundamental technology, now accessible to anyone with a phone. His generation had figured out how to do this, quietly, under classification, and here it finally was in the open. The knowledge had propagated. Just not through the channels that were supposed to carry it.

He kept a green chalkboard in the garage. He would pull out his slide rule and work through things with me. Orbital decay, thrust, specific impulse, delta-v, the rocket equation and why it makes everything harder than it looks. He had a worry he came back to often - society had forgotten how to go to the moon. The knowledge existed in aging engineers and partially classified documents and it was not being transmitted. The chalkboard was what he could do about that.

Last year Destin Sandlin, an aerospace engineer who describes himself as a redneck from Alabama, walked into a room full of the most senior people in American space policy and did something worth an hour of your time to watch. He asked questions that people inside the institutional food chain had stopped asking. Starting with the most basic one: how many rockets does it take to fuel the Artemis lunar lander?

The room went quiet. Nervous laughter. EPublic estimates have varied, but all point to a strikingly high number of launches and on-orbit refueling operations before a landing attempt depending on assumptions about boil-off and reuse, and nobody in the room had a confident answer.

These are not uninformed people. A core operational parameter of their own mission architecture was not common knowledge among the people running it.

Then Destin asked the room a simpler question.

"Is this the simplest solution?"

Silence.

Destin pointed them at NASA SP-287, a document the Apollo engineers wrote and left behind specifically so the next generation wouldn't have to rediscover everything from scratch. The title is "What Made Apollo a Success." It has been sitting there, public, for decades. Most of the people in that room had not read it.

The principle at the center of that document is blunt:

"Build it simple and then double up on as many components or systems so that if one fails, the other will take over."

Simple first. Then redundant. Not complex and hoping.

Simple isn't just aesthetic preference. Simple is how you keep the system inside your head. Simple is how you build procedures all the way down to bolt cutters and still know what comes next. When a system gets complex enough that a room full of its leaders can't answer a basic operational question about it, it has exceeded the boundary of what they actually understand. They are renting the complexity along with the capability.

The Apollo engineers meant it literally. When designing the ascent stage separation, the mechanism that gets astronauts off the lunar surface, they didn't stop at one solution or two. They built redundancy on top of redundancy. Flip the switch. If that fails, go outside and trip the manual release. If that fails, depressurize, suit up, go to the bottom of the spacecraft with bolt cutters, and cut the straps holding the stages together. Harrison Schmitt said there was one more procedure after the bolt cutters. Nobody would say what it was.

That's not genius. That's a chicken farmer's epistemology applied to the hardest engineering problem humans had ever attempted. You don't wait for perfect conditions or perfect knowledge. You start simple, you build every fallback you can think of, and then you think of one more.

Destin argues that Artemis didn't follow that logic. The NRHO/Gateway architecture was publicly justified in part on communications, surface access, stability, and operational grounds, but Destin argues that it also reflects deeper architectural constraints that accumulated into a more complex solution. Destin's read, and he makes a detailed case for it, is that it's an architectural constraint dressed up as a design choice, complexity that accumulated because the real constraints couldn't be named publicly. A room full of program leaders who couldn't tell you the basic parameters of the system they were running.

That's what happens when you lose the thread.

Destin also interviewed an engineer who had worked on the lunar landing training vehicle, the machine that taught Apollo astronauts to land in one-sixth gravity by actually putting them in a vehicle where their life depended on getting it right. Destin asked whether the Apollo engineers were smarter than engineers today. The answer was no. What they had wasn't superior intelligence. It was a bias toward doing, toward simplicity, toward keeping the system inside human heads rather than delegating it to complexity they couldn't fully reason about.

NASA SP-287 exists because those engineers understood something important. Capability doesn't survive on its own. Knowledge doesn't transmit automatically. You have to codify it deliberately or it dies with the people who held it. It is ownership made explicit. Here is what we understood. Here is why it worked. Here is the playbook so the next generation doesn't have to rediscover it at the cost of lives.

The space race created a machine for turning hands-on knowledge into national capability. It found people like my father wherever they were because it needed what they had already taught themselves. It was the on-ramp, the forcing function that pulled curiosity into programs that mattered and gave it somewhere to go. That same forcing function generated SP-287, the discipline to write it down, the institutional pressure to transmit it. When the race ended the machine stopped. The on-ramp closed. The knowledge didn't vanish immediately. It aged out, program by program, engineer by engineer, panel by panel. What remained was credentials and institutional memory of having once known how, which is a different thing entirely from knowing how.

We took that gift and built a lunar return architecture that, at least in its public form, often looks more operationally intricate than the Apollo playbook would have preferred. More complex architecture. Estimates ranging from eight to fifteen or more rockets just to fuel the lander. A room full of its leaders who hadn't read the playbook.

"Is this the simplest solution?"

Silence.

That's not an aerospace problem. That's the pattern. The knowledge transmission problem is older than aerospace. I've been writing about it in other contexts for a while, starting here.

My father spent my childhood pointing at this from a chalkboard in a garage. I didn't become an astronaut. That was his hope, not my path. The chalkboard worked anyway. The knowledge moved. The Iridium launches proved it. The knowledge his generation developed under classification eventually became infrastructure anyone could hold in their pocket. You can't fully control where it lands. You can only decide whether to try.

Now AI is doing to software what the end of the space race did to aerospace. It is consuming the early career tasks that used to serve as scaffolding for building judgment. The debugging, the boilerplate, the routine iteration that taught tradeoffs and edge cases before anyone trusted you with the hard problems. The visible work disappears first. The tacit knowledge becomes unreachable just as it becomes most important. The on-ramp closes. And at some point a room full of senior people goes quiet when someone asks a basic operational question, not because they're uninformed, but because the complexity was delegated before the understanding had time to form.

That is the cautionary tale. Not that AI is bad. That capability outsourced before it is understood leaves you renting decisions you don't control while keeping consequences you can't transfer. The room goes quiet. And eventually nobody even thinks to ask whether this is the simplest solution.

My father saw it coming. That's what the chalkboard was for.

The question isn't whether you work in aerospace or software. It's whether you've stopped asking basic questions about the system you're running. Whether it has exceeded the boundary of what you actually understand. Whether you're renting complexity along with capability and calling it progress.

You don't wait for perfect knowledge. You read every playbook you can find. You build redundancy all the way down to bolt cutters. And then you think of one more thing.

The chemicals are still on the shelves. SP-287 is still public. The Destin talk is an hour of your time and worth every minute.

Read the playbook.

The WebPKI and Client Authentication Are at a Crossroads

Sat, 21 Mar 2026 20:41:51 GMT

The CA/Browser Forum is having its first serious conversation about whether publicly trusted client authentication certificates deserve their own Baseline Requirements. Nick France kicked off the discussion on the public list last week, asking for concrete use cases, and the responses so far have been a useful window into how the industry thinks about this problem. Or rather, how it doesn't.

The timing isn't accidental. Chrome Root Program Policy v1.6 is forcing a structural realignment of the WebPKI, and client authentication is caught in the middle. All PKI hierarchies in the Chrome Root Store must now be dedicated solely to TLS server authentication. Chrome stopped accepting new intermediate CA applications with mixed EKUs in June 2025, and by June 15, 2026, Chrome will distrust any newly issued leaf certificate containing clientAuth EKU from a Chrome Root Store hierarchy. Multi-purpose roots get phased out entirely. Mozilla, Apple, and Microsoft are all aligning with this direction. Every major public CA has published a sunset schedule. Sectigo stopped including clientAuth by default in September 2025, DigiCert followed in October, and Let's Encrypt is phasing it out through ACME profiles. By mid-2026, you will not be able to get a publicly trusted TLS certificate that also works for client authentication.

This is the right call. The historical practice of stuffing both serverAuth and clientAuth into the same certificate, from the same hierarchy, created exactly the kind of entanglement that makes the WebPKI brittle. The SHA-1 migration is the canonical example. Payment terminals that relied on client auth from the same roots as server certs couldn't upgrade, holding back the entire transition for years. Today, Cisco Expressway is the poster child for the same problem, using a single certificate for both server and client auth in SIP mTLS connections and scrambling to decouple them before the deadline. Dedicated hierarchies for dedicated purposes. It's a principle the WebPKI should have enforced from the start.

What to do about it

What's emerging is a clearer, more honest WebPKI, but one with a gap that nobody is cleanly addressing. If you're currently relying on publicly trusted certificates for client authentication, the path forward depends on your use case.

If the client auth is internal to your organization, VPN access, Wi-Fi onboarding, device authentication, mTLS between your own services, you should be moving to private PKI. This was always the right answer for internal use cases, and modern private CA solutions have made it far more practical than it used to be. You get full control over certificate profiles, lifetimes, and revocation without being subject to external root program policy changes. The blast radius of a private CA is contained to your organization, which is exactly what you want for internal trust.

If the client auth is between your organization and a small number of known partners, like B2B API integrations or supply chain connections, private PKI still works well. You exchange trust anchors with your partners and configure your systems to trust their specific CA. This is how most of these integrations should have been built in the first place. The "convenience" of using publicly trusted certs for this was always a false economy, because you were accidentally opening your trust boundary to every entity that could buy a cert from the same CA.

But if the client auth needs to work across organizational boundaries at scale, meaning you can't reasonably pre-configure trust anchors for every potential counterparty, this is where it gets interesting and where the current alternatives fall short. Private PKI doesn't solve this. You need some form of shared trust anchor, which is what public PKI provides for server authentication today. The question is whether a similar model can work for client authentication with properly scoped identifiers and validation methods.

The human identity case is the relatively easy part

On the CA/B Forum list, Sebastian Nielsen argued that public CAs shouldn't issue client auth certificates at all, pointing to the name collision problem. He makes a fair point, but the conclusion is too broad. I'm Ryan Hurst the security practitioner, and there's also Ryan Hurst the actor (Remember the Titans, Sons of Anarchy). A public CA asserting "Ryan Hurst" in a DN doesn't help a relying party figure out which one of us is authenticating. The DN is a vestige of the X.500 global directory that never materialized. There is no global directory. Even local directories that correspond to DN structures don't exist in any meaningful density. Identity in the WebPKI belongs in the SAN, where we have identifiers that are both globally unique and reachable.

S/MIME already handles the human case correctly. The rfc822Name in the SAN is at least unique at the time of issuance. More importantly, it's reachable. You can send a challenge to an email address and get a response. You can't send a challenge to a social security number. You can't send a challenge to "Ryan Hurst, US." The broad intent of the WebPKI is to make things reachable in an authenticated way. DNS names and email addresses fit that model. DNs do not.

Even with email, there's a temporal problem. Addresses get reassigned, domains lapse, providers recycle accounts, and throwaway addresses exist by design. CAs can't monitor for reassignment, so these are inherently short-lived assertions. The certificate lifetime is the outer bound of your trust in that binding. Broader questions around PII and auditability are really about how Key Transparency can be bolted into the ecosystem. I wrote about that previously.

There is valuable work happening in this space. Ballot SMC015v2 enabling mDLs and EU digital identity wallets for S/MIME identity proofing shows this evolving in a meaningful direction. Client authentication and signed email under S/MIME belong together. Apple has argued that emailProtection EKU should mean mandatory S/MIME BR compliance, closing the loophole where CAs omit email addresses from emailProtection certificates to avoid the BRs. I think that's the right direction. One nuance worth calling out though. S/MIME bundles signing, authentication, and encryption, and I think that's right for the first two but not the third. Signing and authentication are real-time assertions that work well as short-lived credentials. Encryption is different. The key is bound to an identifier that may not be durable, and without frequent rotation you risk bygone-SSL style attacks where a new holder of an email address could access messages intended for the previous one. The encryption case deserves its own careful treatment around key lifecycle and rotation.

Browsers are actively looking to remove client auth from TLS certificates, and I don't disagree given how poorly specified and unconstrained it has been. That signals whatever comes next needs to be much more tightly defined. The human client auth case is covered by S/MIME, browser-based client auth is on its way out for good reason, and a new working group doesn't need to revisit the human case.

The machine identity gap

Where it gets interesting is cross-organizational service-to-service authentication on the public internet. Today this is mostly handled with API keys, OAuth client credentials, or IP allowlisting, all with well-known limitations. mTLS with publicly trusted client certs could fill a real gap, but only if the identity model is built correctly.

Many current uses of mTLS with publicly trusted client certs are misplaced. Organizations are often assuming a level of assurance they don't actually get when they accidentally cross security domains by relying on the public WebPKI for what is fundamentally a private trust relationship. A publicly trusted cert for payments.example.com tells you that the entity controlling that domain authenticated, nothing more. It does not mean they are your trusted partner, your approved vendor, or anyone you intended to grant access to. Public trust gives you authenticated identity, not authorization. Organizations that conflate the two will accidentally open up access based solely on someone having obtained a client cert. The examples collected on the list so far, Cisco Expressway and EPP, are mostly legacy compatibility problems being fixed. A working group built on those foundations would produce weak Baseline Requirements.

The better foundation is the emerging need for authenticated service-to-service communication across organizational boundaries. Consider SMTP. Mail servers already authenticate to each other over the public internet using TLS, and MTA-STS is pushing that toward authenticated connections. The logical next step is mutual authentication, where the receiving mail server can cryptographically verify the sending server's identity, not just the other direction. SMTP and mTLS go together like peanut butter and jelly, but there's no clean way to do it with publicly trusted client certs today. Or consider vendor supply chains. If a manufacturer's procurement system needs to query a supplier's inventory API, or a logistics provider needs to authenticate to a retailer's fulfillment service, the options today are API keys, OAuth flows, or standing up an industry-specific trust framework just so machines can talk to each other. mTLS with publicly trusted client certs would let these systems authenticate directly, without building bespoke trust infrastructure for every partnership.

And this need is accelerating beyond any single industry. As AI agents increasingly act as user agents on the open internet, calling APIs, negotiating with services, and transacting across organizational boundaries on behalf of users, mutual authentication between machines that have no pre-established trust relationship is becoming a practical necessity, not a theoretical concern. You can't pre-configure trust anchors for every service an agent might need to interact with any more than you can pre-configure them for every website a browser might visit. I wrote about this dynamic previously, and the trajectory is clear. The machine-to-machine authentication problem on the open internet is starting to look a lot like the server authentication problem that the WebPKI was built to solve, just in both directions.

For machines, the name collision problem largely disappears. DNS names are globally unique by design. A client cert with a dNSName SAN of payments-api.example.com or registry-client.registrar.example.net doesn't have an ambiguity problem. The relying party knows exactly what organization controls that name. Nick's original question on the list asked about what parts of the DN the relying party verifies. I'd argue that's almost the wrong framing. There is no global X.500 directory. The question should be, what SAN types are needed, and what validation methods can we define for them?

For straightforward service identification, dNSName works today with no new validation methods needed.

payments-api.example.com
erp-connector.supplier.example.net
registry-client.registrar.example.com

For more expressive service identification, uniformResourceIdentifier SANs encode not just the organization but the specific service.

https://example.com/services/payments
urn:example:service:billing:v2

This URI-based approach isn't speculative. SPIFFE already uses URI SANs (spiffe://cluster.local/ns/production/sa/checkout) to represent service identities in Kubernetes mTLS contexts. The pattern is proven and widely deployed within private PKI. Extending it to public trust for cross-organizational federation is a natural evolution of an approach the industry has already validated. URI SANs can be validated through .well-known challenge methods (like ACME HTTP-01 scoped to a URI path) and ALPN-based methods, extending battle-tested ACME-era infrastructure rather than building from X.500-era assumptions.

What the industry is doing instead

Almost all the CA and vendor messaging right now says "move to private PKI." That's the right answer for internal use cases, but it doesn't address cross-organizational trust. The most interesting alternative emerging is the DigiCert X9 PKI, launched in partnership with ASC X9, the financial standards body. X9 PKI is a completely independent trust framework, governed by X9's policy committee rather than the CA/Browser Forum or browser root programs. It supports both clientAuth and serverAuth EKUs, uses a common root of trust for cross-organizational interoperability, and is WebTrust audited. It's specifically designed for the financial sector's mTLS needs, though they're expanding to other sectors.

X9 PKI is essentially a "public PKI that isn't the WebPKI" for service-to-service auth. It validates the premise that there's a real need for cross-organizational client authentication with a shared trust anchor. But it's sector-specific and governed outside the CA/Browser Forum, which means it doesn't solve the general case. The EU's eIDAS QWAC framework is another sector-specific approach. These are workarounds for the absence of a general-purpose, properly scoped public client auth certificate type.

If this moves forward

I'm not advocating for or against a working group at the CA/Browser Forum. But if the Forum does decide to take this on, the scope needs to be narrow IMHO. Machine and service client auth only, with identity in the SAN using dNSName and uniformResourceIdentifier. DN fields should not be relied upon for authentication decisions. Validation methods should build on existing domain control mechanisms. Human client auth stays in S/MIME where it belongs. The BRs should address the authentication versus authorization distinction explicitly, so relying parties understand that a publicly trusted client cert tells them who is connecting, not whether that entity should be granted access. This is already how server certificates work, and client auth should follow the same model. And the issuing CAs need to be dedicated, separate from server auth hierarchies. The SHA-1 payment terminal debacle, the Cisco Expressway mess. Every time client and server auth are entangled in the same hierarchy, one use case holds back progress on the other. Don't repeat that.

The bigger picture

What we're watching is a structural realignment of the WebPKI's purpose. The WebPKI is being narrowed to mean "TLS server authentication for web browsers," full stop. Everything else, client auth, S/MIME, code signing, is being pushed to dedicated hierarchies, private PKI, or alternative trust frameworks. That's mostly the right direction. But the service-to-service authentication gap is real, growing, and not well served by any of the current alternatives. Private PKI doesn't solve cross-organizational trust. X9 PKI is sector-specific. The CA/Browser Forum has the institutional knowledge, the validation infrastructure, and the trust framework to define something that works here. Whether they choose to is another question.

The conversation is happening now on the public list. If you have concrete use cases for cross-organizational service authentication with publicly trusted client certificates, this is the time to share them. The shape of what comes next depends on whether the use cases justify the effort, and right now the list is thin.

Introducing the WebPKI Observatory

Sun, 15 Mar 2026 09:34:47 GMT

For as long as I have been in this industry, the WebPKI compliance conversation has run on impressions. People with long memories and regular conference attendance have built up a picture of which CAs are well-run, which are struggling, and where the oversight gaps are. That picture has generally been accurate. It has also been almost entirely unmeasured.

The WebPKI Observatory at webpki.systematicreasoning.com, a project from Systematic Reasoning, is an attempt to change that. It's a public dashboard covering 1,690 compliance incidents drawn from Mozilla Bugzilla between 2014 and 2025, cross-referenced with CCADB membership data, certificate issuance volumes from CT logs, root program trust store compositions, and the complete history of CA distrust events. The goal was simple: replace the shared intuition with actual data, and see what the data shows that intuition missed.

Some of it confirmed what most people in this space already suspected. Some of it was genuinely surprising.

The finding that reframes everything else is detection. When a compliance incident occurs, who finds it? Root programs find 52% of incidents. Automated external tools — CT log monitors, certificate linters, community scanning infrastructure — find 14%. CAs find their own problems in 9% of cases.

That number deserves more attention than it typically gets. One in eleven. CAs have full access to their own issuance systems, their own audits, their own CPSs, their own disclosure obligations, and they are the least effective detection mechanism in the ecosystem. External parties without any privileged access outperform internal CA monitoring by a factor of six or more. The compliance monitoring function has been effectively outsourced to external parties by default, and mostly without anyone deciding that was the right architecture.

Everything else in the data follows from that.

The failure classes that have grown are instructive. Technical misissuance has declined as a share of incidents over the past decade. What has grown is the process layer. In 2019, governance failures represented 21% of all incidents. By 2025 that figure was 60%. Policy violations, CPS failures, disclosure deadline misses. These are by definition things internal compliance programs should be catching. The 260 incidents tagged policy-failure or disclosure-failure in the dataset are a direct indictment of internal compliance operations. A CA that violates its own documented policy is not being surprised by an external attacker.

The oversight picture is also worth examining. In 2017, Mozilla engaged with 79% of Bugzilla compliance bugs. Chrome had no formal root program yet and was near zero. By 2025 the picture had reversed and degraded simultaneously. Chrome now contributes the dominant share of oversight engagement but covers only 18% of incidents. Mozilla covers 8%. The total corpus has roughly doubled since 2017 while combined meaningful oversight coverage has fallen by two-thirds. The Chrome Root Program launched in 2021, and its effect on the governance landscape is visible in the data — Chrome has made 239 substantive oversight comments in recent years versus Mozilla's 158 over the same period. The center of gravity in CA compliance governance has shifted to the browser with 78% market share. That is structurally significant. Microsoft, which operates the largest trust store by root count at 346 trusted roots, has made zero recorded governance comments across all 1,690 incidents spanning 11 years.

The distrust history is also clarifying. The common mental model is that CAs get removed for catastrophic technical failures. The data does not support that model. 14 of 16 distrust events involve compliance operations failures. The behavioral taxonomy matters, negligent noncompliance, willful circumvention, demonstrated incompetence, and argumentative noncompliance. In 10 of the 16 cases, the distrust event was preceded by a documented pattern of prior incidents. The median runway from the first incident to distrust is 3.2 years. The failures were not hidden. They were in Bugzilla the whole time. The CA just was not resolving them systematically.

That means distrust is largely predictable given sufficient data. The indicators show up well before the outcome. That is a sobering observation about past oversight and a useful one for anyone thinking about what the compliance monitoring function should actually do.

The Observatory is a measurement tool, not a verdict. The dataset has limits — Bugzilla under-represents incidents that never reach public disclosure, CT-derived issuance volumes reflect only unexpired certificates at the time of measurement, and the behavioral taxonomy applied to distrust events involves judgment calls. But the patterns are robust enough to be useful.

For CA operators, the detection data alone should prompt hard questions about internal monitoring coverage. For root programs, the oversight gap data quantifies a scaling problem that is currently being absorbed by Chrome without anyone having explicitly decided that is the right architecture. For the policy community, the shift from technical to governance failures as the dominant incident class has direct implications for what audit frameworks should actually measure.

The dashboard is live at webpki.systematicreasoning.com, updated daily. The methodology is documented. Pull requests are welcome

Signed, Auditable, Offline-Tolerant, PQ Secure QR Codes

Sat, 14 Mar 2026 19:13:24 GMT

Signed, Auditable, Offline-Tolerant, PQ Secure QR Codes

A few months ago I wrote about what it would take to make a QR code verifiable in a post quantum world. In this post I wanted to explore what it would look like if we wanted one that is genuinely verifiable, not just signed, but auditable, offline-tolerant, and ready for a post-quantum world. That post was mostly conceptual. A conversation with Bruno Couillard last week nudged me to put down the thoughts I had been carrying about exactly that.

The design draws heavily on the draft for Merkle Tree Certificates, which is working through the IETF right now. MTC is aimed at TLS, but the core insight is that you can replace per-certificate signatures with compact Merkle inclusion proofs against a periodically updated signed root, and that insight translates directly to QR codes once you think carefully about the offline constraint. If you haven't read it, the draft is at datatracker.ietf.org/doc/draft-davidben-tls-merkle-tree-certs.

The result of applying that idea to the QR problem is MTA-QR, a working implementation of what I've been calling Merkle Tree Assertions for QR codes. The demo is live at mta-qr.peculiarventures.com, and the full source is at github.com/PeculiarVentures/mta-qr-demo. There are Go and TypeScript implementations, a browser-only demo that generates and verifies without any backend, and an interoperability test matrix that exercises all three signing algorithms against both runtimes in every combination.

To be clear, this isn't a production-ready library, but building it helped me identify things I had missed while whiteboarding it in my head.

The size problem is real but solvable

The original post flagged signature size as the central constraint. An ML-DSA-44 signature is 2,420 bytes. A Version 40 QR code at medium ECC holds about 1,273 usable bytes. Those two numbers don't fit in the same sentence without a solution.

The solution is separating what goes in the QR from what you need to verify it. The QR carries the assertion content, a Merkle inclusion proof, and coordinates pointing to a signed checkpoint. The checkpoint itself contains the issuer signature, lives outside the QR, and gets cached on the verifier's device, typically during a charge cycle before the device ever sees a QR code. Once cached, verification is fully offline.

The proof is the interesting part. A two-level tiled Merkle tree, with an inner batch tree and an outer parent tree, caps the total proof at eight hashes regardless of how large the log grows. Eight hashes is 256 bytes. That's the ceiling, forever. The QR version stays fixed. The code never gets denser as the issuer accumulates millions of entries.

In practice, a Mode 1 QR carrying bearer claims and a Merkle inclusion proof fits comfortably within a Version 10 to 15 code at medium ECC, well under 500 bytes total. ML-DSA-44 doesn't appear in the QR at all. The issuer signature lives in the checkpoint that the verifier fetched during its last charge cycle.

ML-DSA-44 won't fit in a single QR in Mode 0, the fully embedded mode where the signature is in the QR itself. Mode 0 is the bootstrap mode: it works on air-gapped verifiers, on paper QR codes printed before any checkpoint infrastructure exists, and for scenarios where prefetch is operationally impractical. It's not a niche failure case; it's the starting condition for any new deployment. Mode 0 with PQC will require waiting for NIST to finalize smaller-signature algorithms, or accepting larger QR codes. Mode 1 is the practical path to PQC today.

Offline tolerance is mostly a framing problem

There's a habit of treating offline verification as binary, either the device has connectivity at scan time, or it doesn't. That framing creates a false constraint.

Every verifier with a battery has a window where it is stationary, connected, and idle. That's when it charges. Fetching a checkpoint during a charge cycle is trivially cheap compared to everything else happening during that window. The relevant question isn't whether the device has connectivity at scan time. It's whether the assertion being scanned was issued before the verifier's last checkpoint fetch.

For the common case, the answer is yes. A concert ticket issued last week, a prescription filled this morning, a badge issued at enrollment, all of these predate the verifier's cached checkpoint by hours or days. Verification is fully offline because the relevant checkpoint was already there.

The narrow failure case is an assertion issued and scanned within the same charge cycle, before any checkpoint fetch. That falls back to a single cache-miss network call, which then covers every subsequent scan of the same batch. One round trip, then fully offline for the rest of the operational period.

Witnessing is where the transparency guarantee actually lives

The issuer's signature proves the assertion came from a specific key. That's useful, but it doesn't prevent a compromised issuer from presenting different views of the log to different verifiers. Split-view attacks are subtle and hard to detect after the fact.

Witnesses solve this. A witness cosigns a checkpoint only after verifying it extends the previous one they saw, establishing a consistency guarantee across the full history of the log. Once multiple independent witnesses have cosigned a checkpoint, the issuer cannot retroactively rewrite or fork the log without those witnesses catching it.

The witness protocol comes from c2sp.org/tlog-cosignature, the same infrastructure underpinning the transparency.dev witness network. I worked on that witness network during my time at Google, so it was never far from my mind when designing this. Connecting MTA-QR to it means the issuance of every assertion can be monitored by parties with no relationship to the issuer. That's the difference between a signed QR and an auditable one.

The implementation uses Ed25519 for witness cosignatures regardless of what algorithm the issuer uses for checkpoints. That's not a design choice I made, it's what the spec requires. It means an issuer can use ML-DSA-44 for the checkpoint signature while the witness infrastructure stays on stable, widely deployed Ed25519 keys. The two concerns are separated cleanly, and that separation matters. The quantum threat to the issuer signature and the operational threat to the witness network are different problems on different timelines.

What I had wrong in the original post

The earlier post mentioned UOV and SQISign as especially promising for QR codes because of their smaller signature sizes. That framing isn't wrong exactly; smaller signatures do help with the size constraint, and both algorithms are genuinely interesting work. But the NIST competition covering them isn't finished, which means neither is practical for anything you'd want to deploy or standardize against today. More importantly, once you separate the checkpoint from the payload, signature size matters only for the checkpoint, which isn't size-constrained anyway. The Merkle structure removes the problem that UOV and SQISign were addressing. They may still have a role in Mode 0 once the standards are settled, but they're not the lever that makes the design work.

What's still missing

The spec has a revocation mechanism based on index ranges that a verifier checks at scan time, but the format for distributing and authenticating those revocation lists isn't fully defined yet. This is the most operationally significant open item. An unsigned revocation list is vulnerable to a stale-list attack at the network layer. An adversary who can delay or suppress list delivery can extend the validity of a revoked assertion. The natural fix is issuer-signed lists using the same key that signs checkpoints, but that format isn't written yet. Until it is, revocation is a weak link in any deployment that takes revocation seriously.

Type 0x02 key assertions, where the QR proves possession of a private key rather than just embedding bearer claims, are defined in the log entry format but the challenge-response protocol isn't specified. Two implementations can't interoperate on key assertions without it.

The C2SP tlog-checkpoint format needs registrations for ECDSA and ML-DSA before those algorithms can interoperate with standard tlog-checkpoint parsers. Ed25519 is fully specified today. ECDSA and ML-DSA work in the reference implementation but aren't interoperable with external tooling yet. This is a practical blocker for adoption by anyone not using the reference implementation, and it's the right next conversation to have with the C2SP and MTC communities.

Try it

The browser demo runs entirely in-page with no backend. It generates Ed25519 or ML-DSA-44 keys in your browser, issues assertions, builds the Merkle tree, produces QR codes, and runs the full 15-step verification trace. The tamper panel lets you flip proof bytes, corrupt the TBS, zero the proof, or truncate the payload, and watch exactly which verification step catches each failure. It's a useful way to build intuition for what the protocol is actually checking and why each step is there.

The repo is at github.com/PeculiarVentures/mta-qr-demo. Pull requests welcome, especially on the open items.

When Compliance Records Become the Only Honest Signal

Mon, 09 Mar 2026 06:31:27 GMT

I've been spending a lot of time lately building Systematic Reasoning with my long-time friend Vishal. The core premise is straightforward. Organizations reveal their true operational character through how they design to prevent failure, how they plan to handle it when it happens, and how they actually do. That signal deserves to be tracked, structured, and acted on. We're building an agentic compliance platform to do exactly that.

Systematic Reasoning won't be limited to any single domain, but we decided to start with the Web PKI. The reasoning was simple. It's high impact in a way that's hard to overstate. Every internet user depends, whether they know it or not, on a relatively small number of Certificate Authorities getting things right. The margin for error is zero. If that trust layer breaks, it breaks for everyone.

DigiNotar is the canonical example. A small Dutch CA, compromised so thoroughly that attackers could impersonate any website on the web, and did. That capability was used to spy on Iranian dissidents, intercepting communications that people believed were private and secure. The trust infrastructure that was supposed to protect them was turned into a weapon against them. DigiNotar isn't an edge case or a cautionary tale from a more naive era; it's a demonstration of the actual ceiling of what can go wrong. And it isn't the only one. State-affiliated certificate authorities have been caught performing man-in-the-middle attacks on their own citizens' traffic, something the Baseline Requirements explicitly prohibit, but prohibition only matters if it's enforced. The web's trust model works right up until the moment someone decides it's more useful as surveillance infrastructure.

At the core of Systematic Reasoning, is a belief I've held for a while. Compliance can be a vital sign of organizational security, but only if it's continuous. The reality today is that it isn't. Code ships daily. Audits happen annually. The gap between those two rhythms is where things go quietly wrong.

I've written before about why I have limited faith in the current audit regime. Auditors are engaged by the organizations they assess. Their product is a clean seal; their incentive is to keep the client. They operate on point-in-time sampling with auditee-selected scope, and they're often compliance professionals rather than engineers, which means they're checking whether a policy exists more than whether the system actually behaves correctly. That's if you're lucky. Sometimes the audit is scoped against a version of the Baseline Requirements that was superseded over a year ago.

The same incentive shapes how certificate authorities write their governance documents. A CP/CPS that relies heavily on incorporation by reference, that omits specifics about what the organization actually does and what constraints it operates under, is easier to audit against than one that makes precise, testable commitments. Vagueness isn't always carelessness. Sometimes it's a design choice. The same thing happens in incident reports. A report that attributes a failure to "organic process evolution" or "human error" without describing the actual control gap is easier to close than one that names the broken system and commits to a specific fix. In both cases the document gets the box checked without creating accountability. References establish authority. Commitments establish accountability.

The audit gap isn't compensated for by strong internal monitoring either. The majority of significant compliance failures are not caught internally. They are caught by external researchers, root program staff, or community tooling. A broken validation endpoint runs for five years and the organization finds out because someone posted a 404 error in a public issue tracker. A validation race condition exists undetected for seven and a half years not because it was well hidden but because nobody was looking. The absence of an internal alarm is not evidence that the system is healthy. It is often evidence that the monitoring itself is missing.

So public incident reports and governance documents become some of the most signal-rich material available. Policy documents tell you what an organization claims it will do. Incident reports tell you what happened when reality diverged from that claim. Together they create a longitudinal picture that neither document produces alone.

Building a system to reason over that data surfaced a problem I didn't fully anticipate. When you're working from the outside, with no access to internal systems and no way to verify what actually changed, the public record is almost all you have. The question isn't whether to treat it with skepticism. It's how much skepticism to build in by default.

The temptation is to give the benefit of the doubt. Organizations are required to describe the blast radius of an incident. Not every localized bug is a symptom of something systemic. But accepting minimizing language at face value is its own failure.

"Only" is doing a lot of work when the bug it's describing went undetected for seven and a half years. "No compromise of end-entities" is doing a lot of work when what it really means is that nobody found the gap before you did. Framing survival as security isn't reporting, it's PR. And if an organization believes an incident is no big deal, you can predict with reasonable confidence that the root cause analysis will be shallow and the remediation will be a band-aid.

ForgeIQX, our first offering, tracks those signals longitudinally across both policy documents and incident reports. Not to prosecute organizations for their language choices, but to notice when a commitment made in a CP/CPS quietly disappears in the next version, or when a promised fix is nowhere to be found when the same failure mode surfaces years later. That's commitment decay, the slow evaporation of a promise made under pressure, and it's only visible if you're tracking across multiple documents and incidents over time rather than treating each one in isolation.

The calibration problem is real and doesn't have a clean answer. Get it wrong in one direction and you build a system that cries wolf. Get it wrong in the other and you build a system that launders PR-speak into clean signals, which is just automating the thing we already do too much of.

There's a third failure mode that took me longer to see. A system like this can be gamed. Swap "we got lucky" for "our monitoring detected no active exploitation." Replace "only thirty certificates" with a more clinical impact scoping statement that says the same thing in language that sounds like engineering rigor. The words change; the institutional posture doesn't. A system that can be satisfied by better prose isn't measuring operational maturity, it's measuring communications sophistication.

That means the system has to be built with structural pessimism. Not cynicism for its own sake, but a deliberate prior that clean language is not the same as clean operations, and that the absence of red flags is not the same as the presence of green ones. We can't verify that an organization fixed what it said it would fix. What we can do is watch whether the same failure mode surfaces again and whether the pattern of shallow root cause analyses continues or breaks. The historical record doesn't tell us what's true inside these organizations. It tells us what they were willing to say in public, under pressure, over time. Given the alternatives, that may be the most honest signal available.

A certificate authority with genuine operational maturity should want this kind of scrutiny applied to itself. Not because it will always produce a clean result, but because it surfaces the gaps before an external party does. ForgeIQX gives organizations a way to continuously monitor their own compliance posture, so their practices and code keep pace with their commitments. The same is true for auditors who want their findings to mean something beyond a checkbox. The problem with the current regime isn't that the people in it are careless. It's that the incentive structures don't reward rigor, and the tooling to demonstrate it continuously doesn't exist. That's what we're building.

The Web PKI is where we started because the stakes are concrete and the public record is unusually rich. But any regulated industry where compliance is measured annually, where governance documents are written to satisfy auditors rather than inform relying parties, and where incident reports are drafted with one eye on legal exposure, has the same gap between what the paper says and what the organization actually does. We started here. We don't intend to stop here.