Category Archives: Uncategorized

The Status Quo Outlived Its Status

In security we like to say that the problems live in the gaps between systems. Each system, on its own, is usually coherent. It has a threat model, invariants, and someone who owns it. The seam between two systems is owned by nobody, and each side quietly assumes the other is handling the thing that neither is. The load balancer assumes the backend validates. The parser assumes the canonicalizer normalized. The audit covers the software but not the network it runs on. Attackers don’t have to beat either component. They just have to find the assumption neither side wrote down. The attacker gets to pick the threat model, and they pick the one that lives in the seam.

Once you see this pattern, you see it everywhere, and not just in security.

Quality Lives in the Gaps of Ownership

In software quality, the same topology produces a different failure class. Security gaps produce exploits. Ownership gaps produce jank.

A user’s journey through a product is inherently horizontal. They sign up, configure, use, get billed, get help. But ownership is vertical, carved along team boundaries. So the experience degrades precisely at the handoffs. The onboarding flow owned by one team dumps you into a product owned by another, with terminology that doesn’t match, settings that don’t carry over, and an error message that references a concept from a third team’s domain model. Every screen passed its own review. The journey never got one, because the journey has no owner.

This is the same root cause as the security version. Contracts between components are written in terms of what each side provides, not what the whole must feel like or withstand. Functionality composes. Quality attributes don’t. Not security, not usability, not performance, not consistency. Those are emergent properties of the composition, and emergent properties are exactly what per-team accountability structures can’t see. A dashboard that takes eight seconds to load is usually five services each meeting their SLO.

You Ship the Org Chart in N Dimensions

Conway’s law is usually quoted as a statement about architecture, that organizations design systems which mirror their communication structures. But architecture is just the most legible projection. The org chart also manifests in the security posture, where trust boundaries land wherever the reporting lines do. It shows up in the latency profile, where every org boundary becomes a network hop plus a queue plus a retry policy. It shows up in the data model, where the same “customer” is defined four ways because four VPs own four systems. It shows up in the compliance scope, where audits map to cost centers rather than to where the risk actually lives. It even shows up in the documentation, where each team documents its interior and nobody documents the crossings.

You don’t ship your org chart once. You ship it in every dimension at the same time.

And it’s stickier than the chart on the wall, because the formal org chart is only the visible part. The real partitioning is cultural. It is who trusts whom, which teams have history, where the scar tissue from the last reorg sits, and who won the last budget fight. Systems calcify around those boundaries. That’s why reorgs so rarely fix seam problems. You can redraw the chart in a day, but the shipped artifact embodies the org chart as it existed at every point in the system’s history. Legacy code is really legacy org structure. You’re maintaining the fossil record of decade-old turf wars, and the team that could explain a given seam disbanded three reorgs ago.

Bureaucracy Is the Fixative

Here’s where it hardens. Process is how organizations serialize distrust between units. Every approval gate, every ticket queue, every review board is a treaty boundary between fiefdoms, and treaties optimize for non-aggression, not for the emergent properties of the whole.

The bureaucratic instinct when a seam fails is to add process at the seam. A checklist, a sign-off, a form. This papers over the gap without giving it an owner, and now the seam has a compliance artifact defending its existence.

Which brings us to the uncomfortable part. Bureaucracy defends the status quo long past the point where the status quo lost its status. Not out of malice, and usually not even out of preference. The mechanism is simpler and more forgivable than that. Process is memory without comprehension.

Every rule is a compressed lesson. Some incident happened, someone got burned, a control was born. But the compression is lossy. The rule survives while the context doesn’t. The organization keeps executing the answer long after everyone who understood the question is gone. It’s Chesterton’s fence, except nobody can find the fence’s author, the field it enclosed is now a parking lot, and there’s a Fence Compliance team whose headcount depends on the fence.

That’s the inversion point. Controls that began as instruments become constituencies. A process accretes staff, tooling, budget, an annual review cycle. It stops being a means and becomes a stakeholder. And stakeholders defend themselves.

Asymmetric Bookkeeping

The genius of bureaucratic self-defense is that it never has to argue the status quo is good. It only has to make change expensive.

Every proposal to remove a control gets evaluated by asking what risk removal creates. Nobody asks what risk retention creates. The cost of the existing process is denominated in currencies the review process can’t count, things like velocity, morale, and opportunities that quietly went elsewhere, while the cost of change is denominated in the one currency it’s built to count. With bookkeeping that asymmetric, the ratchet only turns one way.

There’s a reliable tell for when status is lost but the defense continues. The justifications go circular. Ask why we do this and the answer stops referencing a threat or an outcome and starts referencing the process itself. It’s required for the audit. It’s policy. That’s the template. When a control’s referent is another control, you are no longer managing risk. You are maintaining a liturgy. And liturgies are stable. That’s what they’re for.

The people defending the liturgy usually aren’t cynics, either. Institutions promote the people who thrived under the current rules, which means the people with the authority to change the system are precisely the ones whose careers validate it. They don’t defend the status quo because they’ve weighed it and found it good. They defend it because it’s the ladder they climbed, and it’s genuinely hard to see your own ladder as arbitrary. The system doesn’t need guards. It manufactures believers. That’s why correction so often comes from outside, from a competitor, a collapse, or a technology that routes around the institution entirely, rather than from reform. Reform requires the institution to metabolize the idea that its own selection function is the problem, which is roughly asking the liturgy to audit itself.

Deletion Has No Constituency

So what do you do about it? Two things, and both are harder than they sound.

First, admit that the seams are the system, and staff them accordingly. You can’t fix an n-dimensional Conway problem with a one-dimensional intervention. A design system fixes the UX projection. A service mesh fixes the network projection. A GRC tool fixes the audit projection. But the generator is the accountability topology itself, so the pathology just re-expresses in whatever dimension you didn’t treat. The only durable moves are changing the human topology, which is rare, painful, and temporary, or forcing the interfaces to be explicit, adversarially specified, and owned as products. That’s the real lesson of the Amazon API mandate. It wasn’t about services. It was about giving the gaps owners.

Second, build a decay function. Controls get created by incidents, which are vivid and have advocates. Control removal has no incident, no advocate, no ceremony. The beneficiary of deletion is diffuse while the loser is a specific person in the room. So organizations accumulate process the way arteries accumulate plaque, one reasonable deposit at a time. Sunset clauses, zero-based process reviews, and deletion treated as a first-class ritual with the same ceremony as launch are ideas everyone nods at and almost nobody funds. The organizations that stay fast are the ones that treat removing a rule as an achievement, not an admission.

Because the status quo isn’t defended because it won an argument. It’s defended because it’s the null hypothesis, and the burden of proof only ever runs one direction. Every so often, you have to flip the burden and make the process re-justify itself in terms of an outcome rather than another process. If it can’t, it isn’t protecting you anymore.

It’s just protecting itself.

A CA That Produces Evidence, Not Promises

In my last post I argued that high-assurance systems should stop asking to be trusted on the basis of institutional promises and start producing verifiable runtime evidence about what actually happened. This post is the worked example. A certificate authority built that way, what choices it forced, and what is and is not done yet.

When I was at Google I got to work a bit with the BeyondCorp folks. What most didn’t understand is that the BeyondCorp Google used internally was substantially different from the BeyondCorp they launched to customers. On Windows and Linux, the path was TPM-backed device credentials associated with each machine, turning possession of the laptop and an authenticated credential into another factor. The deployment timelines varied by platform, but the important point was the architectural shape hardware-bound device identity became part of the access decision.

You will notice I didn’t mention Macs. That’s because Apple, although it had a similar secure processor on its devices, did not give customers attestations over keys stored inside it. We could put keys in the Secure Enclave. We could not prove to a third party that we had. For years we tried to get Apple to change that. Eventually they did, and what they shipped was not what we asked for.

We didn’t get the ability to put arbitrary keys in the enclave under our control with attestation. We got a device-bound credential signed by Apple that let us verify, at enrollment time, that a request was coming from one of our devices. We used that as a bootstrap to enroll a short-lived credential that the OS stored and used for day-to-day authentication. Apple’s attestation answered the which device question. Our short-lived credential answered everything that came after.

That worked. The standardization piece is draft-ietf-acme-device-attest, an IETF working group document I co-author, which lets the same ACME flow carry an Apple Managed Device Attestation statement, a TPM key attestation, or a YubiKey assertion without the CA needing to special-case each platform. Apple’s adoption was what unlocked normalizing on a single way across the fleet to authenticate these devices.

That normalization is the relying-party side of the credential story. The hard-edge property, this key lives on this specific device, signed by a chain you can verify back to the manufacturer, is now what we expect from a workload, a laptop, a phone, a passkey, a SPIFFE workload identity. MFA was the first compensating control for the weaknesses in passwords and API keys and bearer tokens. Passkeys, SPIFFE, and certificate-based zero-trust segmentation are the structural answer. We replaced the secret-you-know with a key-you-hold and bound it to hardware where we could.

Google’s internal production systems run the same shape, with Titan chips as the foundational substrate for the devices that get on the network for internal and cloud workloads.

That shift is well underway on the side of the wire that uses credentials.

The side that issues them is still mostly software on a server with an API key and a SOC 2 report.

If issuance hasn’t kept up with what we now expect from credential holders, what would catching up actually look like? The previous post made the general argument. The CA should be asked to produce evidence, not to be trusted. This post is what that actually looks like when you build it. A certificate authority where the issuance path itself is the evidence, where the policy that fired is part of what was measured, where the key release is gated on the measurement of the binary asking for it, and where every issued certificate is accompanied by a portable bundle a relying party can verify against trust anchors they already hold.

The architecture is designed to run on both AWS and Google Cloud. On AWS, enclave-based deployments use Nitro Enclave attestations, while VM-level deployments can use NitroTPM-backed evidence for measured boot, instance identity, and workload state. On Google Cloud, Confidential VM deployments use AMD SEV-SNP or Intel TDX attestation for the protected execution environment, with Cloud HSM as the key custodian. Where the system needs VM identity, boot-state evidence, or platform posture outside the confidential-computing attestation itself, it can also use vTPM-based evidence.

That breadth is intentional, and I’ll come back to why. But the real load-bearing architectural choice is not which cloud, HSM, enclave, VM, or attestation primitive we use. It is the shape of the issuance system inside each trust boundary.

Two components, one evidence chain

The split is structural, not procedural.

In a single-binary CA, policy enforcement and signing authority are separated by code paths inside one process. A vulnerability in the policy evaluator is a vulnerability in the signing authority, because they share an address space. The compartments exist in the source code. They do not exist at runtime.

Here, the compartments are real. The architecture splits issuance into two attested components.

The registration authority receives the certificate request, resolves identity from authoritative sources, evaluates the issuance policy against the requester’s posture and attestation, and produces a signed authorization context. The signing oracle holds the path to the CA’s signing key and produces the signature. Each runs in a separate attested environment. Each is measured separately. Each has its own keys. Both are written in Go. Memory-safe in normal code, reproducible without ceremony, and a standard library that already covers most of what a CA needs.

The RA and the oracle are different binaries, with different measurements, with different keys, on different network endpoints, joined by mutually-attested TLS where each side has independently verified the other’s measurement before either will talk. A compromise of the RA does not give the attacker the signing key, because the signing key is not in the RA’s address space and is not reachable from the RA’s network position. A compromise of the oracle does not give the attacker the policy evaluator’s identity providers, because the oracle has no identity providers wired into it. The interface between them is narrow, signed, and replay-protected.

This is the cross-machine version of the kernel-userland boundary. The kernel does not trust userland’s claim that a syscall is authorized. It does the check itself, every time. The oracle does the same thing for the parts of issuance it can verify independently.

What this buys is a bounded blast radius on a compromised RA.

If an attacker takes over the RA, including the RA’s signing key, the damage is bounded only to the extent that the oracle requires independently verifiable evidence for the facts that matter. For profile authorization, key binding, replay protection, RA authorization, and certificate structure, the oracle can check those facts locally.

For domain control validation, the same is true only when the evidence is cryptographic or independently corroborated, such as DNSSEC-validatable DNS evidence or signed observations from independent multi-perspective validators. The current implementation does not yet do either of these things, but adding support for them is a straightforward extension of the model. Without that, however, an RA compromise can still become a validation compromise.

That distinction matters. The oracle does not make an RA trustworthy. It makes the RA’s assertions conditional. Where the RA presents verifiable evidence, the oracle can check it. Where the RA presents only its own statement about what it observed, the oracle can enforce structure, freshness, profile policy, replay protection, and RA authorization, but it cannot turn that statement into ground truth.

The architectural property is not that the oracle magically knows every fact the RA observed. It is structural separation plus independent verification wherever the fact is independently verifiable. Every other property the rest of this post will describe, measured policy, gated key release, per-operation attestation, portable evidence, depends on that boundary.

What I mean by evidence

Before walking through the attestations, it is worth being precise about the word evidence.

A raw measurement is a narrow statement, this binary had this digest, this key was generated by this HSM, this quote was signed by this platform, this public key matches the attested key. Those statements matter because they are the hard edge of the system.

But they are not enough to explain issuance. A certificate is issued because policy evaluated a set of facts and authorized a specific profile for a specific requester at a specific time.

So evidence here means the decision record, the raw attestations where they can be disclosed, the verifier results, the policy digest, the profile binding, the facts the policy evaluated, the signed authorization context, the oracle’s per-operation attestation, the custodian’s key-release evidence, and the transparency proof.

draft-ietf-acme-device-attest helps with the requester-side device and key attestation. The CA-side runtime evidence chain is still built from platform-specific TEE, TPM, HSM, KMS, and transparency-log evidence. The point is not that one standard covers all of it. The point is that the CA should preserve the evidence chain instead of collapsing it into a one-bit pass/fail result or a promise.

What each attestation lets you verify

Cross-checking only matters if the things being checked are concrete. Four attestations cross the issuance flow, each one produced by a different party, each one letting the next party check something specific. It is worth walking through what each one actually lets you verify before going further.

The client’s attestation

What the RA verifies: that the requestor controls the private key whose public half is in the certificate request, that the key lives on a specific piece of hardware, that the hardware has a manufacturer-vouched identity, and that the key was generated under conditions the policy can reason about.

The shape of the attestation changes between platforms. A TPM-bound key on a Windows or Linux machine, a Secure Enclave key on an Apple device, and a key in a YubiKey’s PIV slot each arrive in a different format with a different signing chain and different platform-specific fields. The function is the same. Each one carries a binding between the key in the CSR and a piece of hardware, an identifier for that hardware, the conditions under which the key can be used, and a certificate chain back to a manufacturer the policy can decide whether to trust.

draft-ietf-acme-device-attest is the wire format that carries any of these in the same ACME flow, so the CA doesn’t need to special-case each platform. What the policy sees in the end is the same five things regardless of which platform produced them. Does the requestor control the private key. Is the key on a piece of hardware. Whose hardware. Under what conditions can it be used. Does the manufacturer chain trace back to a root the policy will accept.

The RA’s environment attestation

What anyone holding the document can verify: that the RA ran a specific measured image inside an attested execution environment, that the cloud provider signed off on the measurement, that the workload identity or role is the one the deployment expects, and that the public key bound to that measured environment for the rest of its lifetime is the one in the document.

AWS Nitro, AMD SEV-SNP, and Intel TDX each produce a different document, but the load-bearing contents are the same: a measurement of what was loaded, an identifier for the platform that produced it, the operating context, and a vendor chain back to a root the relying party can verify.

Worth being explicit about what the document does not let you verify. It does not tell you who deployed the measured environment, who runs the cloud account, or what the operator intended to run. It tells you what was measured. The operator’s intent is a separate question, answered by the published image list and the policy that names which measurements are acceptable. A relying party who trusts the hardware vendor’s chain can verify the measurement. They still have to verify, separately, that the measurement is one they should accept.

The oracle’s environment attestation

What it lets the RA verify, during handshake: that the oracle the RA is about to send an authorization context to is running the published image, and that the signing key the oracle will use for its half of the mutual TLS is the one bound to that image at boot.

What it lets a relying party verify, after issuance: the same thing, plus one additional binding. The oracle produces a fresh attestation for every signing operation, and the attestation is bound to the certificate that operation just produced and to the RA on the other end of the conversation that authorized it. Where the RA’s boot-time attestation carries the long-lived public key the measured environment will sign with, the oracle’s per-operation attestation pins this specific certificate to this specific oracle measurement with this specific RA.

The difference between boot-time and per-operation attestation is the difference between “the box looked right when it started” and “the box looked right when it did the thing you actually care about.” Boot-time attestation is what most confidential-computing deployments do today. It tells a relying party the deployment was valid at startup. It tells them nothing about whether the deployment was still valid five hours later when an actual issuance happened. Per-operation attestation closes that gap.

The custodian’s attestation

The CA private key is not loaded by the operator. It is held by a custodian, an HSM with discrete-silicon attestation, a cloud KMS that gates use on attestation, or another measured execution environment. The custody model matters. In the HSM case, the CA key is non-exportable. The signing oracle never receives the private key. It sends a signing request to the HSM, and the HSM signs only when the relevant policy, authorization, and attestation conditions are satisfied. In the software-protected or KMS-protected case, the key, or the key-encryption material needed to use it, may be wrapped so it is usable only inside a signing oracle whose attestation matches a published image.

What the custodian’s evidence lets a relying party verify depends on that model. For an HSM-held key, the evidence is not that the key was released. It is that the non-exportable key was used by a particular custodian, under particular firmware, configuration, policy, and authorization conditions. For a wrapped software or KMS-backed key, the evidence can show that key material was made usable only because the oracle’s measurement matched a measurement on the published list. Either way, if an operator runs a different binary, however benign the reason, the signing path fails. The dragon’s teeth around the data center are still there. They no longer have to do the whole job.

For a concrete look at what one of these documents actually contains, rather than just what it proves, the Peculiar Ventures attestation library parses examples from each of these platforms, including the Marvell HSM attestation produced by Google Cloud HSM. An attestation without a verifier is a claim. With a verifier, it is something a relying party can act on.

Policy as mechanism, not promise

Reading all of those attestations means the policy that evaluated them has to be something concrete enough to read.

Today’s CAs publish a CP/CPS. The Certificate Policy and Certification Practice Statement is a document describing what the CA will and will not do. An auditor samples evidence once a year against the document. The document and the system that produces certificates are not cryptographically linked. The relying party trusts that the document describes the system. The auditor’s annual report is the closing of the loop.

Cedar policies are different. They are a domain-specific language with declarative semantics, written under version control, statically analyzable, and small enough to read in a sitting. The policy that fires inside the signing oracle is compiled into the measured binary. The digest of the policy travels in the evidence bundle that accompanies the certificate. A relying party can re-fetch the source at the named digest, read the rules, and decide for themselves whether the policy that authorized their certificate is a policy they accept.

The contrast that matters operationally is the one between policy-as-promise and policy-as-mechanism. A CP/CPS is a promise. The auditor verifies, by sample, that the practice resembles the promise. A Cedar policy compiled into a measured binary, with its digest in the bundle, is a mechanism. The relying party verifies, per certificate, that the rules that fired are the rules the CA published.

There is a sharp footgun specific to Cedar that is worth naming because the answer to it is part of the architecture. Cedar’s evaluator skips a policy that throws while accessing an attribute that was not present. The convenient result is that policies stay readable in the absence of optional context. The inconvenient result is that a forbid policy with an unguarded attribute access can silently drop, which is a fail-open. The lint at build time requires a has guard before any optional attribute access. A policy that would have failed open is now a build error. The defense-in-depth move is structural. The policy author cannot ship a fail-open by accident.

What this looks like in practice

An employee gets a new smart card from corporate IT in a sealed blister pack. They plug it in.

The enrollment client on the laptop sees a card it doesn’t know. It reads the card’s GlobalPlatform Card Production Lifecycle data and the card recognition data, learns this is a factory-fresh retail token from a manufacturer it has a trust root for, and verifies the card identity attestation back to that root. The card is genuine. It has never been provisioned. The enrollment client knows what kind of token it is looking at and what the policy says to do with one.

The client provisions the token. It generates a new keypair on the card under a policy that requires user PIN for use and marks the key non-exportable. The card produces a key attestation: a statement signed by the card’s manufacturer-installed attestation key asserting that this specific public key was generated on this specific token, has these specific usage constraints, and will never leave the hardware. The enrollment client builds a CSR for the new key and has the card sign it, which is the standard proof that the requestor controls the corresponding private key. It then packages the CSR, the user’s identity claim, and the card’s attestation into an ACME request and sends it to the RA.

That is the first link. The chain is going to have several.

The RA receives the request inside its measured execution environment. It verifies the CSR signature against the public key in the CSR, which proves the requestor controls the corresponding private key. It verifies the card’s attestation against the manufacturer chain. Yes, this is a genuine retail token from a manufacturer we trust. Yes, the key in the CSR is on the card. Yes, the usage constraints match policy. It resolves the identity claim against the corporate identity provider. Yes, this user exists. Yes, they are entitled to this credential type. Yes, their device posture matches. It evaluates the Cedar policy against the cross-product of the device, the identity, and the requested profile. Yes, all of it permits issuance. It builds an authorization context, signs it with the key bound to its measured environment, and sends it to the signing oracle.

The signing oracle receives the context inside its own measured execution environment, over mutually-attested TLS where both sides have verified the other’s measurement before they would talk. It does not simply believe what the RA told it. For facts backed by independently verifiable evidence, the oracle re-verifies that evidence against its own configured verifier. In this example, it re-verifies the card’s attestation, cross-checks the claims the RA made about the card, the requested profile, and the requester, validates the profile binding, the certificate type, the validity window, the structural invariants the profile requires, and confirms that this RA is authorized to ask for this kind of issuance. It rejects replays. Only then does it ask the custodian for the signing key. The custodian releases it because the oracle’s attestation matches the published image. The oracle signs. It produces a fresh per-operation attestation binding this certificate to this measured execution environment and to the RA that authorized it. The issuance is written to a transparency log that independent witnesses cosign.

The bundle that comes back to the enrollment client contains the card’s attestation that the key is on the token, the RA’s signed authorization context naming the identity, the profile, the policy digest, and the verifiers it ran, the oracle’s per-operation attestation binding the signature to a measured binary on a measured platform, the custodian’s evidence that the key was released only because the oracle’s measurement matched, the transparency log inclusion proof and witness cosignatures showing the issuance was published before the certificate was returned, and the certificate itself.

Every party in the flow did the logical equivalent of what every other party did. It verified upstream evidence against a manufacturer, platform, custodian, or witness root it already trusted, did its work, and produced its own evidence for the downstream party to verify. The bundle packages those attestations and proofs so the subscriber, or anyone the subscriber shares it with, can re-walk the chain end to end against the same trust roots, without having to take any single party’s word for it.

The card manufacturer’s root says the key is on the hardware. The chip vendor’s TEE root says the RA ran the measured image. The chip vendor’s TEE root says the oracle ran the measured image. The HSM vendor’s root says the key was released to a measurement that matched. The witness network’s cosignatures say the log is what the operator published, not a fork served to one relying party. A relying party who trusts each of those roots can verify the certificate’s basis of issuance from the evidence bundle, instead of relying only on the CA’s institutional promise.

The CA is not asked only to be trusted. The CA produced evidence.

What is built

The architecture above runs in preproduction today on AWS and Google Cloud.

On AWS, that means Nitro Enclaves for enclave-based issuance components and NitroTPM-backed evidence for VM-level identity, measured boot, and workload posture. On Google Cloud, it means AMD SEV-SNP and Intel TDX Confidential VMs for protected execution, Cloud HSM as custodian, and vTPM-based evidence where VM boot state or workload identity needs to be represented.

Classical ECDSA and post-quantum ML-DSA-65 (FIPS 204) hierarchies operate in parallel. ML-KEM-768 (FIPS 203) is the subject key for TLS key-exchange certificates. Cedar policy with the fail-open lint is enforced in the oracle. Per-operation attestation, evidence bundles, the custodian gating key release on measurement, mutually attested TLS between RA and oracle, end-to-end on both clouds.

Each trust domain signs from its own sub-CA, and classical and post-quantum issuance never share a key. Machine, machine-with-EAP, user, group, workload, smart card, TPM AK, and SSH all sit under separate sub-CAs; classical and PQ are separated within each family. A compromise of any single signing key bounds the damage to one family-and-algorithm slice. The architecture treats hierarchy multiplicity as security-domain separation, not as an algorithm-bridging side effect.

Profiles wired up today cover machine authentication including EAP-TLS, DNS-validated TLS server certificates, workload identity including SPIFFE-style URI identifiers, user and group signing and encryption, smart card and PIV logon, TPM AK bootstrap, and SSH user, workload, and host certificates. Each family that supports both has a classical and a post-quantum variant.

The platform breadth is there because no single TEE family fits every customer environment, and the architecture should not be hostage to one chip vendor or one cloud.

The 2029 problem

None of what I’ve described above is algorithm-bound. That matters because the algorithms are about to change.

In March 2026, Google’s Heather Adkins and Sophie Schmieg set 2029 as the target for completing Google’s migration to post-quantum cryptography. Google’s timeline matters beyond Google. They run Chrome and Android, and when they move, the WebPKI moves with them. CNSA 2.0 puts 2027 on software and firmware signing in National Security Systems and 2030 on general use. The CABF is working through its own timeline. Federal procurement requirements are already moving.

The CA infrastructure that exists today was designed for the snapshot of math problems that the 2029 transition invalidates. Every CA in production is going to be re-architected before it lands. The algorithms expire. The migration is not optional.

The transition itself is going to be heterogeneous. The classical-PQC X.509 path is going to run for a long time alongside what eventually replaces it. Merkle Tree Certificates — batched, transparency-native issuance with much smaller per-certificate overhead — are a likely part of the answer to ML-DSA’s signature size on the wire. The architecture above does not care which container format the certificate is in. The attested issuance pipeline, the custodian-gated key release, the evidence bundle, the transparency log — all of it operates on the issuance side. MTC issuance benefits from runtime evidence the same way X.509 issuance does, and the patterns in this post carry over.

A CA built on the runtime-evidence pattern does not cost more to deploy at the moment you are already rebuilding. It costs more only if you skip the rebuild, and skipping is not on the table. The hardware-anchored credential side of the wire has been arriving in production for a decade. The issuance side is the part still running on the old shape. The PQ deadline is the forcing function that makes the issuance side move. The choice is between rebuilding the old shape with new algorithms, and rebuilding it with the same discipline that the relying-party side has spent the last decade adopting.

Short lifetimes with ARI make the operational side tractable. Seven-day certificates with ARI-driven renewal turn the PQ migration from a flag day into a moving window. The fleet rotates without an emergency, without anyone touching a machine, because the CA can shorten the renewal window for specific machines or profiles whenever it wants to.

That is what the next CA looks like. It is not a different CA than the one I have been describing. It is the same one.

When Compliance Records Become the Only Honest Signal

I’ve been spending a lot of time lately building Systematic Reasoning with my long-time friend Vishal. The core premise is straightforward. Organizations reveal their true operational character through how they design to prevent failure, how they plan to handle it when it happens, and how they actually do. That signal deserves to be tracked, structured, and acted on. We’re building an agentic compliance platform to do exactly that.

Systematic Reasoning won’t be limited to any single domain, but we decided to start with the Web PKI. The reasoning was simple. It’s high impact in a way that’s hard to overstate. Every internet user depends, whether they know it or not, on a relatively small number of Certificate Authorities getting things right. The margin for error is zero. If that trust layer breaks, it breaks for everyone.

DigiNotar is the canonical example. A small Dutch CA, compromised so thoroughly that attackers could impersonate any website on the web, and did. That capability was used to spy on Iranian dissidents, intercepting communications that people believed were private and secure. The trust infrastructure that was supposed to protect them was turned into a weapon against them. DigiNotar isn’t an edge case or a cautionary tale from a more naive era; it’s a demonstration of the actual ceiling of what can go wrong. And it isn’t the only one. State-affiliated certificate authorities have been caught performing man-in-the-middle attacks on their own citizens’ traffic, something the Baseline Requirements explicitly prohibit, but prohibition only matters if it’s enforced. The web’s trust model works right up until the moment someone decides it’s more useful as surveillance infrastructure.

At the core of Systematic Reasoning, is a belief I’ve held for a while. Compliance can be a vital sign of organizational security, but only if it’s continuous. The reality today is that it isn’t. Code ships daily. Audits happen annually. The gap between those two rhythms is where things go quietly wrong.

I’ve written before about why I have limited faith in the current audit regime. Auditors are engaged by the organizations they assess. Their product is a clean seal; their incentive is to keep the client. They operate on point-in-time sampling with auditee-selected scope, and they’re often compliance professionals rather than engineers, which means they’re checking whether a policy exists more than whether the system actually behaves correctly. That’s if you’re lucky. Sometimes the audit is scoped against a version of the Baseline Requirements that was superseded over a year ago.

The same incentive shapes how certificate authorities write their governance documents. A CP/CPS that relies heavily on incorporation by reference, that omits specifics about what the organization actually does and what constraints it operates under, is easier to audit against than one that makes precise, testable commitments. Vagueness isn’t always carelessness. Sometimes it’s a design choice. The same thing happens in incident reports. A report that attributes a failure to “organic process evolution” or “human error” without describing the actual control gap is easier to close than one that names the broken system and commits to a specific fix. In both cases the document gets the box checked without creating accountability. References establish authority. Commitments establish accountability.

The audit gap isn’t compensated for by strong internal monitoring either. The majority of significant compliance failures are not caught internally. They are caught by external researchers, root program staff, or community tooling. A broken validation endpoint runs for five years and the organization finds out because someone posted a 404 error in a public issue tracker. A validation race condition exists undetected for seven and a half years not because it was well hidden but because nobody was looking. The absence of an internal alarm is not evidence that the system is healthy. It is often evidence that the monitoring itself is missing.

So public incident reports and governance documents become some of the most signal-rich material available. Policy documents tell you what an organization claims it will do. Incident reports tell you what happened when reality diverged from that claim. Together they create a longitudinal picture that neither document produces alone.

Building a system to reason over that data surfaced a problem I didn’t fully anticipate. When you’re working from the outside, with no access to internal systems and no way to verify what actually changed, the public record is almost all you have. The question isn’t whether to treat it with skepticism. It’s how much skepticism to build in by default.

The temptation is to give the benefit of the doubt. Organizations are required to describe the blast radius of an incident. Not every localized bug is a symptom of something systemic. But accepting minimizing language at face value is its own failure.

“Only” is doing a lot of work when the bug it’s describing went undetected for seven and a half years. “No compromise of end-entities” is doing a lot of work when what it really means is that nobody found the gap before you did. Framing survival as security isn’t reporting, it’s PR. And if an organization believes an incident is no big deal, you can predict with reasonable confidence that the root cause analysis will be shallow and the remediation will be a band-aid.

ForgeIQX, our first offering, tracks those signals longitudinally across both policy documents and incident reports. Not to prosecute organizations for their language choices, but to notice when a commitment made in a CP/CPS quietly disappears in the next version, or when a promised fix is nowhere to be found when the same failure mode surfaces years later. That’s commitment decay, the slow evaporation of a promise made under pressure, and it’s only visible if you’re tracking across multiple documents and incidents over time rather than treating each one in isolation.

The calibration problem is real and doesn’t have a clean answer. Get it wrong in one direction and you build a system that cries wolf. Get it wrong in the other and you build a system that launders PR-speak into clean signals, which is just automating the thing we already do too much of.

There’s a third failure mode that took me longer to see. A system like this can be gamed. Swap “we got lucky” for “our monitoring detected no active exploitation.” Replace “only thirty certificates” with a more clinical impact scoping statement that says the same thing in language that sounds like engineering rigor. The words change; the institutional posture doesn’t. A system that can be satisfied by better prose isn’t measuring operational maturity, it’s measuring communications sophistication.

That means the system has to be built with structural pessimism. Not cynicism for its own sake, but a deliberate prior that clean language is not the same as clean operations, and that the absence of red flags is not the same as the presence of green ones. We can’t verify that an organization fixed what it said it would fix. What we can do is watch whether the same failure mode surfaces again and whether the pattern of shallow root cause analyses continues or breaks. The historical record doesn’t tell us what’s true inside these organizations. It tells us what they were willing to say in public, under pressure, over time. Given the alternatives, that may be the most honest signal available.

A certificate authority with genuine operational maturity should want this kind of scrutiny applied to itself. Not because it will always produce a clean result, but because it surfaces the gaps before an external party does. ForgeIQX gives organizations a way to continuously monitor their own compliance posture, so their practices and code keep pace with their commitments. The same is true for auditors who want their findings to mean something beyond a checkbox. The problem with the current regime isn’t that the people in it are careless. It’s that the incentive structures don’t reward rigor, and the tooling to demonstrate it continuously doesn’t exist. That’s what we’re building.

The Web PKI is where we started because the stakes are concrete and the public record is unusually rich. But any regulated industry where compliance is measured annually, where governance documents are written to satisfy auditors rather than inform relying parties, and where incident reports are drafted with one eye on legal exposure, has the same gap between what the paper says and what the organization actually does. We started here. We don’t intend to stop here.

Duplicates Are Not the Problem

The Washington House is now arguing that the sign-in dataset for SB 6346 is unreliable because it contains duplicate names. The claim is simple. If the same name appears more than once, you cannot trust the totals.

They are not wrong that duplicates exist. They are wrong about what duplicates mean and what to do about them.

Every real-world dataset contains noise. Names entered twice, typos, outliers, junk. This is not a scandal. It is a property of data collected from human beings at scale. The standard response is not to discard the dataset. It is to trim it. A trimmed mean, cutting the head or tail or both, is one of the oldest tools in data science. The presence of junk data is not a reason to abandon analysis. It is the reason analysis exists.

The birthday-corrected collision test applied in the previous post is a more principled version of exactly that. Rather than arbitrarily cutting a fixed percentage off the tail, it uses the population model to identify which specific windows are statistically anomalous and removes only those. The legislature is being offered a choice between principled trimming and throwing the whole dataset away. One of those is data science. The other is a talking point.

Why Duplicates Happen

Before getting to the test, it is worth being precise about why duplicates appear in the first place, because the innocent explanations are more common than the fraudulent ones.

Approximately 800 people named John Smith live in Washington state. These are real, distinct individuals.

The first is demographics. According to the U.S. Census Bureau, Smith is the most common surname in America, occurring roughly 828 times per 100,000 people. There are an estimated 32,000 people named John Smith in the United States, approximately 800 in Washington state alone. But national averages miss how name frequency actually works in practice. It clusters by community. Redmond and Bellevue have dense South Asian tech worker populations where Patel and Singh recur at rates far above the state average. Tukwila and south King County have large East African and Somali communities where Mohammed appears with predictable frequency. South Seattle and the Puget Sound corridor have substantial Vietnamese communities where Nguyen, already the most common surname in Vietnam, concentrates heavily. Name frequency is never random. It reflects religion, culture, and family tradition. Mohammed is among the most common names in the world because naming a son after the prophet is an act of Islamic devotion practiced across generations. That is not a data quality problem. The same full name appearing two or three times in 80,000 records is not evidence of anything. It is census math applied to a state that looks nothing like the national average.

The second reason duplicates appear is the sign-in form itself. It does not confirm that your submission was received. Anyone who has filled out a web form and stared at the screen knows what comes next. You submit again. Someone might also change their mind and resubmit to correct their position. A household with two people named Michael Johnson might both sign in independently. None of that is fraud. Both causes are real, and a serious analysis accounts for both.

Beyond that, even if we removed all of the duplicates, it would not even move the needle on the ultimate message being sent. With that said, it is worth noting that CON has more removals in absolute terms because it has ten times as many submissions, which is what we would expect based on the collision test.

On Rapid Submissions

A related claim is that submissions arriving within seconds of each other indicate bot activity. The timing observation is real. The interpretation is not supported by the data available.

Rapid same-name pairs are primarily a function of submission volume. When hundreds of people are submitting per hour, two people who share a name will statistically land within seconds of each other by chance alone. The chart below plots same-name rapid pairs against hourly submission rate for both sides. Both follow the same curve. The PRO overnight Feb 20 hours, at roughly 190 submissions per hour, fall below where the trend predicts they should be, which is consistent with what the collision test found. The timing argument does not add new evidence against PRO. It describes a mathematical property of any high-volume submission window.

The public export contains no IP addresses. Without them, rapid sequential submissions cannot be distinguished between three completely different explanations. The first is a single person double-submitting because the form gave no confirmation. The second is two people in the same household on the same connection. The third is two distinct people with different IPs whose submissions happened to land close together during a busy window.

The tool that would actually resolve this is IP address logs from the server. A same-IP rapid duplicate is strong resubmission evidence. A different-IP rapid duplicate from a residential ISP is two real people. A cluster of submissions from a datacenter or known VPN range is a different finding entirely. None of that analysis is possible from the public CSV, which is the only data anyone outside the AG’s office has seen.

This matters because the “within seconds” framing is being used to support a conclusion the available data cannot reach. The previous post noted that IP logs should be preserved before they age out. That recommendation stands. Until that analysis is done, timing alone is not evidence of anything specific.

It is also worth noting what the pattern does not look like. It shows zero name collisions and below-trend rapid pairs, the opposite of what cheap automation produces. What that pattern is consistent with is a large list of pre-generated unique names submitted at a controlled rate. CAPTCHA does not stop that. Each submission looks like a distinct human from the name and timing perspective. The fix legislators might reach for does not address the threat model the data actually points to.

What the Test Is Measuring

The birthday problem tells you that a room of 23 people has a 50% chance of containing a shared birthday. The same math gives you the expected number of name collisions in any random sample drawn from a community of known size. If you have 9,000 PRO supporters and draw 934 names from that pool, some names will repeat by chance. Not because anyone cheated. Because Jennifer Lee exists in multiples, and because some of them hit submit twice when the page did not respond.

The expected number of collisions for that sample is approximately 60. Not zero. Sixty. The test does not flag duplicates. It asks whether the duplication rate is consistent with what a genuine community would produce.

For the Senate PRO February 20th overnight window, the observed collisions were zero. Not fewer than expected. Zero. Across 10,000 simulations drawing from the actual PRO participant pool, the minimum produced was around 30. The overnight batch produced none.

The CON overnight windows tell the opposite story. More collisions than expected across several nights, consistent with resubmission, common names appearing organically, households submitting together. The kind of messy that real participation produces.

What This Means for the Dataset

The argument that duplicates make the dataset unreliable cuts in exactly the wrong direction. The PRO overnight batch from February 20th is anomalous precisely because it has too few duplicates, not too many. A genuine sample from a real community, one that includes people named John Smith and people who hit submit twice, does not produce zero collisions in 934 draws. It is statistically impossible.

Raw duplicate counts, without correcting for population name frequency and sample size, are not a meaningful metric. The legislature is being asked whether these sign-in totals reflect genuine public sentiment, and that is a statistical question with a statistical answer. The answer is not “the dataset has duplicates, therefore we cannot know.” The methodology was built specifically to separate expected duplication from anomalous duplication, and the findings hold.

Discarding the dataset because it contains duplicates is not data analysis. It is avoiding data analysis.

None of this is perfect. IP address analysis would not be definitive because VPNs, shared connections, and mobile carriers complicate attribution. The collision test rests on a population model that is an estimate, not a census. The rapid pairs chart fits a trend to noisy data. Statistical inference is always probabilistic, and anyone who tells you otherwise is selling something.

But the question legislators are actually asking is not whether this dataset is perfect. It is whether the sign-in totals are a reasonable signal of public sentiment, and whether the anomalies identified are significant enough to warrant skepticism about specific windows. For that question, the methodology does not need to be perfect. It needs to be fit for purpose.

A 10:1 ratio that survives deduplication, symmetrical trimming, and a collision test that was explicitly designed to tolerate legitimate duplication is a robust signal. The PRO overnight Feb 20 anomaly does not need to be proven beyond a reasonable doubt to be disqualifying for that window. The standard here is not a criminal conviction. It is whether legislators can treat the aggregate numbers as a directional guide to constituent sentiment. On that standard, the analysis is more than sufficient.

On Impersonation

Named officials discovering their identities appeared in the dataset without their consent is a real incident worth investigating. But the sign-in system was never designed to verify identity or attribute positions to specific individuals. Names are collected not to create a record of who voted, but because a completely anonymous system would be trivially manipulable. A name field is the minimal friction that makes aggregate analysis possible at all.

The relevant question for legislators is not “did John Smith actually sign this?” but “does the distribution of sign-ins reflect genuine public sentiment.” This is a survey mechanism, not a ballot. Washington has 7.8 million residents. Even a perfectly clean dataset with 100,000 CON sign-ins represents a small fraction of the population. Legislators have always understood these numbers as a directional signal, not a binding count. Treating impersonation as the central finding, rather than asking whether the aggregate signal survived manipulation, mistakes the instrument for the measurement.

The numbers behind the impersonation claim deserve scrutiny. Invest in Washington Now reported roughly 100-200 confirmed cases across 123,289 records, less than 0.2% of the dataset. Even tripling that estimate to account for unreported cases, it does not move a 10:1 ratio in any meaningful direction. And if you apply their own deduplication logic symmetrically: remove every name that appears more than once from both sides. CON drops from roughly 110,000 to 91,000 and PRO drops from roughly 10,000 to 9,000. The ratio is still 10:1. Their argument, applied consistently to both sides, does not change the conclusion.

Those confirmed cases were identified because victims self-reported. Public officials monitor mentions of their names, noticed the discrepancy, and came forward. That is the easiest fraud to find. It tells you nothing about what the rest of the dataset contains. Self-reported impersonation is the floor of what happened, not the ceiling, which is precisely why aggregate statistical analysis exists.

It is also worth considering what those confirmed cases likely represent. Some are probably legitimate resubmissions. Someone signed in, was not sure it worked, signed in again, and now appears twice. Some are probably trolling. Actual coordinated impersonation may be in there too, but the self-report mechanism cannot distinguish between the three. Treating 200 high-visibility cases driven by public figures monitoring their own names as representative of the full 123,000-record dataset is not a statistical argument. It is a press conference.

So What Does All of This Mean?

The answer to that is simple. The dataset has duplicates. The timing raised questions. Some names were submitted without consent. None of those observations, examined carefully, change what the data shows: roughly ten Washington residents opposed this bill for every one who supported it in committee. That signal has survived every test applied to it.

WebPKI Market Analysis: Mozilla Telemetry vs Certificate Transparency Data

In the past, I’ve written about how to measure the WebPKI, and from time to time I post brief updates on how the market is evolving.

The other day, Matthew McPherrin posted a script showing how to use Mozilla telemetry data to analyze which Certificate Authorities are more critical to the web. Specifically, what percentage of browsing relies on each CA. Mozilla provides public data from Firefox’s telemetry on how many times a CA is used to successfully validate certificates. This is a pretty good measure for how “big” a CA actually is. The data is pretty hard to view in Mozilla’s public systems though, so he made a script to combine a few data sources and graph it.

I normally focus on total issuance numbers since they’re easier to obtain. That data comes from Certificate Transparency logs, which contain all publicly trusted certificates that you might encounter without seeing an interstitial warning about the certificate not being logged (like this example).

What the Data Reveals

Both datasets feature many of the same major players. But there are some striking differences that reveal important insights about the WebPKI ecosystem.

Let’s Encrypt dominates certificate issuance at 46.1% of all certificates. But it ranks third in Firefox’s actual usage telemetry. This suggests Let’s Encrypt serves many lower-traffic sites. Meanwhile, Google Trust Services leads in Firefox usage while ranking second in certificate issuance volume. This shows how high-traffic sites can amplify a CA’s real-world impact.

DigiCert ranks second in Firefox usage while placing fourth in certificate issuance volume at 8.3%. This reflects their focus on major enterprise customers. With clients like Meta (Facebook, Instagram, WhatsApp), they secure some of the world’s highest-traffic websites. This “fewer certificates, massive impact” approach drives them up the usage charts despite not competing on volume with Let’s Encrypt.

Google’s dominance reflects more than just their own properties like Google.com, YouTube, and Gmail. Google Cloud offers arguably the best load balancer solution in the market (full disclosure I worked on this project). You get TLS by default for most configurations. Combined with their global network that delivers CDN-like benefits out of the gate, this attracts major platforms like Wix and many others to build on Google Cloud. When these platforms choose Google’s infrastructure, they automatically inherit Google Trust Services certificates.

Looking at the usage data reveals other interesting patterns. Deutsche Telekom Security, Government of Turkey, (UPDATE: turns out the Turkey entry is a Firefox bug: they’re using bucket #1 for both locally installed roots and Kamu SM, apparently by accident) and SECOM Trust Systems all appear prominently in Firefox telemetry but barely register in issuance numbers. In some respects, it’s no surprise that government-issued certificates see disproportionate usage. Government websites are often mandated for use. Citizens have to visit them for taxes, permits, benefits, and other essential services.

Microsoft Corporation appears significantly in issuance data (6.5%) but doesn’t register in the Firefox telemetry. This reflects their focus on enterprise and Windows-integrated scenarios rather than public web traffic.

GoDaddy shows strong issuance numbers (10.5%) but more modest representation in browsing telemetry. This reflects their massive domain parking operations. They issue certificates for countless parked domains that receive minimal actual user traffic.

Why This Matters

Mozilla Firefox represents under 3% of global browser market share. This telemetry reflects a smaller segment of internet users. While this data provides valuable insights into actual CA usage patterns, it would be ideal if Chrome released similar telemetry data. Given Chrome’s dominant 66.85% market share, their usage data would dramatically improve our understanding of what real WebPKI usage actually looks like across the broader internet population.

The contrast between certificate issuance volume and actual browsing impact reveals important truths about internet infrastructure. CT logs currently show over 450,000 certificates being issued per hour across all CAs. Yet as this Firefox telemetry data shows, much of that volume serves lower-traffic sites while a smaller number of high-traffic certificates drive the actual user experience. Some CAs focus on high-volume, automated issuance for parked domains and smaller sites. Others prioritize fewer certificates for high-traffic, essential destinations. Understanding both metrics helps us better assess the real-world criticality of different CAs for internet security and availability.

Raw certificate counts don’t tell the whole story. The websites people actually visit, and sometimes must visit, matter just as much as the sheer number of certificates issued. Some certificates protect websites with “captive audiences” or essential services, while others protect optional destinations. A government tax portal or YouTube will always see more traffic than the average small business website, regardless of how many certificates each CA issues.

Regardless of how you count, I’ve had the pleasure of working closely with at least 7 of the CAs in the top 10 in their journeys to become publicly trusted CAs. Each of these CAs have had varying goals for their businesses and operations, and that’s exactly why you see different manifestations in the outcomes. Let’s Encrypt focused on automation and volume. DigiCert targeted enterprise customers. Google leveraged their cloud infrastructure. GoDaddy built around domain services.

Either way, it’s valuable to compare and contrast these measurement approaches to see what the WebPKI really looks like beyond just raw certificate counts.

How ‘Sneakers’ Predicted Our Quantum Computing Future

“The world isn’t run by weapons anymore, or energy, or money. It’s run by little ones and zeroes, little bits of data. It’s all just electrons.” — Martin Bishop, Sneakers (1992)

I was 16 when I first watched Sneakers on a VHS tape rented from my local video store. Between the popcorn and plot twists, I couldn’t have known that this heist caper would one day seem less like Hollywood fantasy and more like a prophetic warning about our future. Remember that totally unassuming “little black box” – just an answering machine, right? Except this one could crack any code. The device that sent Robert Redford, Sidney Poitier, and their ragtag crew on a wild adventure. Fast forward thirty years, and that movie gadget gives those of us in cybersecurity a serious case of déjà vu.

Today, as quantum computing leaves the realm of theoretical physics and enters our practical reality, that fictional black box takes on new significance. What was once movie magic now represents an approaching inflection point in security – a moment when quantum algorithms like Shor’s might render our most trusted encryption methods as vulnerable as a simple padlock to a locksmith.

When Hollywood Met Quantum Reality

I’ve always found it deliciously ironic that Leonard Adleman – the “A” in RSA encryption – served as the technical advisor on Sneakers. Here was a man who helped create the mathematical backbone of modern digital security, consulting on a film about its theoretical downfall. What’s particularly fascinating is that Adleman took on this advisory role partly so his wife could meet Robert Redford! His expertise is one reason why the movie achieves such technical excellence. It’s like having the architect of a castle advising on a movie about the perfect siege engine. For what feels like forever – three whole decades – our world has been chugging along on a few key cryptographic assumptions. We’ve built trillion-dollar industries on the belief that certain mathematical problems—factoring large numbers or solving discrete logarithms—would remain practically impossible for computers to solve. Yep, our most of security is all built on these fundamental mathematical ideas. Sneakers playfully suggested that one brilliant mathematician might find a shortcut through these “unsolvable” problems. The movie’s fictional Gunter Janek discovered a mathematical breakthrough that rendered all encryption obsolete – a cinematic prediction that seemed far-fetched in 1992.

Yet here we are in the 2020s, watching quantum computing advance toward that very capability. What was once movie magic is becoming technological reality. The castle walls we’ve relied on aren’t being scaled—they’re being rendered obsolete by a fundamentally different kind of siege engine.

The Real Horror Movie: Our Security Track Record

Hollywood movies like Sneakers imagine scenarios where a single breakthrough device threatens our digital security. But here’s the kicker, and maybe the scarier part: the real threats haven’t been some crazy math breakthrough, but the everyday stuff – those operational hiccups in the ‘last mile’ of software supply chain and security management. I remember the collective panic during the Heartbleed crisis of 2014. The security community scrambled to patch the vulnerability in OpenSSL, high-fiving when the code was fixed. But then came the sobering realization: patching the software wasn’t enough. The keys – those precious secrets exposed during the vulnerability’s window – remained unchanged in countless systems. It was like installing  a new lock for your door but having it keyed the same as the old one all the while knowing copies of the key still sitting under every mat in the neighborhood. And wouldn’t you know it, this keeps happening, which is frankly a bit depressing. In 2023, the Storm-0558 incident showed how even Microsoft – with all its resources and expertise – could fall victim to pretty similar failures. A single compromised signing key allowed attackers to forge authentication tokens and breach government email systems. The digital equivalent of a master key to countless doors was somehow exposed, copied, and exploited. Perhaps most illustrative was the Internet Archive breach. After discovering the initial compromise, they thought they’d secured their systems. What they missed was complete visibility into which keys had been compromised. The result? Attackers simply used the overlooked keys to walk right back into the system later. Our mathematical algorithms may be theoretically sound, but in practice, we keep stumbling at the most human part of the process: consistently managing the lifecycle of the software and cryptographic keys through theih entire lifecycle. We’re brilliant at building locks but surprisingly careless with the keys.

From Monochrome Security to a Quantum Technicolor 

Think back to when TVs went from black and white to glorious color. Well, cryptography’s facing a similar leap, except instead of just adding RGB, we’re talking about a whole rainbow of brand new, kinda wild frequencies. For decades, we’ve lived in a relatively simple cryptographic world. RSA and ECC have been the reliable workhorses – the vanilla and chocolate of the security ice cream shop. Nearly every secure website, VPN, or encrypted message relies on these algorithms. They’re well-studied, and deeply embedded in our digital infrastructure. But quantum computing is forcing us to expand our menu drastically. Post-quantum cryptography introduces us to new mathematical approaches with names that sound like science fiction concepts: lattice-based cryptography, hash-based signatures, multivariate cryptography, and code-based systems. Each of these new approaches is like a different musical instrument with unique strengths and limitations. Lattice-based systems offer good all-around performance but require larger keys. Hash-based signatures provide strong security guarantees but work better for certain applications than others. Code-based systems have withstood decades of analysis but come with significant size trade-offs. That nice, simple world where one crypto algorithm could handle pretty much everything? Yeah, that’s fading fast. We’re entering an era where cryptographic diversity isn’t just nice to have – it’s essential for survival. Systems will need to support multiple algorithms simultaneously, gracefully transitioning between them as new vulnerabilities are discovered. This isn’t just a technical challenge – it’s an operational one. Imagine going from managing a small garage band to conducting a full philharmonic orchestra. The complexity doesn’t increase linearly; it explodes exponentially. Each new algorithm brings its own key sizes, generation processes, security parameters, and lifecycle requirements. The conductor of this cryptographic orchestra needs perfect knowledge of every instrument and player.

The “Operational Gap” in Cryptographic Security

Having come of age in the late ’70s and ’80s, I’ve witnessed the entire evolution of security firsthand – from the early days of dial-up BBSes to today’s quantum computing era. The really wild thing is that even with all these fancy new mathematical tools, the core questions we’re asking about trust haven’t actually changed all that much. Back in 1995, when I landed my first tech job, key management meant having a physical key to the server room and maybe for the most sensitive keys a dedicated hardware device to keep them isolated. By the early 2000s, it meant managing SSL certificates for a handful of web servers – usually tracked in a spreadsheet if we were being diligent. These days, even a medium-sized company could easily have hundreds of thousands of cryptographic keys floating around across all sorts of places – desktops, on-premise service, cloud workloads, containers, those little IoT gadgets, and even some old legacy systems. The mathematical foundations have improved, but our operational practices often remain stuck in that spreadsheet era. This operational gap is where the next evolution of cryptographic risk management must focus. There are three critical capabilities that organizations need to develop before quantum threats become reality:

1. Comprehensive Cryptographic Asset Management

When a major incident hits – think Heartbleed or the discovery of a new quantum breakthrough – the first question security teams ask is: “Where are we vulnerable?” Organizations typically struggle to answer this basic question. During the Heartbleed crisis, many healthcare organizations spent weeks identifying all their vulnerable systems because they lacked a comprehensive inventory of where OpenSSL was deployed and which keys might have been exposed. What should have been a rapid response turned into an archaeological dig through their infrastructure. Modern key management must include complete visibility into:

  • Where’s encryption being used?
  • Which keys are locking down which assets?
  • When were those keys last given a fresh rotation?
  • What algorithms are they even using?
  • Who’s got the keys to the kingdom?
  • What are all the dependencies between these different crypto bits?

Without this baseline visibility, planning or actually pulling off a quantum-safe migration? Forget about it.

2. Rapid Cryptographic Incident Response

When Storm-0558 hit in 2023, the most alarming aspect wasn’t the initial compromise but the uncertainty around its scope. Which keys were affected? What systems could attackers access with those keys? How quickly could the compromised credentials be identified and rotated without breaking critical business functions? These questions highlight how cryptographic incident response differs from traditional security incidents. When a server’s compromised, you can isolate or rebuild it. When a key’s compromised, the blast radius is often unclear – the key might grant access to numerous systems, or it might be one of many keys protecting a single critical asset. Effective cryptographic incident response requires:

  • Being able to quickly pinpoint all the potentially affected keys when a vulnerability pops up.
  • Having automated systems in place to generate and deploy new keys without causing everything to fall apart.
  • A clear understanding of how all the crypto pieces fit together so you don’t cause a domino effect.
  • Pre-planned procedures for emergency key rotation that have been thoroughly tested, so you’re not scrambling when things hit the fan.
  • Ways to double-check that the old keys are completely gone from all systems.

Forward-thinking organizations conduct tabletop exercises for “cryptographic fire drills” – working through a key compromise and practicing how to swap them out under pressure. When real incidents occur, these prepared teams can rotate hundreds or thousands of critical keys in hours with minimal customer impact, while unprepared organizations might take weeks with multiple service outages.

3. Cryptographic Lifecycle Assurance

Perhaps the trickiest question in key management is: “How confident are we that this key has been properly protected throughout its entire lifespan?” Back in the early days of security, keys would be generated on secure, air-gapped systems, carefully transferred via physical media (think floppy disks!), and installed on production systems with really tight controls. These days, keys might be generated in various cloud environments, passed through CI/CD pipelines, backed up automatically, and accessed by dozens of microservices. Modern cryptographic lifecycle assurance needs:

  • Making sure keys are generated securely, with good randomness.
  • Storing keys safely, maybe even using special hardware security modules.
  • Automating key rotation so humans don’t have to remember (and potentially mess up).
  • Keeping a close eye on who can access keys and logging everything that happens to them.
  • Securely getting rid of old keys and verifying they’re really gone.
  • Planning and testing that you can actually switch to new crypto algorithms smoothly.

When getting ready for post-quantum migration, organizations often discover keys in use that were generated years ago under who-knows-what conditions, leading to them discovering that they need to do a complete overhaul of their key management practices.

Business Continuity in the Age of Cryptographic Change

If there’s one tough lesson I’ve learned in all my years in tech, it’s that security and keeping the business running smoothly are constantly pulling in opposite directions. This tension is especially noticeable when we’re talking about cryptographic key management. A seemingly simple crypto maintenance task can also turn into a business disaster because you have not properly tested things ahead of time, leaving you in a state where you do not understand the potential impact if these tasks if things go wrong. Post-quantum migration magnifies these risks exponentially. You’re not just updating a certificate or rotating a key – you’re potentially changing the fundamental ways systems interoperate all at once. Without serious planning, the business impacts could be… well, catastrophic. The organizations that successfully navigate this transition share several characteristics:

  • They treat keeping crypto operations running as a core business concern, not just a security afterthought.
  • They use “cryptographic parallel pathing” – basically running the old and new crypto methods side-by-side during the switch.
  • They put new crypto systems through really rigorous testing under realistic conditions before they go live.
  • They roll out crypto changes gradually, with clear ways to measure if things are going well.
  • They have solid backup plans in case the new crypto causes unexpected problems.

Some global payment processors have developed what some might call “cryptographic shadow deployments” – they run the new crypto alongside the old for a while, processing the same transactions both ways but only relying on the old, proven method for actual operations. This lets them gather real-world performance data and catch any issues before customers are affected.

From Janek’s Black Box to Your Security Strategy

As we’ve journeyed from that fictional universal codebreaker in Sneakers to the very real quantum computers being developed today, it strikes me how much the core ideas of security haven’t actually changed. Back in the 1970s security was mostly physical – locks, safes, and vaults. The digital revolution just moved our valuables into the realm of ones and zeros, but the basic rules are still the same: figure out what needs protecting, control who can get to it, and make sure your defenses are actually working. Post-quantum cryptography doesn’t change these fundamentals, but it does force us to apply them with a whole new level of seriousness and sophistication. The organizations that suceed in this new world  will be the ones that use the quantum transition as a chance to make their cryptographic operations a key strategic function, not just something they do because they have to. The most successful will:

  • Get really good at seeing all their crypto stuff and how it’s being used.
  • Build strong incident response plans specifically for when crypto gets compromised.
  • Make sure they’re managing the entire lifecycle of all their keys and credentials properly.
  • Treat crypto changes like major business events that need careful planning.
  • Use automation to cut down on human errors in key management.
  • Build a culture where doing crypto right is something people value and get rewarded for.

The future of security is quantum-resistant organizations.

Gunter Janek’s fictional breakthrough in Sneakers wasn’t just about being a math whiz – it was driven by very human wants. Similarly, our response to quantum computing threats won’t succeed on algorithms alone; we’ve got to tackle the human and organizational sides of managing crypto risk. As someone who’s seen the whole evolution of security since the ’70s, I’m convinced that this quantum transition is our best shot at really changing how we handle cryptographic key management and the associated business risks.

By getting serious about visibility, being ready for incidents, managing lifecycles properly, and planning for business continuity, we can turn this challenge into a chance to make some much-needed improvements. The black box from Sneakers is coming – not as a device that instantly breaks all encryption, but as a new kind of computing that changes the whole game. 

The organizations that come out on top won’t just have the fanciest algorithms, but the ones that have the discipline to actually use and manage those algorithms and associated keys and credentials effectively. 

So, let’s use this moment to build security systems that respect both the elegant math of post-quantum cryptography and the wonderfully messy reality of human organizations. 

We’ve adapted before, and we’ll adapt again – not just with better math, but with better operations, processes, and people. The future of security isn’t just quantum-resistant algorithms; it’s quantum-resistant organizations.

From the Morris Worm to Modern Agentic AI Threats

The year was 1988, and at age 13, I found myself glued to news and IRC channels buzzing with news of the Morris Worm. As reports poured in about thousands of computers grinding to a halt, I was captivated by how one graduate student’s experiment had cascaded into the first major internet security crisis. That moment taught me a crucial lesson: even well-intentioned actions can spiral into unforeseen consequences.

Three decades later, we face challenges that young me could hardly have imagined. Today’s AI systems aren’t just following predetermined scripts—they’re autonomous agents actively optimizing for goals, often discovering novel and potentially concerning paths to achieve them.

We’re seeing this play out in both research settings and real-world applications. Language models finding creative ways to circumvent content filters, reinforcement learning agents discovering unintended exploits in their training environments—these aren’t malicious attacks, but they demonstrate how autonomous systems can pursue their objectives in ways their developers hadn’t anticipated.

The parallels to the Morris Worm are striking. Just as Robert Morris never intended to crash 6,000 machines, today’s non-adversarial AI developers don’t set out to create systems that bypass safety controls. Yet in both cases, we’re confronting the same fundamental challenge: how do we maintain control over increasingly sophisticated systems that can act in ways their creators never envisioned??

Some argue that fully autonomous AI agents pose risks we shouldn’t take. Fully Autonomous AI Agents Should Not Be Developed (arXiv:2502.02649) explores why.

Since, as they say, those who cannot remember the past are doomed to repeat it, I’ve put together some thoughts on different aspects of this reality:

The evolution from the Morris Worm to today’s autonomous AI agents isn’t just a fascinating trajectory in technology—it’s a crucial reminder that security must continuously adapt to meet new challenges. As these systems grow more sophisticated, our protective measures must evolve in tandem, informed by the lessons of the past but ready for the challenges of tomorrow.

Timeless Farm Wisdom

My father grew up on a small farm in eastern Washington. They say you can take the boy out of the farm, but you can’t take the farm out of the boy. As a kid, I was always hearing farm life sayings from my grandfather and father. To this day, I think they are some of the best pieces of advice I have ever received. Here are a few:

  • Don’t wrestle with pigs. You both get filthy and the pig likes it: Avoid pointless arguments with people who thrive on conflict. It only wastes your time and energy.
  • Don’t count your chickens before they hatch: Don’t assume a deal is closed until the contract is signed.
  • Make hay while the sun shines: Capitalize on opportunities while they are available.
  • Don’t put all your eggs in one basket: Diversify your investments and strategies to mitigate risk.
  • The early bird catches the worm: Being proactive and early can lead to the best opportunities.
  • Every cloud has a silver lining: Look for opportunities in every challenge or setback.
  • Don’t cry over spilled milk: Learn from mistakes, but don’t dwell on them; move forward.
  • When the rooster crows, it’s time to get up: Start your day early to maximize productivity.
  • You can’t make an omelet without breaking a few eggs: Achieving success often requires taking some risks and making sacrifices.
  • You reap what you sow: Your results are a reflection of the effort and quality you put in.
  • Don’t look a gift horse in the mouth: Be grateful for opportunities and advantages you receive.
  • When in doubt, let your horse do the thinking: Trust the expertise of your team when you’re unsure.
  • Never approach a bull from the front, a horse from the rear, or a fool from any direction: Avoid unnecessary risks and confrontations in business.
  • Don’t close the barn door after the horse has bolted: Take preventive measures rather than waiting until you need to react.
  • A stitch in time saves nine: Address small issues before they become large problems.
  • The squeaky wheel gets the grease: The most noticeable or pressing issues are the ones that receive the most attention and resources first.

Groundhog Day: Learning from Past Key and Credential Compromises

As they say, Those who cannot remember the past are condemned to repeat it, as we look back at the last decade, it seems we are caught in our own little Groundhog Day, reexperiencing the consequences of weak authentication and poor key management over and over.

It is not that we don’t know how to mitigate these issues; it’s more that, as organizations, we tend to avoid making uncomfortable changes. For example, the recent spate of incidents involving Snowflake customers being compromised appears to be related to Snowflake not mandating multi-factor authentication or a strong hardware-backed authenticator like Passkeys for its customers.

At the same time, not all of these key and credential thefts are related to users; many involve API keys, signing keys used by services, or code signing keys. Keys we have been failing to manage appropriately to the risks they represent to our systems and customers.

Timeline of Notable Incidents

Table of Incidents

IncidentType of CompromiseDescription
Stuxnet (2010)Code Signing KeysUtilized stolen digital certificates from Realtek and JMicron to authenticate its code, making it appear legitimate.
Adobe (2012)Code Signing KeysAttackers compromised a build server and signed malware with valid Adobe certificates.
Target (2013)Developer CredentialsNetwork credentials stolen from a third-party HVAC contractor via malware on an employee’s home computer.
Neiman Marcus (2013)Developer CredentialsMalware installed on systems, possibly through compromised credentials or devices from third-party vendors.
Home Depot (2014)Developer CredentialsMalware infected point-of-sale systems, initiated by stolen vendor credentials, potentially compromised via phishing.
Equifax (2017)API KeysExploitation of a vulnerability in Equifax’s website application exposed API keys used for authentication.
CCleaner (2017)Code Signing KeysAttackers inserted malware into CCleaner’s build process, distributing malicious versions of the software.
Ticketmaster (2018)Developer CredentialsMalware from a third-party customer support product led to the compromise of payment information.
ASUS Live Update (2018)Code Signing KeysAttackers gained access to ASUS’s code signing keys and distributed malware through the update mechanism.
Google Cloud Key Leak (2019)API KeysInternal tool inadvertently exposed customer API keys on the internet, leading to potential data exposure.
Facebook Instagram API Breach (2019)API KeysPlaintext passwords were accessible to employees due to misuse of an internal API.
DoorDash Data Breach (2019)API KeysUnauthorized access to systems included the compromise of API keys, exposing sensitive customer data.
Mission Lane (2020)Developer CredentialsMalware introduced into the development environment from a developer’s compromised personal device.
BigBasket (2020)Developer CredentialsData breach with over 20 million customer records exposed, suspected initial access through compromised developer credentials.
Twitter API Key Compromise (2020)API KeysAttackers gained access to internal systems, possibly compromising API keys or administrative credentials.
Amazon AWS Key Exposure (2021)API KeysMisconfigured AWS S3 bucket led to the exposure of API keys and sensitive customer data.
Nvidia (2022)Code Signing KeysStolen code signing certificates were used to sign malware, making it appear legitimate.
Microsoft Storm 0558 (2023)Signing KeysAttackers gained access to email accounts of government agencies and other organizations by forging authentication tokens using a stolen Microsoft signing key.

When we look at these incidents we see a few common themes including:

  1. Repetitive Failures in Security Practices:
    • Despite awareness of the issues, organizations continue to face the same security breaches repeatedly. This suggests a reluctance or failure to implement effective long-term solutions.
  2. Resistance to Necessary Changes:
    • Organizations often avoid making uncomfortable but necessary changes, such as enforcing multi-factor authentication (MFA) or adopting stronger authentication methods like Passkeys, and adopting work This resistance contributes to ongoing vulnerabilities and compromises.
  3. Diverse Sources of Compromise:
    • Key and credential thefts are not solely due to user actions but also involve API keys, service signing keys, and code signing keys. This highlights the need for comprehensive key management and protection strategies that cover all types of keys.
  4. Broader Key Management Practices:
    • Moving from shared secrets to asymmetric credentials for workloads and machines can significantly enhance security. Asymmetric credentials, especially those backed by hardware, are harder to steal and misuse compared to shared secrets. Techniques like attestation, which is the device equivalent of MFA, can provide additional security by verifying the authenticity of devices and systems.
  5. Third-Party Risks:
    • Even if we deploy the right technologies in our own environment, we often ignore the security practices of our upstream providers. Several incidents involved third-party vendors or services (e.g., Target, Home Depot), highlighting the critical need for comprehensive third-party risk management. Organizations must ensure that their vendors adhere to strong security practices and protocols to mitigate potential risks.
  6. Mismanagement of API and Signing Keys:
    • Incidents such as the Google Cloud Key Leak and Amazon AWS Key Exposure point to misconfigurations and accidental exposures of API keys. Proper configuration management, continuous monitoring, and strict access controls are essential to prevent these exposures.
  7. Importance of Multi-Factor Authentication (MFA):
    • The absence of MFA has been a contributing factor in several breaches. Organizations need to mandate MFA to enhance security and reduce the risk of credential theft.
  8. Need for Secure Code Signing Practices:
    • The use of stolen code-signing certificates in attacks like Stuxnet and Nvidia underscores the importance of securing code-signing keys. Implementing hardware security modules (HSMs) for key storage and signing can help mitigate these risks.

Conclusion

Looking back at the past decade of key and credential compromises, it’s clear that we’re stuck in a loop, facing the same issues repeatedly. It’s not about knowing what to do—we know the solutions—but about taking action and making sometimes uncomfortable changes.

Organizations need to step up and embrace multi-factor authentication, adopt strong hardware-backed authenticators like Passkeys, and move workloads from shared secrets to hardware-backed asymmetric credentials for better security. We also can’t overlook the importance of managing third-party risks and ensuring proper configuration and monitoring to protect our API and signing keys.

Breaking free from this cycle means committing to these changes. By learning from past mistakes and taking a proactive approach to key and credential management, we can better protect our systems and customers – It’s time to move forward.

The Rebirth of Network Access Protection with Microsoft’s Zero Trust DNS

The other day Microsoft announced something it calls Zero Trust DNS. At a high level, it is leveraging clients’ underlying name resolution capabilities to establish and enforce security controls below the application.

In design, it is quite similar to what we did in Windows 2008 and Network Access Protection (NAP), a now deprecated solution that if released today would be considered some flavor of “Zero Trust” network access control.

NAP supported several different enforcement approaches for network policy. One of the less secure methods assessed a client’s posture; if found to be non-compliant with organizational requirements, the DHCP server would assign a restricted IP address configuration. The most secure approach relied on certificate-based IPSEC and IPv6. Healthy clients were provisioned with what we called a health certificate. All participating devices would then communicate only with IPSEC-authenticated traffic and drop the rest.

If ZeroTrust DNS had been released in 2008, it would have been another enforcement mode for Network Access Protection. It operates very similarly, essentially functioning on the basis that :

  1. The enterprise controls the endpoint and, through group policy, dictates DNS client behavior and network egress policy,
  2. Leverage mutual TLS and DNS over HTTPS (DoH) to authenticate clients, and
  3. Transform the DNS server into a policy server for network access.

I was always disappointed to see NAP get deprecated, especially the 802.1X and IPSEC-based enforcement models. Don’t get me wrong, there were many things we should have done differently, but the value of this kind of enforcement and visibility in an enterprise is immense. This is why, years later, the patterns in NAP were reborn in solutions like BeyondCorp and the myriad of “Zero Trust” solutions we see today.

So why might this ZeroTust DNS be interesting? One of the more common concerns I have heard from large enterprises is that the mass adoption of encrypted communications has made it hard for them to manage the security posture of their environment. This is because many of the controls they have historically relied on for security were designed around monitoring clear text traffic.

This is how we ended up with abominations like Enterprise Transport Security which intentionally weakens the TLS 1.3 protocol to enable attackers—erm, enterprises—to continue decrypting traffic. 

One of the theses of this Zero Trust DNS solution appears to be that by turning DNS into what we used to call a Policy Enforcement Point for the network enterprises get some of that visibility back. While they do not get cleartext traffic, they do get to reliably control and audit what domain names you resolve. When you combine that with egress network filtering, it has the potential to create a closed loop where an enterprise can have some confidence about where traffic is going and when. While I would not want my ISP to do any of this, I think it’s quite reasonable for an enterprise to do so; it’s their machine, their data, and their traffic. It also has the potential to be used as a way to make lateral movement in a network, when a compromise takes place, harder and maybe, in some cases, even make exfiltration harder.

Like all solutions that try to deliver network isolation properties, the sticking point comes back to how do you create useful policies that reduce your risk but still let work happen as usual. Having the rules based on high-level concepts like a DNS name should make this better, than with, for example, IPSEC-based isolation models, but it still won’t be trivial to manage. It looks like this will have all of those challenges still but that is true of all network segmentation approaches.

What I appreciate most about this is its potential to reduce an organization’s perceived need to deploy MiTM solutions. From a security protocol design perspective, what is happening here is a trade-off between metadata leakage and confidentiality. The MiTM solutions in use today cause numerous problems; they hinder the adoption of more secure protocol variants and objectively reduce enterprise security in at least one dimension. They also make rolling out new versions of TLS and more secure cryptography that much harder. Therefore, in my opinion, this is likely a good trade-off for some organizations.

To be clear, I do not believe that host-based logging of endpoint communications will lead all enterprises to abandon these MiTM practices. For example, some MiTM use cases focus on the intractable problems of Data Leak Protection, network traffic optimization, or single sign-on and privilege access management through protocol-level manipulation. These solutions clearly require cleartext access, and name-based access control and logging won’t be enough to persuade enterprises that rely on these technologies to move away from MiTM. However, there are some use cases where it might.

So, is this a good change or a bad change? I would say on average it’s probably good, and with some more investment from Microsoft, it could be a reasonable pattern to adopt for more granular network segmentation while giving enterprises more visibility in this increasingly encrypted world without needing to break TLS and other encryption schemes.