The WebPKI's Unmanaged Software Supply Chain

In The Private Key Was Four, I examined how eight eMudhra intermediate CA certificates ended up with private keys so weak that one of them was the integer 4. In Trusted Everywhere, Impact Everywhere, I looked at why a failure at a CA serving a small market becomes a problem for every relying party that trusts it, whether or not that relying party ever chose to use the CA.

This article is about a third problem, which gets the least attention of the three and is getting worse as AI agents take on more work. Much of the software ecosystem inherits the WebPKI's global trust without needing its full scope and without receiving the protections browsers built to manage it.

The stakes are hard to overstate. The web's security rests entirely on the WebPKI. Every HTTPS connection a browser makes to a public website is authenticated through it, browsers accept no alternative, and much of what software does besides browsing depends on it as well. Package registries, software update channels, cloud and SaaS APIs, OAuth and single sign-on flows, and the endpoints AI agents call are all reached over TLS connections authenticated by publicly trusted certificates. Some of those systems add their own protections, such as signed packages or pinned keys, but for most of them the certificate a client accepts is the whole of its proof that it is talking to the right party.

When the eMudhra incident was reported, Chrome blocked the affected intermediate public keys through CRLSet within roughly 100 minutes of receiving the report, and Mozilla distributed its own blocks through OneCRL. Those responses reflect years of investment in browser security, incident response, and trust management.

The VPN clients, mail servers, containers, embedded devices, and command-line tools that rely on the same CAs had no equivalent channel. Some received protection through their own updates or platform policies, others may still accept certificates that browsers now reject, and many of them never needed to trust the entire public WebPKI in the first place.

We can inventory the libraries our software depends on, identify their versions, and track vulnerabilities. Yet many applications cannot readily tell us which organizations are authorized to authenticate their remote services, whether those authorities are still accepted by major browsers, or how quickly a distrust decision would reach deployed instances. Those are security dependencies too.

Why does your application trust so many CAs?

Consider a VPN client that connects to one company's infrastructure. It might authenticate three or four service names, all controlled by the VPN provider, and those services could use a small set of CAs or, where the deployment model permits, a private PKI.

The client nonetheless inherits an operating system's general-purpose root store, which contains authorities that have no relationship with the VPN provider or its customers. That happens because it is the default. A TLS library loads the platform's root store, and the application works without any special configuration.

For a browser, broad trust is the right starting point, because a browser must reach practically any public website without asking the user to configure trust first. A thermostat, a VPN client, and a service that calls five known APIs have none of those requirements. A television may need its manufacturer's identity, software update, and telemetry endpoints. A mail server needs to reach other mail servers, but its transport authentication requirements differ from a browser's. A machine-to-machine application may have only a handful of intended peers.

When an unconstrained CA is trusted for TLS, it can potentially authenticate any DNS name, and the client does not check whether the legitimate service operator chose that CA. If an attacker obtains a CA signing key, or a CA issues an unauthorized certificate, the attacker can impersonate an unrelated service provided they can also get into the connection path. A correct hostname on a mathematically valid chain will not distinguish the fraudulent certificate from one the operator authorized.

CAA records provide a partial mitigation. A domain operator can publish which CAs are authorized to issue for its names, and the CA/Browser Forum Baseline Requirements require public CAs to check those records before issuance. That reduces accidental issuance and some forms of misissuance by otherwise compliant CAs. CAA is checked by the issuing CA, however, and not by the TLS client. A CA that disregards CAA, or an attacker who already possesses an intermediate signing key and issues certificates outside the CA's controls, is not stopped by the DNS record. The relying party still accepts a valid path to a trusted root unless it implements an additional restriction.

The WebPKI Observatory snapshot discussed in the previous article found that five CA organizations account for 88.8% of unexpired publicly trusted certificates, and in the Firefox telemetry from the same article, eMudhra accounted for 0.025% of successful validations. Those figures describe the market rather than any particular application, and a small CA can be essential to a specific application. They do make it hard to argue that every specialized program needs every organization in a browser-scale root store, and an application that knows its intended peers can usually define a far narrower policy than either figure suggests.

Every unnecessary trusted CA adds another organization whose failure could become the application's failure, and that exposure exists even when the root store is fully current.

A root store is not a root program

Chrome, Mozilla, Apple, Microsoft, and other browser and platform operators do more than publish trusted certificates. They run continuing security programs around those decisions, which set admission policy, take part in CA/Browser Forum standards work, review audit and compliance information, investigate incidents, and decide when a CA should be restricted or distrusted.

They also maintain protections beyond ordinary X.509 path validation. Chrome uses CRLSets for selected high-impact certificate blocks. Mozilla uses OneCRL and has developed CRLite. Browsers enforce Certificate Transparency for certificates within the scope of their public TLS policies, and they can ship emergency trust decisions without waiting for a conventional application upgrade.

These mechanisms are imperfect, since audits miss failures and root programs react to incidents as often as they prevent them. Certificate Transparency does not make an unauthorized certificate authorized, though where it is enforced it makes that certificate observable and makes unlogged certificates unusable. What root programs have is dedicated teams and operational channels for responding when an earlier trust decision proves wrong.

That is why the eMudhra response could happen so quickly. The security did not live in the root certificate. It depended on an organization able to investigate the report, decide what to block, and push that decision out. A root store is an artifact of a root program, not the program itself.

Copying Mozilla's root certificates does not reproduce its incident response operation, and it does not bring OneCRL, CRLite, CT enforcement, or emergency intermediate restrictions with it. Consuming an updated CA bundle, meaning a packaged copy of a root store such as a Linux distribution's ca-certificates package, does not reproduce a browser's validation behavior either.

How much of the program an application receives depends on how it validates certificates. Software that hands validation to the platform verifier on Windows or Apple platforms inherits part of the operating system vendor's root program, including the vendor's ability to push distrust decisions to the platform. Software that reads a CA bundle from disk, which is the common case on Linux, in containers, and in many language runtimes, receives only the list of root certificates and whatever validation logic its own TLS library applies.

When trust changes, copies do not necessarily follow

CA trust changes over time. Operators are acquired, validation practices fail, keys are exposed, and a browser may distrust an entire root or restrict a particular hierarchy. The Observatory's historical dataset records 16 trust actions from 2011 through 2024, covering both root distrusts and narrower restrictions, and the lesson for software is that a CA list captured at build time is not a lasting security policy.

Applications receive these decisions through very different channels. Some use continuously maintained operating system trust services. Others depend on Linux distribution packages, language runtime bundles, application updates, container rebuilds, or firmware releases. Python's widely used certifi package ships a copy of Mozilla's root list, and Node.js compiles Mozilla's root certificates into its binary, so both change only when the package or runtime is updated. Some software embeds root certificates directly and never refreshes them.

An old CA bundle may continue to accept a CA that every major browser has removed, and a current bundle does not solve the whole problem either. Suppose a root stays generally trusted while one intermediate CA's signing key is compromised. Removing the root would disrupt unrelated legitimate issuance, so a browser blocks the intermediate's public key instead. A TLS client that trusts the same root but never receives the browser's intermediate block gains nothing from refreshing its CA bundle.

The eMudhra response shows the distinction. Mozilla described blocking the intermediates with OneCRL while also removing the relevant root trust in NSS for consumers that do not receive OneCRL updates, and Google described a root store removal meant to protect consumers that do not receive CRLSet updates. Both measures acknowledge that downstream clients inherit different portions of the browser trust stack, and a device that never consumes the newer NSS release or CA bundle receives neither. Distrust is a security decision, and distributing it is a separate engineering problem.

The protections that don't come with the CA bundle

Certificate Transparency shows the gap most clearly. A conventional TLS client verifies the hostname, validity period, signatures, and chain to a trusted root. A browser applies additional public TLS policy on top of that, and for certificates covered by its CT rules, Chrome generally requires acceptable proof of logging. A fraudulent but otherwise valid certificate issued outside CT can be rejected by Chrome while passing ordinary chain validation in another client.

In its eMudhra investigation, Google reported that no TLS certificates chaining to the intermediates within the scope of its CT check had been logged before the block. That finding did not establish that no attacker had created an unlogged certificate. It showed that a CT-enforcing browser had a protective boundary that clients which only validate the chain do not have.

An attacker with a compromised signing key can construct a leaf certificate that is cryptographically valid. Exploiting it still requires presenting it to the intended client, but if that client neither enforces CT nor receives the relevant block, a correct TLS implementation will accept it. That is a policy gap rather than a vulnerability in the TLS library. Private PKI and non-HTTP protocols have their own authentication requirements and need not adopt Chrome's CT rules as written, but an application that relies on public CA authority needs to know which compensating controls it actually has rather than assume the root certificates bring them along. SMTP is a good example, since mail servers often rely on public CAs without a browser's CT enforcement or emergency blocks, and SMTP has its own authentication models, including DANE and MTA-STS, that need to be assessed on their own terms.

Revocation does not close the gap. I contributed to OCSP, and getting timely revocation information to every relying party remains one of the hardest problems in security protocol design. Online checking trades availability against failure behavior and raises privacy concerns. CAs have also been unreliable operators of the CRL and OCSP services relying parties would need, and they have a commercial reason to delay revocation, because revoking a customer's certificate disrupts that customer's service and strains the relationship. Repeated delays in revoking misissued certificates were among the compliance failures behind Entrust's 2024 distrust.

Every major browser has therefore moved toward indirect mechanisms, collecting revocation information from CAs and redistributing it to clients through channels the browser controls, such as CRLSets and CRLite. The direct channel is also disappearing. The CA/Browser Forum made OCSP optional for publicly trusted TLS certificates, and Let's Encrypt shut down its OCSP service in 2025. Software outside the browser that depends on checking revocation directly is relying on the services browsers chose not to depend on, and for a growing share of certificates there is no online responder left to ask.

Shorter certificate lifetimes reduce how long a compromised leaf certificate stays valid, and the industry's move toward them is sensible. An attacker who controls a trusted CA signing key, however, can keep issuing fresh short-lived certificates until clients stop trusting that CA, so short lifetimes do not substitute for propagating distrust.

The supply chain dependency your SBOM misses

We have become much better at asking what code our software contains. We produce software bills of materials, track package versions, scan containers, and work toward reproducible builds. We are less disciplined about asking whom that software is prepared to believe.

The effective trust policy comes from a chain of choices. A VPN client uses a TLS library such as OpenSSL, which, depending on its configuration, loads an application-supplied CA file, system paths, or another trust source. The program may run in a container whose CA bundle was copied from a base image, and its production configuration may differ from the developer's workstation. The developer can know exactly which library version shipped without knowing which authorities the deployed application accepts.

A library update can leave the effective root store unchanged. A base image update can change trusted authorities without touching the application binary. A root removal may appear in neither the application's version history nor its vulnerability reports.

One security update, multiple applications

Correcting this is complicated by the fact that these dependencies are shared. Two applications linked against the same OpenSSL installation, or reading the same system CA store, inherit each other's changes. When the operator of one application updates the shared library or removes a root to address a security problem, the other may lose a connection it legitimately needs or see validation behavior it never tested. The same update can also fix a vulnerability for both, so freezing the shared layer only trades one risk for another. A security change made for one workload becomes a compatibility event for the next, which is why the unit worth managing is the effective trust policy of each application, together with a coordinated way to update the infrastructure they share.

The root store is not executable code, but it changes how the application behaves. If an unnecessary or compromised CA authenticates an attacker-controlled endpoint, the application may disclose information, act on malicious responses, or authorize further activity without a single byte of its executable changing. Fully patched software can still rely on obsolete or excessive trust.

That makes root stores and validation policy part of the software supply chain in a meaningful security sense, because they determine the authority the software grants to outside parties. Unlike a vulnerable package, though, that relationship is rarely inventoried, assigned an owner, or tied to an update obligation.

The problem is older than AI, but agents make it worse

Agentic workloads concentrate the problem. Agents do most of their work through command-line tools, SDKs, package managers, and API clients running in Linux VMs and containers, which is exactly the environment where trust comes from a copied CA bundle with no CT enforcement and no emergency blocks. A single agent session may run curl, pip, npm, git, and a cloud CLI, each of which can resolve trust differently, inside a container image that may have been built months earlier.

When agents do use a browser, it is usually a headless one, and a headless browser looks like a browser without necessarily behaving like one. Playwright, one of the most widely used automation frameworks, launches Chromium with component updates disabled by default, and Puppeteer disables background networking. Component updates are the channel through which Chrome receives CRLSets and its Certificate Transparency log list. Chrome stops enforcing CT once that log list goes stale, and the Chromium builds these frameworks download are pinned to the framework's release rather than kept current. A long-lived or outdated automated browser can therefore lose protections its operator assumes are maintained automatically, and what any particular headless browser actually enforces depends on its build, its configuration, and how fresh its security data is.

A successful TLS connection can feed information into an autonomous workflow, in which an agent retrieves instructions, consumes API data, downloads dependencies, or takes actions based on what a remote service returns. Impersonating that service does not hand an attacker control of the agent, since permissions, provenance handling, and the treatment of untrusted content still matter, but compromised endpoint authentication can be the first step in a longer chain.

We are investing heavily in deciding what agents are allowed to do, while often leaving which remote services they are willing to trust to an inherited CA bundle. The risk is old, and what has changed is how much autonomous action now rests on an accepted TLS connection.

Narrow trust has its own failure mode

Most applications are not browsers, and the first question for any of them is whether it needs public WebPKI trust at all. Where destinations are controlled and limited, an application may use a dedicated private trust anchor, a narrower issuer policy, or another authenticated relationship. Narrower trust is still not a maintenance-free design. Issuers and service endpoints change, keys roll over, and cryptographic algorithms age, and a root or public-key pin becomes a source of outages if the application has no safe way to migrate it. The goal is to minimize unnecessary authorities and keep a workable path to replace the ones that remain. For applications with known peers, the useful boundary may be finer than the root store, so that each service accepts only designated issuers or keys, with planned rollover and emergency replacement. The decision that matters is which authorities may authenticate each destination.

The payment terminal ecosystem shows what happens when that path is missing. VeriSign negotiated with card reader makers to embed only the VeriSign public root in their terminals, which narrowed the terminals' trust but tied them to one public root and to the hash algorithms the public web supported at the time. When the CA/Browser Forum's January 1, 2016 deadline for SHA-1 issuance arrived, Worldpay still depended on more than 10,000 terminals that could not validate SHA-2 certificates, according to Mozilla's public policy discussion. By then, moving the servers to a private root would not have helped devices that had never been provisioned to trust one.

Browsers were left to choose between holding back the SHA-2 transition and allowing one CA, by then Symantec, to remain the only source of new SHA-1 certificates on the public web. Mozilla allowed a narrowly scoped, temporary exception for nine Worldpay certificates issued through Symantec, limited to 90 days and logged in Certificate Transparency, rather than a general extension of SHA-1 issuance.

The terminal makers' threat model was not the web's threat model, and the consequences of their decision landed on browsers and everyone who relied on them. Choosing a smaller trust store is only the beginning, because the application must also own how that trust is provisioned, updated, migrated, and withdrawn. A dedicated private trust anchor avoids coupling a device's lifecycle to public WebPKI deadlines, although it cannot rescue hardware that was never designed to change its cryptography. Skipping that lifecycle work up front does not eliminate its cost. It defers the bill and often hands it to someone else.

Manage trust as an intentional dependency

The right scope for trust depends on the application rather than on which CAs dominate the market, and applications that do need broad public interoperability face a different problem from those with known peers. They need timely trust decisions, a clear understanding of their own validation policy, and a path for emergency restrictions that does not depend on someone rebuilding the program at the worst possible moment. The rustls/upki project is one effort to close part of that gap. It brings CRLite-based revocation data to clients outside the browser through command-line, C, OpenSSL, and rustls interfaces, and lists Certificate Transparency enforcement among its future goals. It is worth watching as its deployment, policy ownership, and compatibility questions get worked out.

Any application can be held to a few accountability questions. Which CAs does it trust, and which of those does it need for its intended connections? Who decides when trust is withdrawn, and how does that decision reach deployed instances? Where its own services use public certificates, do their domains publish CAA records, and does the application enforce CT or other relevant restrictions, or does it only check that the signatures form a valid chain? These are ordinary supply chain questions applied to authentication authority.

Trust does not stop at the browser

Once a CA’s authority is copied into the software supply chain, it can persist long after browsers have restricted it, without the monitoring, policy enforcement, or emergency response machinery browsers built to protect their users, and often in software that never needed such broad trust.

We cannot eliminate every CA failure, but we can reduce the number of parties whose failures each application depends on, and we can make trust decisions visible, maintainable, and responsive to new evidence. The software industry has learned to inventory vulnerable code, and it should be just as serious about inventorying the organizations its software allows to authenticate the world. For software that only needs to trust a few known services, authentication authority should be an explicit, application-specific security policy, with an owner, an update mechanism, and a way to revoke trust when the evidence changes.

A root store is not just a file of public keys. It is a set of security dependencies, and we should manage it like one.

---

Sources and methodology

The eMudhra timeline and mitigations are described in Google's incident account and Mozilla's incident account. Market concentration and the 16 recorded trust actions follow the WebPKI Observatory snapshot discussed in Trusted Everywhere, Impact Everywhere, and those trust actions include restrictions as well as complete distrusts. The eMudhra validation share comes from Mozilla's cert.validation_success_by_ca_2 telemetry for Firefox 157 and 158 on the release channel, as aggregated in mcpherrinm/cert-count. The Worldpay account is grounded in Mozilla's February 2016 policy discussion and Mozilla's contemporaneous announcement. The Worldpay certificate count and conditions are also reported in CSO's coverage. CAA checking requirements are in the CA/Browser Forum Baseline Requirements. Runtime examples refer to certifi and Node.js. The discussion of UPKI treats it as an exploratory approach. Application scenarios are illustrative threat models, not claims that particular devices or services were compromised in the eMudhra incident.