The Private Key Was Four
Every time a browser loads a page over HTTPS, it is trusting a chain of signatures. A certificate authority signs the website’s certificate, and the browser accepts it because that authority sits on a short list the browser ships with. Publicly trusted CAs keep their top-level root keys offline and do their daily signing with intermediates, subordinate keys the root has vouched for. Whoever holds an intermediate’s private key can issue a certificate for any website, and browsers will accept it.
So everything rests on the private key staying secret and unguessable. For the elliptic curve in this story, P-384, a private key is a number picked at random from a range about 116 digits long. No computer that will ever be built could search that range.
In April 2025 eMudhra, a certificate authority based in India and trusted by Apple, Google, Microsoft and Mozilla, created a batch of new intermediates. The private key of one of them, EM OV TLS CA - G3C, was the number 4. Written out as a P-384 key it looks like this:
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000004A properly generated one looks like this:
84ac06790886c80c48d31691fd4c0fe9
ce3207f9807f2fe571d6e4ec426d7864
4298fc53df2b92566970e757ac998c34The keys came out of a key ceremony, the scripted and witnessed session in which a CA generates keys inside its hardware security modules. Seven sibling intermediates from the same ceremony were no better. Their private keys were 56, 66, 115, 155, 156, 230 and 237. Every one was below 256. Anyone with a copy of the certificate could recover the key by trying each number from 1 to 255 until one produced the public key printed in the certificate. I checked all eight myself. It takes less time than reading this sentence.
Those certificates were public the whole time. CAs must disclose every intermediate in the CCADB, the shared database the browser root programs use to track CAs, and these went in on 10 May 2025. On 17 October 2026 a researcher who goes by Wayne ran paranoid_crypto, a free key-testing tool Google released in 2022, over the CA certificates in CCADB. Twenty-seven minutes after he started, it flagged the eight keys. About two hours later he reported them to Apple, Chrome, Microsoft, Mozilla and eMudhra, with a message signed by each key as proof. Chrome blocked the six intermediates that could issue website certificates within two hours of his email, and Mozilla within five. By 20 October both had decided to remove the eMudhra root above them, emSign ECC Root CA - G3, entirely. On 22 October eMudhra announced it was leaving the global TLS certificate business.
The proof was barely needed. Dennis Jackson of Mozilla recovered all eight keys independently within two hours of receiving the report, and noted that AI agents running on his own machine recovered them on the first try when told the task was a capture-the-flag puzzle. Within three days several people had posted the keys, and one showed a forged google.com certificate working against an unpatched client.
For seventeen months, then, the private keys of six website-signing intermediates trusted by every major browser could be computed by anyone who looked. Wayne had even seen the first symptom. In June 2025 he commented on one of two public bug reports that, as it turns out, documented these exact keys misbehaving. eMudhra had the same evidence in hand that June. It knew its certificates did not hold the keys it meant them to hold, switched to the keys the certificates named, and kept operating for sixteen more months.
The CRLs that told the truth
The 30 April 2025 ceremony produced 24 issuing CAs in three series (A, B and C), each with RSA-4096 and P-384 versions for DV, OV, EV and code signing. The HSM mapping eMudhra posted shows the A series was generated on an Entrust nShield Connect XC, and the B and C series on a Utimaco Se-Series Gen2. Only the P-384 keys from the Utimaco went wrong. The RSA siblings from the same Utimaco look sound on every cheap test I ran (no shared factors with any other CCADB key, no Fermat weakness, no small factors), though cheap tests prove nothing about generation. The B series belonged to eMudhra BV in the Netherlands and the C series to eMudhra Inc in the US. eMudhra says they were built for data-sovereignty customers and only one, EM DV TLS CA - G3C, was ever switched on. None issued a certificate.
| Date | What happened |
|---|---|
| 30 Apr 2025 | Ceremony creates 24 CAs. Per eMudhra, a wrap and unwrap test produces a second, weak key pair for each B and C P-384 CA |
| 10 May 2025 | All 24 published to CCADB, two days late (Bug 1965559). From here the public keys are open to anyone |
| 14 May 2025 | Sectigo cross-signs all 24 from AAA Certificate Services |
| 22 May 2025 | Chrome tells eMudhra in Bug 1965559 that its earlier preventive actions failed and it is not learning from other CAs’ incidents |
| 23 May 2025 | Mozilla later reports an archived crt.sh page showing the six P-384 CRLs failing from this date |
| 29 May 2025 | The CRLs eMudhra dates the problem to are published. The A series is revoked and replaced by A-1 the same day, for reasons not yet explained |
| 3 Jun 2025 | Rob Stradling asks eMudhra whether the CRLs were signed with the wrong private key. The email is quarantined |
| 5 Jun 2025 | Stradling opens Bugs 1970727 and 1970728 |
| 6 Jun 2025 | eMudhra publishes CRLs that verify, signed with the weak keys |
| 18 Jun 2025 | Sectigo cross-signs the eight A-1 replacements |
| 16 Jul 2025 | Bug 1970728 closed |
| 8 Nov 2025 | Chrome reviews eMudhra’s hierarchy metadata during a root inclusion evaluation |
| 6 Aug 2026 | BDO issues unmodified WebTrust opinions naming Bug 1970728 |
| 2 to 9 Oct 2026 | Mozilla’s intent to approve eMudhra’s new roots and last call |
| 14 Oct 2026 | An eMudhra root CRL fails signature verification |
| 17 Oct 2026 | Wayne reports. Chrome and Mozilla block within hours |
| 20 Oct 2026 | Mozilla removes emSign ECC Root CA - G3 from NSS. Chrome commits to the same |
| 21 Oct 2026 | Chrome posts twelve questions for eMudhra in Bug 2083449 |
| 22 Oct 2026 | eMudhra announces it is leaving the global TLS certificate business |
On 29 May 2025 eMudhra published CRLs for six of the new intermediates, the same six TLS CAs whose keys were later recovered. None of the signatures verified. crt.sh flagged them, and on 3 June Rob Stradling of Sectigo sent eMudhra a problem report asking whether the CRLs had been signed with the wrong private key. The email sat in a spam quarantine. On 5 June he opened Bug 1970728.
eMudhra’s incident report said a manual CRL process had used an incorrect CA key configuration. The keys came from the issuing environment but did not match the keys expected for those CRLs. It regenerated the CRLs, and committed to automating CRL signing and verifying every CRL after signing. The bug closed on 16 July 2025.
The CRLs eMudhra attached to that bug say more than the report does. From an ECDSA signature and the signed data you can compute the small set of candidate public keys that could have made it, usually two. eMudhra attached five distinct CRLs, uploading one of them twice and leaving out EM DV TLS CA - G3B. I did that for each. Every candidate verifies the signature, none has a small private value, none matches the key in the CA’s certificate, and no candidate key appears in any of the 10,313 CA certificates in CCADB. So in June 2025 these CAs each had two keys on the HSM. One was a normal key that eMudhra’s systems treated as the CA’s key. The other was the key the certificate actually certified.
In October 2026 eMudhra explained what happened next. The CRLs had been signed with the original keys, the ones generated with appropriate entropy. Those signatures failed, so eMudhra re-signed them with the flawed keys. The fix for the invalid CRLs was to start using the weak keys, because they were the ones that matched the certificates.
That was the decision that mattered. eMudhra held cryptographic proof that its certificates did not contain the keys its systems treated as the CA’s, and it resolved the discrepancy in favor of the certificates. Then it kept those CAs in service for another sixteen months. The 2025 investigation reached the right symptom and stopped one question short. Both of its action items concerned the CRL signing path. Neither asked why a CA had two keys, which one was legitimate, or whether the certified key had been generated as intended. Answering that required no cryptanalysis, only treating “the expected key does not match the certificate” as a question about the certificate rather than about the CRL tooling.
According to eMudhra’s preliminary root cause, the 30 April ceremony included an approved deviation to test key wrap and unwrap, because the B and C series keys were meant for export to European and US data centers. A flaw in the script generated a second, weak key pair on the Utimaco HSM. The key labels were similar, and the CSRs were built from the weak keys.
eMudhra has not published the script, the mechanism and parameters it passed to C_WrapKey, or the attribute template it gave C_UnwrapKey. A wrong parameter, or a mismatch between the encoding the wrap produced and the one the unwrap expected, could turn a 384-bit scalar into a single byte. Every recovered key is below 256, which fits a parsing error far better than a weak random number generator. The ceremony needed a known-answer test before any production key went through that path. Wrap a key generated in software under a wrapping key the operator also holds in software, decrypt the blob to confirm its encoding, unwrap it on the HSM, and check that the resulting public key matches the original. That dry run belongs after every change to the script or the HSM firmware as well.
Commenters in the bug have pressed the same point. eMudhra’s own CP says issuing CA keys are generated in a device that is at least FIPS 140-2 Level 3. A P-384 scalar below 256 is not something approved in-module generation produces, so the weak keys most likely came from material the script supplied, through a call such as C_UnwrapKey or C_CreateObject. In that case the weak keys may never have been confined to an HSM at all. The private value existed outside the module wherever the script ran, and possibly in its logs, before it was loaded. With a key below 256 the secrecy question is moot, since anyone can compute it. The control question is not. A CA key that was loaded rather than generated inside the module does not meet eMudhra’s own CP. One commenter has asked eMudhra for the script, the exact call, whether the module was in its FIPS-approved mode, and who approved the deviation. Another proposed that root programs require a manufacturer-signed attestation that every CA key was generated inside the HSM rather than loaded into it. eMudhra’s public statement says it has not yet confirmed the exact root cause.
The 2025 report even has a section titled “Where we got lucky”. It says the new intermediates had not yet issued any end-entity certificates, which limited the exposure. That was true, and for a reason the authors could not see. Its lessons-learned section called for automating manual steps in creating new issuing CAs. Automating CRL signing against the certified key would only have put the weak keys to work sooner.
The auditors saw this incident too. BDO’s WebTrust report for June 2025 through May 2026 lists all eight affected intermediates in scope, names Bug 1970728 as an Other Matter, and says the auditor examined the disclosed root cause. The opinion was unmodified. The ceremony itself fell in the prior audit period, and I have not been able to find that report.
Why no one else looked
Once eMudhra chose the certificates over its own key records, nothing else in the system was positioned to catch it. Six publicly trusted keys sat in plain view for seventeen months because no rule asked anyone except eMudhra to look, and the one check the rules recommend would have passed them.
The Baseline Requirements treat CA key generation as a process control. For root keys and for subordinate CAs run by an unaffiliated operator, a Qualified Auditor must witness or video the ceremony and then report on whether it followed the script. eMudhra BV and eMudhra Inc are affiliates, so for these keys both were a SHOULD. A commenter in the bug has asked whether Sectigo’s cross-certification two weeks later changes that, since relative to Sectigo’s root eMudhra is an unaffiliated operator. The ceremony would still have had internal attendees and witnesses. eMudhra still had to keep the script, logs and witness records as cradle-to-grave evidence of each key, and the period audit is where that evidence gets examined. The weak-key rules in section 6.1.1.3, covering Debian keys, ROCA and close primes, apply to subscriber keys. Nothing tells an issuing CA, a cross-signer or a root program to screen a subordinate CA’s key that way.
The ECDSA quality check in section 6.1.6 is NIST public-key validation, and it is a SHOULD. It confirms the point is on the curve and has the right order. A private key of 4 produces a perfectly valid point. Public-key validation looks only at the point, so by design it cannot tell how the private key was chosen. The recommended check would have passed every one of these keys.
The check that did work has existed for years. Google published paranoid_crypto in June 2022. Its CheckWeakECPrivateKey test, a default check rated critical, looks for EC keys whose private value fits in 32 bits. It was public nearly three years before these keys were generated.
I wanted to know what running it at disclosure time would cost, so I pulled every CA certificate in CCADB, 10,313 of them, and tested each EC key for a private scalar below 65,536. The scan took 0.3 seconds after a one-time table build of under a minute. It returned exactly sixteen certificates. Eight were the eMudhra intermediates and eight were Sectigo’s cross-certificates of the same keys. There were no false positives. A batch GCD across all 6,669 unique RSA moduli found no shared factors either.
Several parties had a reason to look. Chrome reviewed eMudhra’s hierarchy metadata in November 2025 during a root inclusion evaluation. Mozilla posted its intent to approve eMudhra’s new roots, including emSign Root TLS CA - G3, the root that issued the six TLS intermediates, on 2 October 2026. The CCADB public discussion ran from August to September without objection. Sectigo cross-signed all 24 keys from the April ceremony. None of these processes is designed to test key values, and none of them caught the problem.
| Who | What they had | What the rules asked of them | What happened |
|---|---|---|---|
| eMudhra, at the ceremony | The HSM, both key pairs, the CSRs | Effective controls so keys are generated as the CPS and script describe (BR 6.1.1.1) | A script flaw and similar key labels; CSRs built from the weak keys |
| eMudhra, in June 2025 | Six CRLs signed by keys matching no certificate, and a report asking about the wrong key | A root cause analysis that reaches the real cause | Fixed the CRL path, re-signed with the weak keys |
| Third-party auditor at the ceremony | Internal attendees and witnesses were present; unknown whether a Qualified Auditor attended | Auditor witness or video, and an auditor’s key generation report, are a SHOULD for affiliate sub-CAs (BR 6.1.1.1) | No public record of auditor attendance or a key generation report |
| Auditor for the ceremony period (June 2024 to May 2025) | The ceremony script, logs and witness records eMudhra must keep as key lifecycle evidence | WebTrust key generation and lifecycle criteria, tested in the period audit | Period audit report not found |
| BDO, June 2025 to May 2026 | Bug 1970728 and eMudhra’s root cause | Reasonable assurance on key integrity | Examined it, unmodified opinion |
| Sectigo, as cross-signer | 32 eMudhra keys across two batches | No rule to test a subject CA’s key | No evidence of a key test |
| Sectigo, via crt.sh | Automated CRL verification | None | Raised the only public warning |
| Root programs | CCADB records from May 2025, an inclusion review | Disclosure, audits, incident handling | No key scan; nothing requires one |
| Certificate Transparency | Nothing, since no certificate was issued | None | Could not have seen it |
| Researchers | Public keys from May 2025, a free tool from 2022 | None | Found in 27 minutes, seventeen months late |
Each layer of WebPKI assurance was built for a different failure. Audits test whether controls exist. Root programs watch disclosure and incident handling. Certificate Transparency watches issuance, and these CAs never issued a logged certificate, so CT had nothing to see. Key validation tests curve membership. The one property that was wrong sat outside all of them.
Where the keys reached
The six TLS intermediates were issued by emSign Root TLS CA - G3, a 2024 root that only Microsoft includes directly. Mozilla, Chrome and Apple trusted them through a cross-certificate from emSign ECC Root CA - G3, which has been in all four programs since 2019. The two code-signing CAs mattered only on Windows, where anyone could have signed software that chained to a trusted root until Microsoft’s Disallowed list update reached the machine.
The third path is the one most coverage has missed. On 14 May 2025 Sectigo cross-signed every CA from the April ceremony, 24 certificates, from its AAA Certificate Services root. The eight affected keys therefore appear in sixteen certificates.
AAA Certificate Services dates from 2004 and sits in most trust stores built in the last twenty years. Mozilla removed its Websites trust bit in April 2025 and Chrome dropped it from the Chrome Root Store the same month. Apple and Microsoft still trust it for TLS, and Microsoft for code signing. Every Mozilla-derived bundle from before April 2025 trusts it for TLS. A device with a 2015 trust store never trusted eMudhra, and it still accepts a forged certificate chained through Sectigo’s cross-cert. That path closes when AAA expires on 31 December 2028. The path through emSign ECC Root CA - G3 lasts until 2039 in any store that keeps it.
The constraints narrow what a forger can do, but not by much. The TLS intermediates carry no name constraints, so they can sign for any domain. They are limited to path length zero, so a forger cannot create a new intermediate. Their EKU includes client authentication as well as server authentication.
The timing of the cross-signs is the governance story. Mozilla requires approval before a root in its program cross-signs a CA outside it. Chrome requires notice and express approval for certificates that extend its trust boundary. Sectigo’s cross-signs came three weeks after AAA left Chrome and a month after Mozilla stopped trusting it for TLS, so neither rule applied. Apple added a prior-approval rule for external cross-signs on 1 August 2026, fifteen months later. Microsoft requires disclosure and audits, not approval. The cross-signs reached exactly the stores with no approval gate.
| Program | Rule for cross-signing a CA outside the program | Since | Applied to the 14 May 2025 cross-certs? |
|---|---|---|---|
| Mozilla | Approval before cross-signing a CA not in Mozilla’s store; the signer is accountable for it (MRSP 8.4) | 2022 | No. AAA lost TLS trust on 16 April 2025 (Bug 1957685) |
| Chrome | Three weeks’ notice and express approval for certificates that extend the trust boundary (policy 1.6.1) | 2022 | No. AAA removed on 24 April 2025 |
| Apple | Approval before each cross-signed certificate to an externally operated CA (policy 1.4) | 1 Aug 2026 | No. The rule came later |
| Microsoft | Disclosure and audits only (requirements) | No rule to apply |
Precedent says this matters. Mozilla removed Certinomis in 2019 partly because it cross-signed StartCom, a CA Mozilla was in the middle of distrusting. Today’s approval rules each cover their own store. None can reach a root that has already left. Revocation follows the same boundary. Asked in the bug whether the AAA cross-certs belonged in OneCRL, Mozilla replied that AAA is not in its root store, so they do not.
When one store retires a root and another keeps it, cross-signs from that root become the route around the first store’s rules. Sectigo’s cross-signs took that route. Cross-signing from a root that browsers are retiring is common practice, Sectigo did the same for SSL.com in 2024, and no rule asks a cross-signer to test the subject CA’s key. Two questions still belong to Sectigo. It cross-signed eight more eMudhra CAs on 18 June 2025, two weeks after its own staff flagged wrong-key signatures on CAs it had already cross-signed. And it extended a root that browsers had just retired into the platforms with the least oversight of cross-signs.
The response, and who it missed
The browser response was fast and well run. Chrome pushed a CRLSet block on the six TLS keys at 19:45 UTC on 17 October, under two hours after Wayne’s email. Mozilla, which received the report through Chrome at 20:06, recovered the keys independently by 21:48 and completed its OneCRL block at 22:59. Google extended the block to Android through Play Services on 19 October. Apple blocked the certificates on 19 October and Microsoft added all sixteen to its Disallowed list on 20 October. eMudhra revoked its eight between 09:24 and 09:38 UTC on 18 October, and Sectigo revoked its cross-certs at 10:35 UTC on 19 October.
The mechanisms that worked best matched on the public key rather than the certificate. Chrome’s CRLSet, Microsoft’s list, Go’s patched certificate verifier and the CT log software patches all blocked by key, so the cross-certs were covered without anyone having to enumerate them. Cloudflare’s log patch says so in its commit message. Mechanisms keyed on issuer and serial need an entry per certificate, and every cross-sign is one more entry to miss.
| Mechanism | What it blocked | Live | Matches on |
|---|---|---|---|
| Chrome CRLSet | Six TLS intermediates and their cross-certs | 17 Oct, 19:45 UTC | Public key |
| Mozilla OneCRL | Six TLS intermediates | 17 Oct, 22:59 UTC | Issuer and serial |
| Android, via Play Services | Same as CRLSet | about 19 Oct | Public key |
| Apple | The affected certificates; mechanism not documented | 19 Oct; iOS clients still accepting on 21 Oct | Unknown |
| Microsoft Disallowed list | All sixteen certificates | 20 Oct | Certificate |
| CT log software (Sunlight, TesseraCT, certificate-transparency-go, Azul) | Submissions chaining to the eight keys | Released 19 to 20 Oct; deployed log by log | Public key |
| Go crypto/x509 (CVE-2026-111943) | The eight keys, hard-coded | Go 1.27.3 and 1.26.10 | Public key |
| NSS root removal | emSign ECC Root CA - G3 | NSS 3.131.1, 20 Oct | Root |
| Chrome Root Store | emSign ECC Root CA - G3 | v43, about 27 Oct | Root |
Pushed lists like CRLSets are how browsers revoke today. Live CRL and OCSP checking is broken on the web for many legitimate reasons, including latency, privacy and unreachable responders, so each browser ships its own revocation data instead. Chrome uses CRLSets, Firefox uses OneCRL and CRLite, and Apple and Microsoft push their own lists.
Certificate Transparency did not detect the weak keys, but it recorded their use once they were public. Chrome reports that no certificate chaining to these CAs was logged before the block. The log software fixes that reject these chains by key shipped on 19 and 20 October, but each operator deployed them on its own schedule, and until it did, its logs accepted whatever chained to a trusted root. People used that gap. Cloudflare closed submissions to three shards of its Raio log on 20 October to stop what it called the eMudhra DoS vector, and the ctlogs.dev monitor counted 105,901 certificates naming google.com, issued from EM EV TLS CA - G3B and logged on 21 October.
I scanned every entry in two of those logs, Microsec Eszigno2027h1 and GoDaddy Aquamarine2027h1, from 18 October to the morning of 22 October. Both tell the same story. The first certificate chaining to a weak key was logged at 09:40 UTC on 20 October, for emsign.filippo.io. That evening came pairs issued two minutes apart with the same key, one chained through eMudhra’s root and one through the Sectigo cross-cert, for markgamache.com and emudhra.salesforce.com, carrying no AIA, CRL distribution point or policy extensions. Three more single certificates followed on 21 October. Then, between 21:42 and 21:57 UTC, came 1,042 at once in each log, 1,030 of them for .invalid names and 12 for the community site’s own test hosts. By the morning of 22 October Microsec’s log held 1,050 such certificates and GoDaddy’s held 1,055, one of them logged twice. All of them look like researchers and pranksters testing the keys, not like an attack in progress.
Microsec and GoDaddy are separate operators, and Chrome had qualified both logs in September, so each of these precertificates collected SCTs from two independent qualified logs. Two of the pairs, all issued by EM DV TLS CA - G3B with a P-256 subject key, as recorded in each log:
| Names | Chain submitted | Serial | Logged (UTC) | Microsec index | GoDaddy index |
|---|---|---|---|---|---|
| emudhra.salesforce.com | eMudhra root | 39d03c305fc13f907618681fe3cf7cb3 | 20 Oct 2026, 23:46:08 | 21768393 | 141565655 |
| emudhra.salesforce.com | Sectigo AAA cross-cert | 6b5995b57a535b8ecf7839a79064afaf | 20 Oct 2026, 23:44:30 | 21768345 | 141565082 |
| markgamache.com, www.markgamache.com | eMudhra root | 02e0978c639f7e464f9ba5a4e46d9e85 | 20 Oct 2026, 23:38:47 | 21768199 | 141563038 |
| markgamache.com, www.markgamache.com | Sectigo AAA cross-cert | 228939ef131f959dcb20510501886c35 | 20 Oct 2026, 23:37:08 | 21768160 | 141562540 |
During that gap, CT on its own stopped nothing. A forgery could collect SCTs from logs that still accepted these chains, and what refused it in a browser was the browser’s own key block. Only once a log enforced the fix did it become containment, because a forged certificate could no longer be logged there. A forged certificate presented to a client that does not enforce CT was never going to be logged or seen.
On 20 October Mozilla removed emSign ECC Root CA - G3 from NSS. Firefox was already protected by OneCRL. Mozilla removed the root for the software downstream of NSS that copies its root list but never reads OneCRL. That is a root program acting on behalf of clients it does not ship, and it is the clearest admission I have seen that browser-side revocation does not reach the rest of the Internet.
The rest of the Internet is patching unevenly. certifi, webpki-roots, Node.js and Debian removed the root within days, and Go shipped a hard-coded key block. As of 21 October the OpenJDK cacerts file and the AOSP system store still contain both emSign ECC Root CA - G3 and AAA Certificate Services. Java does not check revocation by default.
| Trust store | Status on 21 October |
|---|---|
| python-certifi | 2026.10.19 removes emSign ECC Root CA - G3. Versions from before April 2025 also carry AAA for TLS |
| rustls webpki-roots | 1.0.10 removes emSign ECC Root CA - G3 |
| Node.js | v24.22.1 removes emSign ECC Root CA - G3 |
| Debian | Blocks emSign ECC Root CA - G3 and AAA |
| Ubuntu | Blocks emSign ECC Root CA - G3 |
| Go | Hard-coded key block |
| OpenJDK cacerts | Still contains both roots |
| AOSP system store | Still contains both roots; Play Services block covers some apps |
Announcing a block is not the same as delivering it, and in this incident the gap can be measured. The community site that carries the proof, patch list and workarounds, we-have-a-problem.com, runs test hosts that serve certificates from each compromised intermediate, directly and through the Sectigo cross-certs, and counts the clients that accept them. In the five minutes before 23:04 UTC on 21 October, 19,415 of 139,023 sampled connections succeeded. Safari on iOS was the largest group that accepted, two days after Apple’s block. Edge on Windows, Chrome on Android and the curl that ships with macOS appeared too. The counts come from people testing their own devices, and user agents can be faked, so they show which kinds of clients still accepted the certificates rather than how many. Firefox had a delivery gap of its own. Some Firefox for Android profiles did not apply the updated blocklist at startup until Mozilla shipped a fix in 158.0.2. CT records what was issued under these keys, not which clients would accept it, so measurement like this is the only view of that side. Root programs timestamp their blocks, but I have found none that publishes how far a block has reached. Current counts are on the site’s live statistics page.
A client is exposed only if four things are true together. It trusts either root for TLS. It has not received a block list or root update, which for a browser is its revocation check. It does not enforce CT. It does not fetch CRLs or OCSP responses itself, or fails open when it cannot reach them. Patched browsers fail the second condition. Many appliances, embedded systems and long-lived applications meet all four. We cannot rule out that someone found these keys before Wayne and used them against such clients, because an attacker never needed eMudhra’s HSM and never needed to log anything.
What happens to eMudhra
On 22 October eMudhra announced that it is leaving the global SSL/TLS certificate business. The board called the decision voluntary. It cited TLS being about 1 percent of revenue, the cost of data-sovereignty requirements, competition from free certificates, and a “level of social media amplification around this incident” that “has brought scrutiny that is out of proportion to SSL/TLS’s place in our business.” That treats public attention as the problem. The problem is CA keys anyone could compute, which eMudhra had evidence of sixteen months earlier. The release does not mention the June 2025 CRLs or Chrome’s twelve questions, and it presents the exit as a business decision made under unfair pressure. The scrutiny was in proportion to the failure. A browser does not trust a CA any less because TLS is 1 percent of its revenue, and a CA key anyone can compute does no less damage. By then Mozilla had removed emSign ECC Root CA - G3, Chrome’s removal was set for 27 October, and Chrome had said it was investigating eMudhra’s remaining CAs. The exit is voluntary only in the sense that it came before the distrust that was coming anyway.
The same release says eMudhra handled the incident diligently, revoked the affected CAs within 14 hours of discovery, and suffered no compromise of its infrastructure. Its incident statement says it learned of the issue late on 18 October, India time, and revoked on 19 October. The revocation dates recorded for the eight certificates are 09:24 to 09:38 UTC on 18 October, mid-afternoon in India and about fifteen hours after Wayne’s report was sent to eMudhra directly. eMudhra’s public timeline puts its revocations a day after its own revocation records do. Nobody broke into eMudhra’s infrastructure because nobody needed to. The statement also says the impact is “contained to a single customer ecosystem in the United States.” Chrome’s CT count shows which one. Of about 30,800 unexpired precertificates chaining to the root, 30,791 came from two eMudhra-operated intermediates branded for InCommon, InCommon Intermediate CA - OVG3C and DVG3C, which serve the InCommon certificate service for US universities. They are separate CAs from the eight with weak keys, which issued nothing. Eighteen came from eMudhra’s own intermediates. eMudhra put the number needing migration at about 30,000, 97 percent issued through ACME, and is moving them to its RSA root, emSign Root CA - G1, which Mozilla kept “at this time”.
On 21 October Chrome posted twelve questions in the bug. They include when eMudhra first knew the B and C CAs had two key pairs, why it re-signed the CRLs with the second key instead of halting, why its 2025 report left out the duplicate keys and the ceremony deviation, and whether Sectigo or any root program was told that the cross-certified keys were not the intended ones. Chrome also noted that eMudhra’s status page and press release refer only to “eMudhra’s ECC root” and do not name the affected root or intermediates, give removal dates, or tell subscribers how to check their chain. eMudhra announced its exit the next day.
The precedents that matter are the recent ones. Chrome distrusted Entrust in 2024, and Chunghwa Telecom and NetLock in 2025, citing a pattern of compliance failures and unmet improvement commitments. Each case used a forward-dated distrust, so existing certificates kept working while new ones stopped. eMudhra fit that pattern more closely than any of them. It has 31 compliance incidents in Bugzilla since 2020, twelve of them in 2026. In May 2025 Chrome wrote in Bug 1965559 that eMudhra was not applying lessons from other CAs’ incidents and that its earlier preventive actions had failed. The fix for the 2025 CRL incident did not just miss the weak keys. It put them into service.
One more recurrence sharpens that. On 14 October 2026, three days before Wayne’s report, eMudhra published CRL 11 for emSign Root TLS CA - G1. Wayne flagged it in the bug, and I checked it. The signature does not verify under the root’s key, and CRL 12, with identical entries, verified fourteen hours later. Invalid CRL signatures were the failure eMudhra promised in 2025 to catch by verifying every CRL after signing.
The rest of the record shows a CA whose own monitoring finds less each year. Through 2024 eMudhra detected and filed most of its own incidents. From June 2025 most were found by others, including Sectigo, Chrome, outside researchers, customers and problem reports. All five incidents eMudhra filed between 12 and 16 October 2026 name a third party as the source. A 2024 incident found by Google (Bug 1931683) is the closest relative of this one. It involved a key-blocking control that checked only some reissue paths. Key validation implemented narrowly is a pattern, not a one-off.
One removal that landed the same week is not a consequence. On 19 October eMudhra asked Mozilla to remove two older dual-purpose roots, emSign Root CA - C1 and emSign ECC Root CA - C3, completing a retirement plan from February 2026 (Bug 2083709). Coverage that counts three roots removed is counting a scheduled retirement as a sanction.
| Year | CA | Trigger | Outcome |
|---|---|---|---|
| 2011 | DigiNotar | Intrusion and rogue Google certificates, concealed | Full removal everywhere |
| 2011 | DigiCert Sdn Bhd | 22 certificates with 512-bit keys | Intermediate distrusted |
| 2019 | Certinomis | Cross-signed StartCom, plus misissuance | Mozilla removal |
| 2021 | Camerfirma | Repeated misissuance and delayed revocation | Chrome, Mozilla, later Apple |
| 2023 | e-Tugra | Insecure systems attached to the CA, weak response | Mozilla removal |
| 2024 | Entrust | Compliance failures, unmet improvement commitments | Forward-dated distrust in Chrome, Mozilla, Apple |
| 2025 | Chunghwa Telecom, NetLock | Same rationale | Chrome forward-dated distrust |
| 2026 | eMudhra | CA keys anyone could compute | Intermediates blocked in hours, one root removed in three days, exit from global TLS announced in five |
This case is new in two ways. No earlier WebPKI CA key was derivable from public data while trusted. DigiNotar’s keys were used by an intruder, not exposed to everyone. And no earlier root removal was justified mainly by clients the root program does not ship.
Mozilla was days from approving a new eMudhra root when the keys surfaced. It posted its intent to approve emSign Root TLS CA - G3 on 2 October 2026. In that thread, a participant asked for an overview of eMudhra’s incidents and Mozilla replied that it lacked the capacity to prepare one. Last call closed on 9 October. Eight days later, the root it would have approved turned out to have issued six intermediates with keys anyone could compute. Root programs are run by small teams, and that capacity is the resource this incident spent.
After classical PKI
Merkle Tree Certificates would have removed most of the structures this incident moved through. Under draft-ietf-plants-merkle-tree-certs a CA’s key signs batches of its own issuance log, clients require cosignatures from independent mirrors, and there are no intermediates or cross-signs. Chrome’s draft policy for its quantum-resistant root store forbids subordinate CAs outright. There would have been no side ceremony for regional intermediates and no cross-cert to carry them into Apple and Microsoft. A client could distrust a CA’s issuance after a given log index, one cut in place of per-certificate block lists, and would never accept an unlogged forgery.
MTC does not fix key generation. A CA key with a private value of 4 is just as forgeable in a Merkle tree, and mirrors only make the forged entries visible. It also does nothing for the clients exposed here, because they are the ones that will never adopt it. Chrome plans an MTC-only quantum-resistant root store for the third quarter of 2027, alongside the existing one. Removing non-MTC CAs from the Chrome Root Store appears only as a second-half 2028 target on a slide titled “An (optimistic) timeline”, and that slide notes no impact on user-added roots or private PKI.
When the browsers that fund root-program oversight stop depending on classical CAs, those roots will stay in frozen trust stores. The path through emSign ECC Root CA - G3 stays valid until 2039 in any store that keeps it. Classical WebPKI could outlive the people supervising it, and the eMudhra inclusion review shows how thin that supervision already is. A device that will run for fifteen years should trust a private root its operator controls and can rotate, with a managed path for trust updates. I covered the MTC side of this in From Chains to Trees and The Post-Quantum WebPKI.
What we still don’t know
- When eMudhra’s staff first realized each CA had two keys, and who decided to sign with the certified ones instead of halting. eMudhra says it re-signed the CRLs with the flawed keys, so someone made that choice. Chrome asked both questions in the bug on 21 October.
- What the wrap and unwrap script actually called. The mechanism, its parameters and the C_UnwrapKey template would show how a 384-bit key came out as a single byte.
- Who audited the ceremony period, and what they tested. The June 2024 to May 2025 report covers 30 April 2025. I have not found it. Nor have I found an auditor’s key generation report for that date, which for affiliate sub-CAs is optional.
- What BDO did with Bug 1970728. Its report says it examined the root cause. Whether that included reconciling HSM keys to certificates is not public.
- Whether Sectigo tested anything when it cross-signed. Its policy for external cross-certification has not been discussed in the bug. Chrome has asked whether Sectigo or any root program was told in 2025 that the cross-certified keys were not the intended ones.
- Why the A series was revoked and replaced on 29 May 2025, the same day as the bad CRLs. Chrome has asked. eMudhra says the A and A-1 series were generated on Entrust HSMs for use in India, and Mozilla found none of their keys recoverable.
- Which keys went to the European and US data centers. The B and C series were generated for export. If the strong keys were the ones shipped, the regional deployments held keys that matched no certificate and could never have produced a valid signature. That would mean the export path was never tested end to end.
- What signed the bad root CRL on 14 October 2026, and whether other eMudhra roots have more than one key pair. Chrome has asked both.
- What Apple and Microsoft will do with the remaining eMudhra roots, and whether Microsoft acts on the code-signing roots.
- Whether anyone used the keys before Wayne. HSM logs cannot answer this, because an attacker never needed the HSM.
Who runs the check
A CA has one job. It holds keys that only it can use, and it knows which keys those are. Ceremonies go wrong, and the script bug is not the failure that matters most. The failure is that eMudhra had cryptographic proof in June 2025 that its certificates did not hold the keys it thought they held, resolved the discrepancy in favor of the certificates, and kept going.
Every layer of WebPKI oversight did what it was designed to do. Audits examined controls. Root programs examined disclosures and incident reports. CT examined issuance. Formal key validation examined curve membership. None of them examined the keys, and nothing required any of them to. We have become good at documenting that controls ran. We are much worse at checking what those controls produced.
The fix is cheap enough that the absence of an owner is the only obstacle. CCADB already receives every CA certificate within seven days of creation. Running paranoid_crypto, or the twenty lines it took me to scan for small scalars, at upload time costs less than a second for the whole corpus. Run it on the certificate, again on any cross-certificate of the same key, and publish the results. Make a failure block disclosure until the CA explains it. Root programs could require it in a policy update, and CCADB could run it without one. They could also require what one commenter in the bug proposed, a signed attestation from the HSM manufacturer that each CA key was generated inside the module.
This is not over. The bug is still open, eMudhra has promised root programs a full root-cause report, and none of Chrome’s twelve questions has been answered. I am sure we will learn more, and I will update this post as we do.
Wayne found these keys with a free tool and twenty-seven minutes. The rest of the system had seventeen months, a public incident on the same six certificates, two audit reports and an inclusion review. Plenty of people could have caught it. Nobody was assigned to.
Sources
- Mozilla Bugzilla: 2083449 (compromise), 1970728 (2025 CRLs), 1970727, 1965559, 1999241, 2084101, 1957685, 1931683, 2083709
- dev-security-policy: incident and operator guidance thread, eMudhra inclusion thread
- Google: incident post, quantum-safe HTTPS plan, PQWS slides, paranoid_crypto
- Community: we-have-a-problem.com (proof, patch list, workarounds), connection statistics
- Audits: BDO WebTrust for CAs, June 2025 to May 2026, BDO WebTrust BR
- Policies: Baseline Requirements, Mozilla Root Store Policy, Chrome Root Program, Apple Root Program, Microsoft Trusted Root Program
- Patches: Go 1.27.3 and 1.26.10, NSS 3.131.1, certifi 2026.10.19, webpki-roots 1.0.10, Node.js v24.22.1, Debian, Ubuntu
- Data: CCADB resources (certificate records and PEMs), crt.sh, ctlogs.dev, Cloudflare’s Azul commit, Raio shard closure, impacted certificates and revocation dates
- draft-ietf-plants-merkle-tree-certs, Mozilla Security Blog post
- eMudhra: incident statement, exit announcement
Key recovery, CRL key recovery and the CCADB scans are my own analysis of the public artifacts listed above.