You need to use FIPS 140 because of compliance, but have you ever asked what that requirement is actually for? What security properties are the authors of these policies trying to achieve?
In high-assurance deployments, the practical goal is usually to establish a meaningful security boundary around cryptographic keys. Organizations want explicit controls over who and what can use a key, and they do not want the answer to be every application or administrator with access to the host. They are also worried about key theft and abuse. For important signing and decryption keys, keeping the key out of the hands of the application and host OS is often the simplest way to force reasonable key-protection practices.
These are real problems. When the threat group Storm-0558 acquired a highly sensitive Microsoft MSA signing key, operational failures allowed key material to escape the isolated signing environment and become accessible from a compromised engineering environment. That single extraction let the attackers forge tokens and compromise customer email accounts at scale. It is much harder to see that happening when a key is non-exportable and managed inside a hardware boundary. HSMs are not the only way to get these properties, but they are the one tool that forces you to think hard about how you operationalize a key, and that discipline has value.
The trade-off is that you end up running old software you cannot patch for upstream security vulnerabilities. In the best case, you are years behind.
Worse, this is usually non-memory-safe code that is entirely opaque to you. The firmware, the middleware, and the technical documentation are kept strictly behind lock and key by the vendor. You cannot inspect the code to see if it is vulnerable, and independent review is virtually impossible. The HSM vendor may have proactively patched those vulnerabilities, but more likely they have not.
The cryptography in these modules almost never breaks. What breaks is the plumbing. For example, when an HSM takes in a payload like an administrative command or an authentication token of some sort, it has to parse it. This may mean relying on complex parsers or business logic, often written in aging C code. Because that plumbing is not memory safe, a single malformed input can lead to a buffer overflow or remote code execution right past the validated boundary, or even worse, a long-forgotten feature combined with new code could bypass policy controls around accessing the module altogether.
Take the U-Boot forks as another example. When you look at how these embedded systems are actually built, they lean heavily on bootloaders, embedded operating systems, and vendor firmware that sit outside the cryptographic boundary but inside your trust story. A vendor might fork U-Boot, validate their module, and then that code is essentially frozen.
Think about what boot verification actually involves. Something has to parse the firmware image, figure out which bytes are covered by the signature, compute the digest, and check the signature. The math for the signature might be FIPS validated, but the parsing, the offset arithmetic, and the handling of attacker-controlled structures is all just non-memory-safe C code.
When researchers find vulnerabilities in X.509 parsing or U-Boot image processing, the flaws are not in the cryptography. They are in the plumbing. And because vendor version strings in this opaque, locked-down firmware often have no meaningful relationship to upstream release numbers, that fork may be a decade old.
The real danger is that two curves are moving in opposite directions. The attack surface is being examined continuously. AI and automated variant analysis are getting exponentially better at finding previously unknown exploitable memory-safety bugs in old C code. Meanwhile, the validated base firmware remains essentially fixed.
We do not have to guess about this. I recently published a data project analyzing recent FIPS 140-3 validations. The data shows how frozen these systems actually are. Out of 415 validated modules we looked at, 324 of them, 78 percent, showed no recorded public update after their initial validation. The knowledge of security issues compounding every week is colliding with code that barely moves.
FIPS 140 compliance buys you necessary domain separation and forces good operational habits. But the way the market achieves that separation often leaves you depending on a static, aging, and opaque codebase. The certificate tells you that a particular version met the requirements when it was evaluated. It does not tell you that the same code remains secure years later. At some point, the certificate becomes evidence not only of what was validated, but of how long the underlying system has stood still.
Your 78% number says it all, but it’s not because the vendors are lazy.
Vendors are leaving in old code because the rules punish them for fixing it. Touch the code and you’re back in line for a new validation, which costs a lot of money and a year or more of waiting. Leaving the bug alone is the cheap move.
We’re also testing the wrong thing. A certificate answers “did this build pass on the day we checked?” What matters is “can this ship a fix to a bug in weeks?” Our testing processes are broken.
NIST already has ACVP to do continuous and automated testing of the crypto algorithm implementations, they could continuously test that after each bugfix and publish how stale a certificate is. Right now, “validated in 2019 and never touched” looks identical to “validated in 2019 and patched eleven times” on their lists. Unfortunately, the way the standards are written, the rest of the testing is still very much a slow and expensive documentation exercise.
Agreed. This is exactly why I wanted to put this dataset together so we can have these kinds of conversations 🙂