In the mid 90s I was a security consultant, I principally worked on authentication systems (Smart cards, One Time Passwords, Kerberos, PKI, etc.).
Back then the only people who cared about these things were organizations concerned with protecting lives or money. This meant most of our contracts were with governments, banks, and fortune 50s. This was an amazing experience that I would not trade for the world — it gave me the chance to work with some amazing people in some of the most paranoid and security conscious environments in the world.
While not my first exposure to PKI the first time “it was all I did” was when I worked for a company called ValiCert. The founders saw a problem:
Who was watching the certificate authorities and who would make sure that the revocation infrastructure would scale to meaningfully work in the event miss-issuances or key compromises happened?
We had developed technologies that were intended to address these problems. This technology looked very similar to Certificate Transparency, OCSP stapling and certificate pinning which are again all-the-rage these days.
Unfortunately the Certificate Authorities did not like the the idea of being “watched” by a third-party; the largest CA went so far to threaten with lawsuits and modified their Relying Party Agreements to state that third parties could not re-distribute any information about what certificates they had revoked or issued.
Another entity had patents they claimed covered some of our optimizations and given the browsers were minimally investing in this area we did not get adequate traction so we pivoted into other areas.
For personal reasons I ultimately ended up at Microsoft where I was responsible for a number of security technologies and one of the “little things” I ran was the Microsoft Root Program.
When this was assigned to me I was told it was the least important thing on my plate and that I could measure my success through the number of escalations we got relating to it — basically I was told to invest as little as possible to keep things quiet. The root program was a necessity but shipping software was what we were all about.
The first thing I did for the root program was review its requirements and try to understand who were its participants and what agreements we had with them. I was surprised to see there were in-essence no requirements, no authoritative list of contacts at each of the organizations and no contracts with any of its members. I felt marginally better when I found that Netscape had only one requirement and that was your check for $250,000 USD cleared, the upside of which also meant they probably had contracts with each CA but there were no technical or audit requirements in their program either.
To remedy I began to work with my AWSOME paralegal and lawyer on defining the first “root program” with both technical and audit requirements. We did not want to approach this as a profit center like Netscape but instead establish a set of requirements that were technically sound that encouraged CAs to spend on improving their infrastructure and having it reviewed by others
To this end I picked up a project that had been begun by my predecessor to work with the American Institute of Public Accountants (AICPA) to help define and adopt what is WebTrust for CAs today.
We were the first root program to adopt this new audit. I remember being interviewed by the AICPA for a video on their website on how excellent it was to work with them – they must have taken 50 cuts during that session because of my bumbling.
With these new requirements in hand we set out to get contractual agreements with each of the CAs where they would commit to meet these new requirements and make clear conditions on which we could kick them out for not complying. Given this required them to make operational changes to their practices as well as budget and manage a third-party audit it took a complete product release cycle to get all of this in place.
At the end of the operating system release we had an audited set of CAs and contractual agreements with each one of them. Now our goal was to get these CAs into one room so we could encourage them to adopt common issuance practices.
This was important for a number of reasons, one of the most obvious was that each one of the CAs used a different taxonomy to describe what they did. The simplest example of this was that one CAs in-person verified certificate would be called a Class 1 and another’s was a Class 3.
To top things almost all of the CAs wanted to see the browser “chrome” differentiate between their weakly authenticated certificates and those that were strongly authenticated. This of course was not possible without a common practices and means of marking certificates to make it clear what practices were used in the vetting of the subscriber.
The internal consensus was that there would be value to users to be able to tell the difference so we decided to try to make this happen. To do that we arranged to get these CAs in one room so we could talk about standardizing practices and certificate formats. To make this happen I reached out to my contact at the AICPA and asked him to work with me to arrange what was the very first gathering of publicly trusted CAs and trust store providers. We met in Washington DC because I felt we could leverage the work done by the US Government to accelerate the standardization of these things.
Unfortunately one of the newest CAs who only issued low assurance certificates saw adopting common standards for vetting and labeling a risk to their business and as a result they through a wrench in the my plan. They filed a claim with the FTC that what the event an attempt to create anti-competitive marketplace and as a result I was deposed by the DOJ. Ultimately the issue was closed and I understand the disposition was that the claim was baseless.
At this point I was instructed by management and our legal council to stop pushing for this standardization as it represented too much legal risk for the company.
As an aside a few months later the largest CA acquired the troublemaker.
About a year and a half later the CAs self-organized and attempted to agree on a smaller set of standardization, the definition of what is called Extended Validation today. This was effectively a new label for what most CAs were offering in their “high assurance” certificates. The CABFORUM was now born.
At this point I had moved onto another team at Microsoft. During my time at Microsoft I worked on a number of very cool projects with some great people. Several of the projects I worked on used PKI but my involvement was much more on the peripheral to the industry at that point.
Years later I decided to leave Microsoft — the Diginotar incident was a big contributor to this decision. I felt that the industry was a mess, they were under investing in their infrastructure, not supporting the open source community they were dependent on and not actively working to improve adoption of SSL. I wanted to change this, I had decided I would start my own Certificate Authority and set an example for the industry on how a CA should approach these things.
This is when GlobalSign approached me and asked me to join as their CTO, I really liked the team, they were principled, hard working and looking to change the way things were done. I spent nearly three years in this role and we accomplished a great deal, I also still work with them on technical research / direction but I have since moved onto a startup doing work on Bitcoin related technologies.
I did not accomplish all of the things I wanted to but I still have hopes that these systemic issues will be resolved as I do believe trusted-third parties are needed on the internet.
Anyway this is how I got into PKI.
Ah, ValiCert, which reminds me of another story. IE3 shipped with several roots, but most of them were useless and was removed by IE 5.0, leaving only KeyWitness, Verisign, GTE CyberTrust, and Thawte. This was obviously a bad situation and MS added many CAs in IE 5.01/Win2000, including ValiCert (it was just after IE 5.01 released that VeriSign bought Thawte). Unfortunately, since then the list (in initpki.dll) has been frozen. Only XP had auto root update, and most other users had to manually install the root updates from Windows Update. Obviously this made the roots in the IE 5.01 list very valuable, and ValiCert was one of the roots that was sold (to SECOM, GoDaddy, RSA)..
Yuhong, I was the guy who created a few of those roots at ValiCert. I am a little embarrassed at how the cert profiles turned out, thankfully the root content isn’t the most relevant.
As for the IE root store my memory of events was it was a a never ending list, by the time I took over the root store there were a ton of CAs in it (I don’t recall a pruning until we shipped auto-root update). The problem with embedding roots is you could only practically update roots when the OS shipped and we needed a more regular cadence than could be addressed that way, this is where auto-root update came from. This of course was very problematic in Asia given the lack of patching and slow OS updates there but in countries that picked up new versions of the OS and patches the mechanism worked fairly well, we did keep publishing manual updates for these legacy clients though.
The core issue being how we implemented it did not at any given point give a clear idea of what was trusted in the MMC, that said almost no one used it and there was always clear documentation on what the OS trusted in the support site which by numbers was used more than the MMC by several multiples.
As for the sale of the ValiCert roots, I recall getting a call from GoDaddy asking me if a CA was available for sale; ValiCert was having a hard time and was merging into another firm at the time so I suggested they contact them and a few months later the roots were sold to them.
Its been an interesting ride.
I think back in early 2000s consumers did not commonly run Windows Update at all. For example, I think it took adoption of broadband and three worms (Blaster, Sasser, Zotob) before the Win2000 RTM userbase was reduced to almost zero. Which reminds me that MS had to ship an encryption floppy in the package even though they know that the US government was going to remove the key length restrictions on cryptography soon when Win2000 RTMed back in Dec 1999. When IE 5 for Mac was shipped a while later, they did the right thing and did not ship the “export” version at all.
Yes, servers almost never (even today) do auto-updates and patch rate of W2K desktops was even lower than XP for quite a long time.
Its amazing to me that today we can see 80% of Chrome user base on the newest version within just a few months. It is a different world for sure.
As a side note, I think that root cert updates was always considered optional updates, right?
Yes, before auto-root update no one ever automatically got root updates they were essentially optional.
I mean that if someone went to WIndows Update would they see the root updates as automatically checked important updates or unchecked by default optional updates.