Monthly Archives: October 2014

How did I get involved in PKI?

In the mid 90s I was a security consultant, I principally worked on authentication systems (Smart cards, One Time Passwords, Kerberos, PKI, etc.).

Back then the only people who cared about these things were organizations concerned with protecting lives or money. This meant most of our contracts were with governments, banks, and fortune 50s. This was an amazing experience that I would not trade for the world — it gave me the chance to work with some amazing people in some of the most paranoid and security conscious environments in the world.

While not my first exposure to PKI the first time “it was all I did” was when I worked for a company called ValiCert. The founders saw a problem:

Who was watching the certificate authorities and who would make sure that the revocation infrastructure would scale to meaningfully work in the event miss-issuances or key compromises happened?

We had developed technologies that were intended to address these problems. This technology looked very similar to Certificate Transparency, OCSP stapling and certificate pinning which are again all-the-rage these days.

Unfortunately the Certificate Authorities did not like the the idea of being “watched” by a third-party; the largest CA went so far to threaten with lawsuits and modified their Relying Party Agreements to state that third parties could not re-distribute any information about what certificates they had revoked or issued.

Another entity had patents they claimed covered some of our optimizations and given the browsers were minimally investing in this area we did not get adequate traction so we pivoted into other areas.

For personal reasons I ultimately ended up at Microsoft where I was responsible for a number of security technologies and one of the “little things” I ran was the Microsoft Root Program.

When this was assigned to me I was told it was the least important thing on my plate and that I could measure my success through the number of escalations we got relating to it — basically I was told to invest as little as possible to keep things quiet. The root program was a necessity but shipping software was what we were all about.

The first thing I did for the root program was review its requirements and try to understand who were its participants and what agreements we had with them. I was surprised to see there were in-essence no requirements, no authoritative list of contacts at each of the organizations and no contracts with any of its members. I felt marginally better when I found that Netscape had only one requirement and that was your check for $250,000 USD cleared, the upside of which also meant they probably had contracts with each CA but there were no technical or audit requirements in their program either.

To remedy I began to work with my AWSOME paralegal and lawyer on defining the first “root program” with both technical and audit requirements. We did not want to approach this as a profit center like Netscape but instead establish a set of requirements that were technically sound that encouraged CAs to spend on improving their infrastructure and having it reviewed by others

To this end I picked up a project that had been begun by my predecessor to work with the American Institute of Public Accountants (AICPA) to help define and adopt what is WebTrust for CAs today.

We were the first root program to adopt this new audit. I remember being interviewed by the AICPA for a video on their website on how excellent it was to work with them – they must have taken 50 cuts during that session because of my bumbling.

With these new requirements in hand we set out to get contractual agreements with each of the CAs where they would commit to meet these new requirements and make clear conditions on which we could kick them out for not complying. Given this required them to make operational changes to their practices as well as budget and manage a third-party audit it took a complete product release cycle to get all of this in place.

At the end of the operating system release we had an audited set of CAs and contractual agreements with each one of them. Now our goal was to get these CAs into one room so we could encourage them to adopt common issuance practices.

This was important for a number of reasons, one of the most obvious was that each one of the CAs used a different taxonomy to describe what they did. The simplest example of this was that one CAs in-person verified certificate would be called a Class 1 and another’s was a Class 3.

To top things almost all of the CAs wanted to see the browser “chrome” differentiate between their weakly authenticated certificates and those that were strongly authenticated. This of course was not possible without a common practices  and means of marking certificates to make it clear what practices were used in the vetting of the subscriber.

The internal consensus was that there would be value to users to be able to tell the difference so we decided to try to make this happen. To do that we arranged to get these CAs in one room so we could talk about standardizing practices and certificate formats.  To make this happen I reached out to my contact at the AICPA and asked him to work with me to arrange what was the very first gathering of publicly trusted CAs and trust store providers. We met in Washington DC because I felt we could leverage the work done by the US Government to accelerate the standardization of these things.

Unfortunately one of the newest CAs who only issued low assurance certificates saw adopting common standards for vetting and labeling a risk to their business and as a result they through a wrench in the my plan. They filed a claim with the FTC that what the event an attempt to create anti-competitive marketplace and as a result I was deposed by the DOJ. Ultimately the issue was closed and I understand the disposition was that the claim was baseless.

At this point I was instructed by management and our legal council to stop pushing for this standardization as it represented too much legal risk for the company.

As an aside a few months later the largest CA acquired the troublemaker.

About a year and a half later the CAs self-organized and attempted to agree on a smaller set of standardization, the definition of what is called Extended Validation today. This was effectively a new label for what most CAs were offering in their “high assurance” certificates. The CABFORUM was now born.

At this point I had moved onto another team at Microsoft. During my time at Microsoft I worked on a number of very cool projects with some great people. Several of the projects I worked on used PKI but my involvement was much more on the peripheral to the industry at that point.

Years later I decided to leave Microsoft — the Diginotar incident was a big contributor to this decision. I felt that the industry was a mess, they were under investing in their infrastructure, not supporting the open source community they were dependent on and not actively working to improve adoption of SSL. I wanted to change this, I had decided I would start my own Certificate Authority and set an example for the industry on how a CA should approach these things.

This is when GlobalSign approached me and asked me to join as their CTO, I really liked the team, they were principled, hard working and looking to change the way things were done. I spent nearly three years in this role and we accomplished a great deal, I also still work with them on technical research / direction  but I have since moved onto a startup doing work on Bitcoin related technologies.

I did not accomplish all of the things I wanted to but I still have hopes that these systemic issues will be resolved as I do believe trusted-third parties are needed on the internet.

Anyway this is how I got into PKI.


T-Mobile : How very “carrier” of you.

“Don’t let your mouth write checks you’re a** can’t cash” — that captures my experience with T-Mobile thus far.

Ever since I saw John Legere announce the T-Mobile “un-carrier” campaign I have been anxiously watching T-Mobile with the hope they will instigate positive changes in the mobile telecom space.

AT&T on the other hand has proven to me over the last two decades as a customer they have agility and customer service of ol’ Ma Bell. The silly games they play, even with their most lucrative customers, are abhorrent, and the pricing strategies they apply are nothing less than usurious.

That’s why when I saw the latest round of the “un carrier” campaign I decided to switch; conveniently this announcement was aligned with the release of the iPhone 6 which I wanted to get anyways.

I, like many got up early to place my pre-order online, setting my alarm to go off right when the pre-orders began, I tried for an hour and a half to place an order but the T-Mobile site kept timing out.

Writing this experience off to poor capacity planning, I went to bed and woke up a few hours later to try again – things were no better.

Over the next several days I continued to attempt to place an order getting to various points in the order workflow before the site would time-out and I would have to start over again.

I managed twice to sign the IUP with them via DocuSign, the last time actually completing the order. This was literally the fourth day and who knows how many attempts later.

I knew I would not be in the first rounds of the iPhone deliveries but I was not in any big hurry, I was just relieved my order was placed and soon I would no longer be under the thumb of the AT&T. To top things off I was going to save money each month!

Two days later my excitement was crushed as I received an email from T-Mobile instructing me to call them as there was a problem with my order. The next day I found the time to call them back (after holding for about 30 minutes) and was told the mail was sent in error and my order was fine — in-fact I could expect my phones within the week!.

A week passes and I get a phone call from T-Mobile, apparently there is in-fact a problem with my order and I need to cancel my order and place a new one. The woman I spoke to quickly cancels my existing order and begins to place a new one, after about an hour of problems with the ordering system she informs me she will not be able to place my order and will transfer me to someone else, I am told two of several of phones are in stock and that once my order is placed they will be held until the remaining ones are in stock, I am told more phones are a week out and that this new person will flag my order for overnight shipping once it is placed.

The new person also struggles with their ordering system, it takes him about 45 minutes to place the order, he does not put over night shipping on the order and informs me my order will be fulfilled in 3 weeks.

At this point I am not thrilled but as long as I get the phones before my sons birthday in the end of October I will be satisfied. After I sign the IUP for this new order several days pass with no confirmation of my order so I call to try to verify the status of my order, apparently they can see my order but can not give me any status.

The next day I get an email with an order number and a link where I can check my order, I begin checking this page almost every day.

On the first day I see in-fact two phones are in and they are waiting for the last one; the page indicates these two phones will be held until the remaining phone is in. Several days later those two phones have apparently been re-allocated since all are now marked as expected on or before the end of the next week.

A day or two later I see one phone is in and the next none.

This process continues for a few days, shipping dates moving in and out occasionally the order having one or more phones in stock to having none.

Nearly a month has passed, the current date of delivery is now the day after my sons birthday. My sons birthday arrives and now the final date moves into November, again a single phone is showing in stock and the order is being held until the rest arrive. Again the next day they all show as being expected in November.

To be clear this is not a post from a customer complaining about not having the iPhones they ordered, this is a customer complaining about how disorganized, under prepared and apparently under invested T-Mobile is in their internal software systems let alone their network which everyone knows is not as good as AT&Ts.

In their defense they are the cheaper solution, in my case I would save $20 / mo or $240 / yr by switching to them but is it worth it? As a general rule the absolute best experience you have with a service provider is the one you have before you’re a customer and this has been a miserable experience.

Realistically based on the way they are handling fulfillment it also seems the only way I will get these phones is if I go into a T-Mobile or Apple store and get them myself as their fulfillment system as implemented will keep me at the end of the line since I have several phones on order.

As for the un-carrier campaign; I think John Legere gets where the future is at and the direction he is pushing his company is the right one but unless this is paired with significant improvements in the technology the company is based on its not much more than a marketing message.

[UPDATE 11/7/14] Two months later I have the phones I ordered but they sent the wrong colors, very cool T-Mobile.

[UPDATE 11/8/14] Even though they sent the wrong phones I decided to go on with the switch. Unfortunately the box containing the phones did not contain any instructions on how to get started so I had to drive to a T-mobile store for their help. When I arrived they looked up my order and apparently there was no lines associated with it, additionally it was not associated with my equipment order. My plan has been to take advantage of the T-mobile offer pay termination fees but the agent now says that since the order was not setup with lines we could not take advantage of their promotion. He said if I want to take advantage of it I will have to call T-mobile, get and RMA and place a new order making sure that they do it right next time. That’s right I need to make sure they do their job right next time.

[UPDATE 11/8/14] Called T-Mobile, Got RMA number, will just return these phones and stay with AT&T. They may be evil but at-least they are semi-competent.

[UPDATE 11/10/14] I just went and purchased iPhones at the Apple Store and registered them with AT&T. I pay a little more each month than I would have with T-Mobile but at least they are semi-competent.

[UPDATE 11/25/14] Today I got an e-mail from T-Mobile confirming they received my returns, that is they claim to have only received 2 of 3 of the phones we sent back; There were 3 phones in the box when it was sealed up (in the original packing using their shipping label) they have lost this third phone and are apparently intent on me paying for it.

[UPDATE 12/04/14] On the 25th I wrote T-Mobile asking for them to refund the cost of the third-phone and I have still heard nothing, they owe me $171.23 and more importantly their mess up will surely come back to them claiming I owe them for service or even worse the full unsubsidized price of the phone they lost. I have just emailed them once more and will try the twitter account.

[UPDATE 12/06/14] I finally got ahold of someone, they claim they have now refunded me the amount they owe to a Visa debt card and that I will see it soon. Let’s see if they can manage to do this right.

[UPDATE 12/17/14] I just got another email from t-mobile asking me to “verify my account” over email or twitter along with a comment sayting the partial refund is expected for up to 30 days and implying the assurances the prior person gave me this has been resolved are not correct. Sigh. Now to start this sillyness all over again.