A quick look at SSL performance

2 Replies

When people think about SSL performance they are normally concerned with the performance impact on the server, specifically they talk about the computational and memory costs of negotiating the SSL session and maintaining the encrypted link.  Today though it’s rare for a web server to be CPU or memory bound so this really shouldn’t be a concern, with that said you should still be concerned with SSL performance.

Did you know that at Google SSL accounts for less than 1% of the CPU load, less than 10KB of memory per connection and less than 2% of network overhead?

Why? Because studies have shown that the slower your site is the less people want to use it. I know it’s a little strange that they needed to do studies to figure that out but the upside is we now have some hard figures we can use to put this problem in perspective. One such study was done by Amazon in 2008, in this study they found that every 100ms of latency cost them 1% in sales.

That should be enough to get anyone to pay attention so let’s see what we can do to better understand what can slow SSL down.

Before we go much further on this topic we have to start with what happens when a user visits a page, the process looks something like this:

  1. Lookup the web servers IP address with DNS
  2. Create a TCP socket to the web server
  3. Initiate the SSL session
  4. Validate the certificates provided by the server
  5. Establish the SSL session
  6. Send the request content

What’s important to understand is that to a great extent the steps described above tasks happen serially, one right after another – so if they are not optimized they result in a delay to first render.

To make things worse this set of tasks can happen literally dozens if not a hundred times for a given web page, just imagine that processes being repeated for every resource (images, JavaScript, etc.) listed in the initial document.

Web developers have made an art out of optimizing content so that it can be served quickly but often forget about impact of the above, there are lots of things that can be done to reduce the time users wait to get to your content and I want to spend a few minutes discussing them here.

First (and often forgotten) is that you are dependent on the infrastructure of your CA partner, as such you can make your DNS as fast as possible but your still dependent on theirs, you can minify your web content but the browser still needs to validate the certificate you use with the CA you get your certificate from.

These taxes can be quite significant and add up to 1000ms or more.

Second a mis(or lazily)-configured web server is going to result in a slower user experience, there are lots of options that can be configured in TLS that will have a material impact on TLS performance. These can range from the simple certificate related to more advanced SSL options and configuration tweaks.

Finally simple networking concepts and configuration can have a big impact on your SSL performance, from the basic like using a CDN to get the SSL session to terminate as close as possible to the user of your site to the more advanced like tuning TLS record sizes to be more optimum.

Over the next week or so I will be writing posts on each of these topics but in the meantime here are some good resources available to you to learn about some of these problem areas:

Reading ocspreport and crlreport at x509labs.com

2 Replies

As you may know I have been hosting some performance and up-time monitors at: http://ocspreport.x509labs.com and http://crlreport.x509labs.com.

I started this project about six months ago when I walked the CAB Forum membership list, visited the sites of the larger CAs on that list, looked at their certificates and extracted both OCSP and CRL urls and added them into custom monitor running on AWS nodes.

Later I tried Pingdom and finally settled on using Monitis because Pingdom doesn’t let you control which monitoring points are used and doesn’t give you the ability to do comparison views. That said as a product I liked Pingdom much better.

As for how I configured Monitis, I did not do much — I set the Service Level Agreement (SLA) for uptime to 10 seconds which is the time required to be met by the CABFORUM for revocation responses. I also selected all of the monitoring locations (30 of them) and set it loose.

I put this up for my own purposes, so I could work on improving our own service but I have also shared it publicly and know several of the other CAs that are being monitored are also using it which I am happy to see.

OK, so today I found myself explaining a few things about these reports to someone so I thought it would be worthwhile to summarize those points for others, they are:

  1. Why is it so slow to render? – Unfortunately despite numerous requests to Monitis there is nothing I can do about this – Monitis is just slow.
  2. Why does it show downtime so often? – I do not believe the downtime figures, most of the time the failures show up on all of the urls. The times I have looked into theses it turned out the failures were at Monitis or due to regional network congestion / failures. Unfortunately this means we cannot rely on these figures for up-time assessment, at best they are indicators when looked at over long periods of time.
  3. Why do some tests show at 0-1 ms? – This is likely because the Monitis testing servers are located in the same data center as the OCSP servers in question. This skews the performance numbers a little bit but the inclusion of many perspectives should off-set this.

At this point I suspect you’re wondering, with these shortcomings what is this thing good for anyways? That’s a good question; OCSP (and CRLs) are a hidden tax that you and your users pay when they visit your site.

This is important because studies have found a direct correlation between latency and user abandonment and seriously who doesn’t just want their site to be fast as possible.

My hope is these resources help you understand what that tax is; if you’re a CA operator it can also help you tweak your performance as well as get an idea of what the global user experience is for the relying parties of your certificates.

On a related note I do think someone could make a pretty penny if they made an easy to use, yet powerful monitoring site 🙂

RESTful X509, CRL and OCSP to JSON web-service

Leave a reply

So the other day I got a bee in my bonnet and decided I wanted a simple web service I could pass common day X509 objects to and get a JSON representation of that same object. We had recently done a project in Go at work and we found it quick, robust and easy to build, additionally it looks it’s certificate support decent enough so I thought it was the way to go.

In comes Freelancer, I threw my rough (and that’s kind) goals in a paragraph or two and a few days later I had a bid proposal from an engineer in Chicago — Eli Frey.

Based on a quick review of the Go documentation for cryptography it looked like this was going to be pretty straight forward, and for the most part it was – we did find that there were a few cases that just were not possible without more work than we wanted to put in, I will summarize those a little later.

As things progressed we also decided to add the ability to get an X509 certificate from the interface. Normally one would do this by generating a PKCS #10 request (CSR) and sending it to a CA for processing, unfortunately one of those cases that required more work than we wanted to put in was parsing PKCS #10s since go does not as of yet support it. With that said a CSR is really just a self-signed certificate we just did the same thing with a self-signed X509 certificate request.

So how do these interfaces work? Here are a few examples of how you would call them:

 

Decode a PEM encoded X509 certificate
curl  -F “[email protected]” “api.x509labs.com/v1/x509/certificate?action=decode&inputEncoding=PEM”
 
Decode a DER encoded X509 certificate
curl –fail -F “[email protected]” “api.x509labs.com/v1/x509/certificate?action=decode&inputEncoding=DER”
 
Request and issue an X509 certificate based on a DER encoded self-signed certificate with one hostname
openssl genrsa -out request.key 2048
openssl req -config openssl.cfg -subj “/CN=www.example.com” -new -x509 -set_serial 01 -days 1 -key request.key -out request.cer
curl –fail -F “[email protected]” “api.x509labs.com/v1/x509/certificate?action=issue&hostnames=bob.com&inputEncoding=DER”
 
Request and issue an X509 certificate based on a PEM encoded self-signed certificate with one hostname
openssl genrsa -out request.key 2048
openssl req -config openssl.cfg -subj “/CN=www.example.com” -new -x509 -set_serial 01 -days 1 -key request.key -out request.cer
curl –fail -F “[email protected]” “api.x509labs.com/v1/x509/certificate?action=issue&hostnames=bob.com&inputEncoding=PEM”
 
Request and issue an X509 certificate based on a PEM encoded self-signed certificate with several hostnames
openssl genrsa -out request.key 2048
openssl req -config openssl.cfg -subj “/CN=www.example.com” -new -x509 -set_serial 01 -days 1 -key request.key -out request.cer
curl –fail -F “[email protected]” “api.x509labs.com/v1/x509/certificate?action=issue&hostnames=bob.com,fred.com&inputEncoding=PEM”
 
Decode a set of PEM encoded X509 certificates
curl –fail -F “[email protected]” “api.x509labs.com/v1/x509/certificates?action=decode&inputEncoding=PEM”
 
Decode a PEM encoded X509 crl
curl –fail -F “[email protected]” “api.x509labs.com/v1/x509/crl?action=decode&inputEncoding=PEM”
 
Decode a DER encoded X509 crl
curl –fail -F “[email protected]” “api.x509labs.com/v1/x509/crl?action=decode&inputEncoding=DER”
 
Decode an OCSP response
openssl ocsp -noverify -no_nonce -respout ocsp.resp -reqout ocsp.req -issuer ca.cer -cert www.cer -url “http://ocsp2.globalsign.com/gsextendvalg2” -header “HOST” “ocsp2.globalsign.com” -text
curl –fail -F “[email protected]” “api.x509labs.com/v1/x509/ocsp?action=decode&type=response”

 

 

So even though this started out as a pet project I actually think these interfaces are pretty useful, the largest limitations of these interfaces are:

X509Certificate

  1. Not every element of the structures is included in the JSON serialization, for example AIA, CDP, Name Constraints and Certificate Policy are not present (most extensions actually); this is because there is not a decoder for them in GO.
  2. ECC based certificates are not supported, this is because at this time the released version of GO doesn’t include support for these.
  3. Only issuing certificates based on self-signed X509 certificates are supported, this is as I mentioned a result of the lack of support for the PKCS #10 object in GO.
  4. No OID is specified for the Signature algorithm, this is because it’s not exposed in GO.
  5. Only one certificate profile is supported when using the issue action, this is mostly due to limitations in go (time was also a factor) for example the lack of AIA and OCSP support mean these regardless of CA key material these certs are just good for playing around.
  6. No user supplied information is included in the generated certificate, this was really just a function of time and building a proper workflow that would not be valuable without addressing other go limitations.
  7. Requested certificates that contain RSA keys must have a bit length of at least 2048 bits in length, just a best practice.
  8. Requested certificates will only be issued if the submitted certificate contains a self-signed certificate with a valid signature, this is to ensure the requestor actually holds the private key.
  9. Not all SAN types are supported, only DNSnames really again a limitation of GO.
  10. Certificates with name constraints are not supported, again a limitation of GO.
  11. Not possible to put EKU in certificates, again a limitation of GO.

 

X509OCSP

  1. ResponderID is not specified, this is because it’s not exposed in GO.
  2. Only responses with a single response are supported, this is because more that response is not exposed in GO.
  3. No OCSP extensions are supported, this is because this is not exposed in GO.
  4. Only responses are supported, this is because the request is not supported in GO.

 

Here are some things you might want to know about these interfaces:

  1. Both X509crl and X509ocsp default to DER but you can specify PEM in the encode query string parameter.
  2. X509Certificate defaults to the PEM encoding but DER is supported via the encode query string parameter.
  3. X509Certificates defaults to PEM encoding but DER is not supported.
  4. X509Certificates takes the file you might use in Apache or Nginx to configure which certificates to send — a concatenation of PEM encoded certificates.
  5. All interfaces use HTTP error codes to report issues.
  6. I can’t propose they will always be up and available, be reliable, performant or accurate 🙂

All in-all I think this was a fun project and I really enjoyed working with Eli and Freelancer (though its mail client is awful and the site needs some UI work).

Ryan

A look at revoked certificates

Leave a reply

So today I have done posts on the browser user experience for expired and untrusted certificates but we wouldn’t have proper coverage on the topic of bad certificate user experience if we did not cover revoked certificates.

VeriSign is kind enough to host a test site that uses a revoked certificate (I know we do too I just can’t find it right now) so we will use that (https://test-sspev.verisign.com:2443/test-SSPEV-revoked-verisign.html)

Again what we want to see here is:

  1. Users are warned or prohibited from going to the site in question.
  2. The warning language used is easy to understand and explains the risks.
  3. The warning language used is related to the fact that the certificate is expired.
  4. The trust indicator does not show or is marked to indicate that there is a problem.

In this case I think again Internet Explorer and Chrome do the best; The worse experience is in Opera as it leads the user to believe there is a connectivity problem unless they expand the error message.

Chrome

Internet Explorer

Mozilla

Opera

Safari

A look at untrusted certificates

1 Reply

Today I did a blog post on how browsers show expired certificates. I figured I would take the opportunity to capture a few of the other failure cases for certificates.

 

The most severe example is that of an untrusted root certificate, for this scenario I figured the use of https://cacert.org was the most direct example.

 

There are a few cases where this error condition will come up, for example another one is if a server doesn’t include all of the intermediate certificates the clients cannot determine which Certificate Authority issued the certificate.

According to the current SSL Pulse data about 7.4% of the servers in the Alexa top one million may fall into this case.

 

Chrome

Internet Explorer

Mozilla

Opera

 

Safari

A look at expired certificates

3 Replies

Today I was on a mail thread where the topic of how browsers handle expired certificates; this is particularly relevant for a few reasons.

The first of which is that there is a large number of sites operating with expired certificates out on the Internet today, the other is that the adoption of short lived certificates (which I am a fan of) is at least in part dependent on how browsers deal with certificates that are expired.

In any event I was not sure how the most recent versions of browsers were handling these cases so I dug up an example site where an expired certificate was in use (https://www.appliancetherapy.com – it uses a certificate that expired a few weeks ago and has not as of yet been replaced).

So what did I want to find? In a perfect world I believe that the following should be true:

  1. Users are warned or prohibited from going to the site in question.
  2. The warning language used is easy to understand and explains the risks.
  3. The warning language used is related to the fact that the certificate is expired.
  4. The trust indicator does not show or is marked to indicate that there is a problem.

The good news is that for the most part browsers behaved fairly close to this, they all could have improved language but I believe Internet Explorers was the best.

The worst behaving client was Mozilla, as it doesn’t report the certificate as expired but instead indicates that it tried to make an OCSP request but got a response it was not expecting. This has two problems – the first of which being it should not have made an OCSP request for the status of an expired request.

RFC 5280 Section 5 states that:

 

   A complete CRL lists all unexpired certificates, within its scope,

   that have been revoked for one of the revocation reasons covered by

   the CRL scope.  A full and complete CRL lists all unexpired

   certificates issued by a CA that have been revoked for any reason.

 

And RFC 2560 is written largely based on OCSP responses being fed from CRLs. What this means is that it is not appropriate to ask the revocation status of a certificate that is expired.

The next problem is that Mozilla also doesn’t handle the unauthorized response in a usable way. RFC 5019 Section 2.2.3 states:

 

   The response “unauthorized” is returned in cases where the client

   is not authorized to make this query to this server or the server

   is not capable of responding authoritatively.

 

A user who receives this message would believe the issue is related to their permissions but based on the true reason for the error the failure as really that the responder in question doesn’t have the information that’s needed.

This lack of information on the server is likely due to the fact that it isn’t required to maintain information for expired certificates and the message Mozilla delivered should have been about the certificate being expired.

In any event the browsers behaved much better than I expected, IE and Chrome did the best (I really like Chromes red / over the https as a visual queue there is a problem).

 

 

Chrome

Internet Explorer

Mozilla

Opera

Safari

 

How to tell DV and OV certificates apart

Leave a reply

Introduction

There are in essence three kinds of SSL certificates: Domain Validated, Organization Validated and Extended Validated. I am not going to write about the differences here it seems that there are hundreds of articles on this topic on the Internet.

What I think has not been given sufficient coverage is how one is able to look at a certificate and determine what type it is.

One would think that this would be easy; In theory if nothing was explicitly stated it would be a Domain Validated certificate (since it is the weakest validation), otherwise someone would put something in the certificate making it clear that the certificate was either Organization Validated or Extended Validated.

Unfortunately it’s not this simple, the main issue being the historic lack of coordination within the CA industry.

Each Certificate Authority (CA) has its own unique practices relating to how they mark their certificates so with the existing deployed certificates there is no singular rule or approach can be used to definitively know what level of validation was done for a given certificate.

Thankfully it looks like that this problem is betting better thanks to the adoption of the Baseline Requirements but in the meantime we have to make do with heuristics.

Deterministic Approach

Today the only way to know with confidence that a certificate is of a specific type is to know the practices of each CA.

In X.509 the way an issuer is supposed to express something like this is via the Certificate Policies extension which is defined in RFC 5280.

This allows a CA to express a unique identifier (an OID) in their certificates that maps to a document that describes its practices associated with this certificate. This identifier can be used programmatically to do make trust decisions about a certificate or to differentiate the user interface in an application based on what type of certificate is being used.

This is exactly how browsers today can tell if a certificate is an Extended Validation (EV) certificate. In essence they have some configuration that says “I trust GlobalSign to issue EV certificates, when a certificate is presented to me from them that has this policy OID show the EV user experience”.

The Baseline Requirements use the same approach defining identifiers for Domain Validated and Organization Validated certificates, these are:

Type Policy Identifier
Domain Validated 2.23.140.1.2.1
Organization Validated 2.23.140.1.2.2

 

Having these identifiers takes us a long way towards our goal of deterministic evaluation of certificate issuance policy — that said not all CAs have adopted them which is technically alright since the Baseline Requirements do allow them to use their own Policy Identifiers.

Heuristic Approach

Since the Baseline Requirements were only established this year it will take some time for the existing install base of certificates to be re-issued to use these Policy Identifiers called about above. This doesn’t mean you can’t tell the certificates apart today, it does mean it is quite a bit messier though.

Here is some pseudo-code provided to me as an example from a friend that they used in one of their projects:

type = null;

if (cert is self-signed) then

     type = SS;        /* SS = Self-signed */

else if (cert was issued by a known “CA”) then

     type = DV;  /* DV = Domain Validation */ else if (cert contains a known EV Policy OID) then

     type = EV;  /* EV = Extended Validation */ else if (cert “Subject O” and “Subject CN” are the same or “Subject OU” contains “Domain Control Validated”) then {

     if (cert contains no Subject L, St or PostalCode) then

         type = DV;

}

else if (cert “Subject O” is “Persona Not Validated” and the cert’s issuer was StartCom

     type = DV;

if (type is null)

     type = OV;

This logic is not comprehensive but should work well enough for most uses.

Summary

Unfortunately today there is not a deterministic way to tell if a certificate was Domain or Organization Validated, that said things are changing and within a few years hopefully it will be possible.

In the mean-time there are heuristics you can use that help tell these types of certificates apart.

Windows XP and Name Constraints

7 Replies

Recently I blogged about how Windows XP processes Name Constraints a little different than the RFC specifies — with the help of a friend I have a good set of examples of what would work and what would not work that illustrate what it does.

Assuming our Subject was:

C = US;S = Washington;L = Kirkland;O = GlobalSign;CN = globalsign.com

 

And that our Constraint was:

Permitted

     [1]Subtrees (0..Max):

          DNS Name=globalsign.com

We would see different results when validating a certificate on XP than we would on a later version of Windows.

Notice we did not include any directoryName attributes? That is supposed to mean that there is no constraints on the directoryName. On Windows XP however if you include a directory name in the subject there MUST be at least one Directory Name attribute in the RDN to match against otherwise it will not pass its Name Constraints check.

So if we instead made our constraint:

Permitted

     [1]Subtrees (0..Max):

          RFC822 Name=globalsign.com

     [3]Subtrees (0..Max):

          Directory Address:

          C = US

          S = Washington

          L = Kirkland

          O = Globalsign

Excluded=None

 

A certificate with the following subject would match:

  • An empty DN, no RDNs
  • C = US
  • C = US;S = Washington
  • C = US;S = Washington;L = Kirkland
  • C = US;S = Washington;L = Kirkland;O = Globalsign
  • C = US;S = Washington;L = Kirkland;O = Globalsign;CN = globalsign.com
  • C = US;S =””;L = Kirkland;O =””;CN = globalsign.com

When XP processes the RDNs it starts with the first and progresses from there. You can’t skip an RDN. If an RDN is present it must match the entire RDN value or be empty.

As such the following wouldn’t match in our example:

  • S = Washington (Skipped the first RDN)
  • C = US;L = Kirkland (Skipped the second RDN)
  •  C = US;S = Washington;L = Kirkland;O = Globalsign Development Center (partial “O” value).

The prior blog post on this topic I described before talks about how an enterprise can work around this behavior (by setting some registry keys) but a public certificate issuer can too, for example by inserting just one RDN value and ensuring the subordinate CA issues with that RDN value in its certificates.

This way a site can have the flexibility it wants to change its directory structure without re-issuing the certificate containing the Name Constraints.

Ryan

Algorithms, key size and digital certificates

Leave a reply

Introduction

On the surface the digital certificates are not complicated — a third-party (a certificate authority) verifies some evidence and produces a piece of identification that can be presented at a later date to prove that the verification has taken place.

As is usually the case when we look a little deeper things are not that simple. In the this case we have to care about a few other things, for example what are the qualifications of the third-party, what are their practices and what cryptographic algorithms did they use to produce the digital certificate?

As an administrator using digital certificates like in the case of SSL these things also can have impact on your operational environment – by using a certificate from a certificate authority you take dependencies on their practices and operational environment.

This is especially true when it comes to decisions relating to what cryptographic algorithms and key lengths are accepted and used by that third-party.

Thankfully you do not need to be a cryptographer to make good decisions on this topic, first we need to start with an understanding of the history, future and then considerations.

History

In recent history the industry has relied on two algorithms, the first being an encryption algorithm called RSA the second being a hash algorithm called SHA-1. Both of which have are considered weaker now due to advances in cryptanalysis.

RSA’s strength and performance is based on the size of the key used with it, the larger the key the stronger and slower it is.

These advances in cryptanalysis have driven the increase in key size used with this algorithm which in turn has increased the amount of computing power necessary to maintain the same effective strength.

The problem with this is that that every time we double the size of an RSA key the decryption operations with that key become 6-7 times slower.

As a result as of all of this as of January 2011 trustworthy Certificate Authorities have aimed to comply with NIST (National Institute of Standards and Technology) recommendations by ensuring certificates all new RSA certificates have keys of 2048 bits in length or longer.

Unfortunately this ever increasing key size game cannot continue forever, especially if we ever intend do see SSL make up the majority of traffic on the internet – the computational costs are simply too great.

That takes us to SHA-1, hash algorithms take a variable amount of input and reduce it to a typically shorter and fixed length output the goal of which being to provide a unique identifier for that input. The important thing to understand is that hash algorithms are always susceptible to collisions and the advances in the cryptanalysis have made it more likely that such a collision can be made.

The problem here is that there is no parameter to tweak that makes this problem harder for an attacker, the only way to address this issue is to change to a stronger algorithm to produce the hash.

Future

For the last decade or so there has been slow and steady movement towards using two new algorithms to address these advances — SHA-2 and ECC.

ECC has the potential for significant performance benefits over RSA without reducing security and SHA-2 has three versions each with progressively longer lengths which help it both address the current risks and give it some longevity.

Considerations

Our goal in configuring SSL is enabling users to communicate with us securely; to accomplish this goal we need to be able to do this with the fewest hassles, lowest costs and comply with any associated standards.

Interoperability is the key that ensures the fewest hassles — if it was not for this we would simply switch to these new algorithms and be done with it. As is normally the case when it comes to security this is where Windows XP rears its ugly head, SHA-2 was added to XP in Windows XP Service Pack 2 and ECC in Windows Vista.

These facts set the adoption clock for these new algorithms; if you care about XP (about 30% of the Internet today) you can’t adopt ECC and SHA-2 in full for about 5 years.

This leaves us with RSA 2048 and SHA-1 which thankfully is broadly considered sufficient for the next decade.

Performance is always a concern as well — a RSA 2048-bit RSA certificate used in SSL will result in around a 10% CPU overhead not huge but something to keep in mind.

As mentioned previously we can’t forget compliance — whether it is the Payment Card Industry / Data Security Standards (PCI/DSS), Federal Information Processing Standards (FIPS) 140-2 or some other set of criteria you need to meet this always needs to be considered.

Conclusion

The decision of what algorithm’s and key lengths to use in your digital certificates is dependent on a number of factors including security, interoperability, performance and compliance. Each situation may require a different trade-off to be made however a rule of thumb if you stick with SHA-2 and RSA 2048-bit certificates and keys you should be fine for now.

 

Resources

[1]   BlueKrypt Cryptographic Key Length Recommendations

[2]   Recommendation for Key Management, Special Publication 800-57 Part 1 Rev. 3, NIST, 05/2011

[3]   Fact Sheet Suite B Cryptography, NSA, 11/2010

[4]   Worldwide Operating System Statistics, Stat Counter, 9/2012

[5]   RSA Algorithm, Wikipedia

[6]   RSA Key Lengths, Javamex

[7]   ECC Algorithm, Wikipedia

[8]   Performance Analysis of Elliptic Curve Cryptography for SSL, Sun

[9]   Using ECC keys in X509 certificates, UnmitigatedRisk

[10] Using SHA2 based signatures in X509 certificates, UnmitigatedRisk

[11]Payment Card Industry / Data Security Standards – PCI

[12]Federal Information Processing Standards 140-2 – NIST