Tag Archives: OCSP Stapling

Example Nginx SSL / TLS configuration

6 Replies

Configuring your server for SSL can be a little overwhelming. To help with this I am writing three posts (one for Nginx, Apache and IIS) with example configurations that (to the extent possible) result in the same configuration regardless of what server you are using.

Let’s start with Nginx, for this site :

  1. Running nginx/1.4.1 and openssl 1.0.1e
  2. All static content is handled by Nginx.
  3. All dynamic content is handled by Node.js and Express.
  4. We use the X-Frame-Options header to help protect from Click-Jacking.
  5. We use the X-Content-Security-Policy
    header to help protect from Cross-Site-Scripting.
  6. All requests for content received over http are redirected to https.
  7. Once the user visits the https version of the site the Strict-Transport-Security header instructs the browser to always start with the https site.
  8. We have chosen SSL cipher suites to offer a blend of performance and security.
  9. We have disabled SSL v2 and v3 and enabled all versions of TLS.
  10. We have enabled OCSP stapling.
  11. We have enabled SSL session caching.
  12. We have put all certificates and keys into their own folder (certs.d/).
  13. Set the owner of the of the certs.d folder to the process that the server runs as.
  14. We have restricted the certs.d folder and key files so only the owner can read and write (chmod 600).

Here is the configuration file:

server {
listen  80;
server_name  example.com;

# tell users to go to SSL version this time
if ($ssl_protocol = "") {
rewrite     ^   https://$server_name$request_uri? permanent;
}

}

server {
listen  443 ssl;
server_name  example.com;

# tell users to go to SSL version next time
add_header Strict-Transport-Security "max-age=15768000; includeSubdomains;";

# tell the browser dont allow hosting in a frame
add_header X-Frame-Options DENY;

# tell the browser we can only talk to self and google analytics.
add_header X-Content-Security-Policy "default-src 'self'; \
script-src 'self' https://ssl.google-analytics.com; \
img-src 'self' https://ssl.google-analytics.com";

ssl_protocols               TLSv1 TLSv1.1 TLSv1.2;

# ciphers chosen and ordered for mix of performance, interoperability and security
#ssl_ciphers                 AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH;

# ciphers chosen for security (drop RC4:HIGH if you are not worried about BEAST).
#ssl_ciphers                  RC4:HIGH:HIGH:!aNULL:!MD5;

# ciphers chosen for FIPS compliance.
#ssl_ciphers !aNULL:!eNULL:FIPS@STRENGTH;

# ciphers chosen for forward secrecy an compatibility
ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";


ssl_prefer_server_ciphers   on;
ssl_certificate_key         certs.d/example.key;
ssl_certificate             certs.d/example.cer;

ssl_session_cache    shared:SSL:10m;
ssl_session_timeout  10m;

# enable ocsp stapling
resolver 8.8.8.8;
ssl_stapling on;
ssl_trusted_certificate certs.d/example.cer;

# let nginx handle the static resources
location ~ ^/(htm/|html/|images/|img/|javascript/|js/|css/|stylesheets/|flash/|media/|static/|robots.txt|humans.txt|favicon.ico) {

root /usr/share/nginx/example/public;
access_log off;
expires @30m;
}

# redirect to node for the dynamic stuff
location / {
proxy_pass http://localhost:8003;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;

proxy_hide_header X-Powered-By;

#proxy_redirect off;
#proxy_set_header   X-Real-IP            $remote_addr;
#proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
#proxy_set_header   X-Forwarded-Proto $scheme;
#proxy_set_header   X-NginX-Proxy    true;
}

error_page  404              /404.html;

# redirect server error pages to the static page /50x.html
#
error_page   500 502 503 504  /50x.html;

location = /50x.html {
root   /usr/share/nginx/html;
}
}

How Facebook can avoid losing $100M in revenue when they switch to always-on SSL

2 Replies

Recently Facebook announced that they will be moving to Always-On-SSL, I for one am thrilled to see this happen – especially given how much personal data can be gleamed from observing a Facebook session.

When they announced this change they mentioned that users may experience a small performance tax due to the addition of SSL. This is unfortunately true, but when a server is well configured that tax should be minimal.

This performance tax is particularly interesting when you put it in the context of revenue, especially when you consider that Amazon found that every 100ms of latency cost them 1% of sales. What if the same holds true for Facebook? Their last quarter revenue was 1.23 billion, I wanted to take a few minutes and look at their SSL configuration to see what this tax might cost them.

First I started with WebPageTest; this is a great resource for the server administrator to see where time is spent when viewing a web page.

The way this site works is it downloads the content twice, using real instances of browsers, the first time should always be slower than the second since you get to take advantage of caching and session re-use.

The Details tab on this site gives us a break down of where the time is spent (for the first use experience), there’s lots of good information here but for this exercise we are interested in only the “SSL Negotiation” time.

Since Facebook requires authentication to see the “full experience” I just tested the log-on page, it should accurately reflect the SSL performance “tax” for the whole site.

I ran the test four times, each time summing the total number of milliseconds spent in “SSL Negotiation”, the average of these three runs was 4.111 seconds (4111 milliseconds).

That’s quite a bit but can we reduce it? To find out we need to look at their SSL configuration; when we do we see a few things they could do to improve things, these include:

Let’s explore this last point more, the status check the browser does is called an OCSP request. For the last 24 hours their current CA had an average world-wide OCSP response time of 287 ms, if they used OCSP Stapling the browser would need to do only one OCSP request, even with that optimization that request could be up to 7% of the SSL performance tax.

Globalsign’s average world-wide OCSP response time for the same period was 68 milliseconds, which in this case could have saved 219 ms. To put that in context Facebook gets 1.6 billion visits each week. If you do the math (219 * 1.6 billion / 1000 / 60 / 24), that’s 12.7 million days’ worth of time saved every year. Or put another way, it’s a lifetime worth of time people would have otherwise spent waiting for Facebook pages to load saved every two and a half hours!

If we consider that in the context of the Amazon figure simply changing their CA could be worth nearly one hundred million a year.

Before you start to pick apart these numbers let me say this is intended to be illustrative of how performance can effect revenue and not be a scientific exercise, so to save you the trouble some issues with these assumptions include:

  • Facebook’s business is different than Amazons and the impact on their business will be different.
  • I only did four samples of the SSL negotiation and a scientific measurement would need more.
  • The performance measurement I used for OCSP was an average and not what was actually experienced in the sessions I tested – It would be awesome if WebPageTest could include a more granular breakdown of the SSL negotiation.

With that said clearly even without switching there are a few things Facebook still can do to improve how they are deploying SSL.

Regardless I am still thrilled Facebook has decided to go down this route, the change to deploy Always-On-SSL will go a long way to help the visitors to their sites.

Ryan

Priming the OCSP cache in Nginx

5 Replies

So recently GlobalSign, DigiCert, and Comodo worked together with Nginx to get OCSP stapling supoported in Nginx 1.3.7, unfortunately architectural restrictions made it impractical to make it so that pre-fetching the OCSP response on server start-up so instead the first connection to the server primes the cache that is used for later connections.

This is a fine compromise but what if you really want the first connection to have the benefit too? Well there are two approaches you can take:

  1. Right after you start the server you do a SSL request to prime the cache.
  2. You manually get the ocsp response and plumb it where Nginx is looking for it.

The first model is easy, right after you start your server use the OpenSSL s_client to connect to the server with OCSP stapling enabled  just like I documented in this post, the first request will trigger the retrieval of the OCSP response by Nginx.

The second model can be done before you start the server, you need to find the URI for the OCSP responder, do a OCSP request and populate the Nginx cache manually, this would look something like:

#!/bin/sh
ISSUER_CER=$1
SERVER_CER=$2

URL=$(openssl x509 -in $SERVER_CER -text | grep “OCSP – URI:” | cut -d: -f2,3)

openssl ocsp -noverify -no_nonce -respout ocsp.resp -issuer \
$ISSUER_CER -cert $SERVER_CER -url $URL

Where “ocsp.resp” is whatever file you have configured in Nginx for the “ssl_stapling_file“.

Each approach has its pros and cons, for example with the first approach your execution of the s_client call may not be the first request the server sees, with the second approach if you are using a certificate that doesn’t contain a OCSP pointer and have manually told Nginx where to fetch certificate status from then it won’t work.

It is worth noting you can run this same script in a cron script to ensure your server never needs to hit the wire (and potentially block when doing so) when it tries to keep its OCSP cache up to date.