Testing OCSP Stapling

9 Replies

So you have configured OCSP stapling and you want know if it’s actually working, it’s easy enough to check using the openssl  s_client command:

openssl s_client -connect login.live.com:443 -tls1  -tlsextdebug  -status

Loading ‘screen’ into random state – done

CONNECTED(0000017C)

TLS server extension “status request” (id=5), len=0

….

OCSP response:

======================================

OCSP Response Data:

OCSP Response Status: successful (0x0)

…

Cert Status: good

This Update: Jun 12 02:58:39 2012 GMT

Next Update: Jun 19 02:58:39 2012 GMT

…

In this example you see that the client is requesting the servers OCSP response, you then see the server providing that response successfully and openssl determining the servers certificate is good.

For another example we can query the US Mint’s website for an example of a site that has not yet (and probably won’t for some time since it’s a government site) configured OCSP stapling:

openssl s_client -connect www.usmint.gov:443 -tls1  -tlsextdebug  -status

Loading ‘screen’ into random state – done

CONNECTED(0000017C)

OCSP response: no response sent

….

Hope this helps you deploy OCSP Stapling successfully.

Ryan

OCSP Stapling in IIS

3 Replies

Windows Server 2008 and later support a feature called OCSP stapling. When enabled a server pre-fetches the OCSP response for its own certificate and delivers it to the user’s browser during the TLS handshake. This approach offers a privacy advantage. But, the main benefit is the browser doesn’t have to make a separate connection to the CA’s revocation service before it can display the Web page. This gives better performance and reliability.

For this pre-fetching to work the web-server certificate needs to contain a pointer to the OCSP responder, this is a best practice and a recommendation of the CA/Browser Forums baseline requirements so it’s almost certain your certificate has it.

Unlike Apache this feature is enabled by default, it’s possible your servers are already doing OCSP stapling and you do not even know it.

With that said chances are you have a firewall between your webservers and the internet; it’s also likely that firewall disallows outbound connections from your servers unless explicitly allowed. So you might need to allow your web servers to communicate with the OCSP responder before it will work.

To figure out what host and port you will need to open you will need to look at the certificates you use on your webserver; one way to do that is to browse to your current site and view the details of the certificates you are currently using, for example:
   

The value you want is in the “Authority Information Access” field, you want the ones (there may be multiple) that have the “Access Method” of “On-line Certificate Status Protocol”.

Once these two conditions are met OCSP Stapling will “Just work” there is nothing else you need to do.

Ryan

OCSP Stapling in Apache

4 Replies

Apache 2.3 and later support a feature called OCSP stapling. When enabled a server pre-fetches the OCSP response for its own certificate and delivers it to the user’s browser during the TLS handshake. This approach offers a privacy advantage. But, the main benefit is the browser doesn’t have to make a separate connection to the CA’s revocation service before it can display the Web page. This gives better performance and reliability.

For this pre-fetching to work the web-server certificate needs to contain a pointer to the OCSP responder, this is a best practice and a recommendation of the CA/Browser Forums baseline requirements so it’s almost certain your certificate has it.

Chances are you have a firewall between your webservers and the internet; it’s also likely that firewall disallows outbound connections from your servers unless explicitly allowed. So before you enable OCSP stapling you are going to need to allow your web servers to communicate with the OCSP responder.

To figure out what host and port you will need to open you will need to look at the certificates you use on your webserver; one way to do that is via OpenSSL, for example:

1. Get the certificate using s_client

openssl.exe s_client -connect www.globalsign.com:443
…
…
—–BEGIN CERTIFICATE—–
—–END CERTIFICATE—–
…
…

You need to copy the PEM header and footer (“—–BEGIN/END CERTIFICATE—–“) and the Base64 between them into a file.

2. Identify the OCSP responders within the server certificate

openssl.exe x509 -in globalsign.com.cer -text
…
…
X509v3 extensions:
..
Authority Information Access:
CA Issuers – URI:
http://secure.globalsign.com/cacert/gsextendvalg2.crt
OCSP – URI:
http://ocsp2.globalsign.com/gsextendvalg2
…
…

You need to find the “OCSP – URI” section, in the example certificate above the OCSP responder is http://ocsp2.globalsign.com/gsextendvalg2 there may be multiple responders specified, you should allow your servers to initiate outbound traffic to all of them.

Once your servers can request OCSP responses enabling stapling is very straight forward, there are just two directives that need to be added, these directives can be global or specific to a specific to one instance:

SSLUseStapling on
SSLStaplingCache “shmcb:logs/stapling_cache(128000)”

Their purpose is self-evident; SSLUseStapling turns the feature on while SSLStaplingCache specifies where to store the cache and how big it should be.

There are other directives also you can use but you should not need to worry about them.

As long as you are running the most recent stable versions of Apache and OpenSSL enabling this feature is safe. It is only used when the client supports it so there won’t be compatibility issues and if the server for some reason fails to populate its cache with a valid OCSP response the client will typically fall back to doing a live OCSP request on its own.

Ryan

 

Additional Resources

Overclocking Mod_SSL

Flame and Certificate Revocation

Leave a reply

Microsoft has published patches that insert the CAs directly associated with the Terminal Services Licensing PKI into the “Untrusted” certificate store, this has the same effect as revoking the certificates for those that:

  1. Apply the patch
  2. Use Microsoft’s chain validation engine

But what about those that do not apply the patch? Or those that simply import the roots from the Microsoft store to use in another engine (like NSS or OpenSSLs, one example of such an application would be Adobe X)?

Well today those people will still trust certificates issued by those certificates, they potentially would not though if Microsoft published revocation information for those CAs.

As it stands most of the certificates in question do contain revocation pointers (a CRLdp extension in this case) but the URLs they point to are invalid, for example:

  1. “Microsoft Enforced Licensing Intermediate PCA” – No revocation information
  2. “Microsoft Enforced Licensing Registration Authority CA” – http://crl.microsoft.com/pki/crl/products/MicEnfLicPCA_12-10-09.crl
  3. “Microsoft LSRA PA” –  http://crl.microsoft.com/pki/crl/products/MicEnfLicRegAutCA_2009-12-10.crl
  4. “Terminal Services LS” – http://tk5paxprdsrv09.partners.extranet.microsoft.com/CertEnroll/Microsoft%20LSRA%20PA.crl

This may not be beneficial for the attack vector used in Flame (I do not know if it even does revocation checking – though it very probably does) but it would certainly help other cases.

Today the invalid URLs in these certificates would result in a “Unknown or Offline” response which would likely be ignored by applications due to Microsoft’s “soft-fail revocation checking” defaults.

Even with those defaults though if they published these CRLs clients would get some benefit, it would:

  1. Reduce risk until the patch could get applied.
  2. Help others who mistakenly trusted the Microsoft roots when they “imported” them from the “Trusted Root Store”.

It’s my hope Microsoft decides to publish these CRLs for relying party applications to rely on.

[3:22 PM 6/19/2012] FYI as of 6/19 the MicEnfLicPCA_12-10-09.crl is published (good), it appears to have been created June 14th several days after this post. The other two are still invalid, it’s likely they decided against publishing them since a well behaved client would fail with just one of the CAs being bad. It’s not a unreasonable decision, I am guessing the URL choices for the CRLs made it difficult to do all of the CRLs.

Ryan

Flame was just one use of the Terminal Services Licensing PKI

Leave a reply

I wanted to do a post on how the Terminal Services Licensing PKI could be used in another attack; though the variants are endless I have one concrete example of how this was used in in an attack in 2002 in this certificate:

   
   

 

What we see in this certificate is that the attacker took the License Certificate (Terminal Services LS) and issued himself a certificate underneath it. This only worked because of another defect in the Windows chain engine, this one relating to how it handled Basic Constraints (see MS 02-050) – that was fixed in 2003.

But why would he have done this? I see two possible reasons:

  1. Bypassing the terminal services licensing mechanism.
  2. Bypassing some client authentication scheme based on X.509.

We won’t explore #1 as I am not interested in the attacks on license revenue for Microsoft but how this infrastructure designed to protect that license revenue was used to attack customers and as such I will focus on #2.

Now, before I go much further I have to be clear – I am just guessing here, I don’t have enough data to be 100% sure of what this was used for but it seems pretty likely I am right.

What I see here is the X.509 serial number is different than the Serial Number RDN the attacker put into the Subject’s DN.

This certificate also has a value in the CN of the Subject DN that looks like it is a user name (PORTATILE).

Neither of these should be necessary to attack the licensing mechanism as legitimate license certificates don’t have these properties or anything that looks roughly like them.

My theory is that the attacker found some Windows Server configured to do client authentication (possibly for LDAPS or HTTPS) and wanted to trick the server into mapping this certificate into the principal represented by that CN and Serial Number in the DN.

Why would this have worked? Well as I said there was the Basic Constraints handling defect, but that wouldn’t have  been enough the server would have had to had blindly trusted all the root certificates in the root store for “all usages”.

This was a fairly common practices back then, web servers and other server applications would be based on other libraries (for example SSLEAY/OpenSSL) and just import all the roots in the CryptoAPI store and use them with this other library.

This is actually still done by some applications today, for example Adobe Reader X:

 

The problem with this behavior is that these applications do not necessarily (read almost never) import the properties associated with these certificates, nor do they support the same enforcement constraints (for example Nested Extended Key usages).

The net effect of which is that when they import the roots they end up trusting them for more than the underlying Windows subsystem does.

The attackers in this case were likely taking advantage of this bad design of this client authentication solution as well as the bad design of the Terminal Services Licensing PKI.

The combination of which would have allowed the attacker to impersonate the user PORTATILE and perform any actions as that user they liked.

Even if I am wrong in this particular case, this sort of thing was definitely possible.

What lessons should we take away from this?

For one always do a formal security review of your solutions as you build them as well as before you deploy them; pay special attention to your dependencies – sometimes they are configuration as well as code.

Also if an issue as fundamental as the Terminal Services Licensing PKI can fly under the radar of a world class security program like Microsoft’s you probably have some sleepers in your own infrastructure too – go take a look once more for the assumptions of the past before they bite you too.

Ryan

How did the Terminal Services Licensing PKI effect you?

Leave a reply

The other day I did a post on the age of the Microsoft PKI that was used for Terminal Services Licensing, today I thought I would talk about what that age meant in the context of the vulnerabilities it introduced.

The oldest certificate I have been able to find from the same hierarchy is from April 1999 (that’s the issuance date for the Microsoft Enforced Licensing PCA).

Based on the post from the Microsoft Security Research & Defense blog  we know that the reason the attacker had to do the MD5 collision was that as of Vista there was a change in the way critical extensions were handled.

This change made it so that Vista clients would fail when they saw a certificate that contained an unknown critical extension in a signing certificate making it an ineffective attack vector for those clients.

But what does this mean for the period of time before that? Well Windows Vista was released in November of 2006, that’s nearly 8 years in which any enterprise with a terminal services deployment could have owned a Windows PC “As Microsoft” or potentially attack a PKI based system with “trusted” but fraudulent certificate.

But did it really get better in with the release of VISTA? According to StatCounter Windows Vista received its maximum market share of 23% in October of 2009. Yes, two years after the release of Vista 77% of the Windows clients on the Internet were still vulnerable as a result of the design of the terminal services licensing solution.

Things didn’t really start to get better until XP SP3 which was released in April of 2008 as it contained the same certificate validation engine that was found in Vista.

While I do not have any public statistics I can share I can say that this service pack was picked up faster than any other service pack up until that point which says a lot since it was not a “forced” update.

If we are optimistic and say that it took only one year to get 100% penetration and we believe stat counters statistics for XPs Market share of 71% in April 2009 that it took till 2009 to get to 95% patched.

Now these numbers are just for clients on the Internet and not servers. This “fixed” chain validation engine wouldn’t have even found its way into the Windows Server code base until Windows Server 2008 which was released in February 2008 but took some time to get broadly used.

While Windows Servers are not terribly common on the Internet they are extremely common in the Enterprise, especially in 2009 where it had 73.9% market share. Again I don’t have numbers but antidotal we know that Enterprises are notoriously slow to upgrade or patch.

So where does this leave us in 2012? Still vulnerable that’s where.

The other day Microsoft released a patch that in-essence revokes the PKI in question and today WSUS announced their patch that introduces additional pinning.

You need to apply both to secure your systems.

Ryan

How one typically verifies code comes from Microsoft

1 Reply

Microsoft signs all code that ships from it (that is except for the case when it doesn’t); so how does one verify that the code came from them vs. anyone with a fraudulent certificate claiming to be them?

Well the answer is pinning, but the pinning for the most part does not happen at the signing certificate level as those keys change to frequently to be pinned. The pinning happens at the certificate issuer level, much like has been proposed by CAA.

So to understand how this happens in code you need to understand code signing in Windows a little. There are two types of signatures those that are “embedded” and those that are “attached”.

Authenticode would be an example of an embedded signature; Catalog Files on the other hand are detached signatures. There are other examples of both of these but in general these other formats have the same abstract properties.

So when do you use one vs. the other? Well if a user will download your binary directly (ActiveX, Setup, etc.) you would typically embed a signature.

If that’s not the case you would use a detached signature because a single signature can be placed on a list of binaries without increasing the package size (it’s one signature on n hashes). You produce these signatures with SignerSignEx.

For the two signature formats I mentioned you do your signature verifications using an API called WinVerifyTrust. You can also use this API to get information out of a signature (who signed it, when, etc.).

With that information you do a certificate validation that is accomplished with the CertGetCertificateChain API, that gives you a CERT_CHAIN_CONTEXT which you can use to call CertVerifyCertificateChainPolicy.

This API is used to do application specific verifications, for example in the case of TLS the CERT_CHAIN_POLICY_SSL provider is used – this is where the name and EKU verification logic for Windows exists.

The providers we care about for our story today are:

  • CERT_CHAIN_POLICY_AUTHENTICODE – Is this certificate appropriate for code-signing?
  • CERT_CHAIN_POLICY_AUTHENTICODE_TS – Is this certificate appropriate for time stamping code?
  • CERT_CHAIN_POLICY_MICROSOFT_ROOT – Is this certificate ultimately issued by a Microsoft Product Root?

The last one is the one in has an array of hashes in it, I don’t recall if they are key hashes or thumbprints of the certificate. In essence what it does is look at the top of the chain to see if the Root matches one of these hashes if it does it passes, if it doesn’t it fails.

Applications may choose to do additional “pinning” also, for example in the case of Windows Update Services a manifest is exchanged between the client and server, that manifest comes down over SSL they may choose to also do pinning to a set of certificates for the SSL session or apply digital signatures to the manifest and do pinning for the keys that are used to do signing.

I do not know personally for a fact that any such pinning takes place but I have been told by several people I respect that it does.

It looks as if Windows Update was not doing pinning but as of this week it will, this is of course  a best practice and I am glad to see them doing it now.

So now you know how to validate that code is coming from Microsoft, or at least how it’s supposed to work.

Issues like represented in MSRC 2718704 / Flame put some of these assumptions in question, it’s clear that pinning is an important concept though and one that should be supported more broadly.

Ryan

The tale of two (Microsoft) PKIs

Leave a reply

As you know I used to work at Microsoft on areas surrounding cryptography, certificates, protocols and other such things.

While I can’t talk about the internal workings of things I can explain the high level roles the two sets of PKIs Microsoft maintains.

The first PKI is the Microsoft “Product Roots”; these are the keys and certificates that are managed with the sole purpose of being used for product scenarios. You can see the names and serial numbers of these CAs here.

The next is the “IT Root”, that is the one folks see associated with a S/MIME email from a Microsoft employee, some of the corporate websites have SSL certificates from this hierarchy and if you were to inspect the badges of an employee you would see a certificate from here that is used for smart card login and other related scenarios.

The “Product Roots” are trusted only by Windows and applications that have either built on the Windows CryptoAPI or simply imported the roots from the Microsoft Root Program.

The “IT Root” is trusted by third-party browsers because it has been cross signed with a public CA, currently that CA is Verizon the certificate is labeled “GTE CyberTrust”.

I hope that helps clarify for people,

Ryan

MSRC 2718704 and the age of the rogue certificates

Leave a reply

One of the things I love about security is how security researchers are so passionate about their jobs; they for the most part respect their peers and enjoy learning about attacks so they are willing to share information.

As a result of this dynamic over the past few days I have had a chance to look at a number of details about the #Flame malware as well as the rogue certificates it used as part of the its Windows Update attack.

With that said the raw data having already been published here quite a long time ago I think it’s OK to go into details on this now (Another interesting data point is that this link is discussing how to bypass the Terminal Services license mechanism).

This certificate chain gives us some insights into the age of the solution Microsoft was using to license Terminal Server Clients, for example one certificate chain I received shows us the following:

   
   
   
   
 

 

From this we can tell a few things, including:

  1. The licensing solution has been in use since at least 1999 – Its issuance date is ‎Thursday, ‎April ‎15, ‎1999 12:00:00 AM, we can assume this is the case if we assume the “Microsoft Enforced Licensing Intermediate PCA” is in a Microsoft vault (very safe to assume).
  2. The “Licensing Server” was built to include unverified information in each license originally – We can assume this since the other certificates we saw included no organization details in the subject name while this one has a DN of  “CN = Terminal Services LS, L = Stradella, S = PVC =  IT, E = ater**@libero.it”
  3. The PKI was rebuilt in around 2004 – In the first certificates we saw the Microsoft operated CA that issued the “licensing server” (Microsoft Enforced Licensing Registration Authority CA) expired in 2009, in the most recent certificate the chain is a one leaf shorter but same logical entity expires in ‎2004.
  4. The format of the certificates changed around 2004 – In the first certificates we saw only a single proprietary extension, in this one we see four. This suggests that there was a code change in how licenses were validated around this time. This also approximately aligns with when one might expect the Windows Server development to begin for the next version (at the time development cycles were 3-4 years).
  5. Attackers were using this in combination with other attacks even then – In 2003 there was a security defect in the way Windows handled Basic Constraints (see MS 02-050). This allowed certificates that were not CAs to be trusted to issue CA certificates.

In this most recent certificate chain we can see that a Terminal Services LS (a license) has issued another certificate, the validity of this certificate was based on a combination of two factors:

    • The root CA being trusted by the entity validating the certificate.
    • The license certificate being able to issue other certificates.

This combination allowed the attacker to simply have access to a CAL, not even the License Server and produce a certificate that could be used to attack another system – I will do a post about this later.

  1. Terminal Services doesn’t do full certificate validation on its licenses – The post I got this certificate from states it works on machines as recent as 2008 but the “Terminal Services LS” expires in in 2004, the certificate that was minted bellow it (the evil “PORTATILE” one) is good until 2038. This would not pass the CryptoAPI certificate validation logic on those platforms due to the BasicConstraints restrictions on the certificate and the date issues alone.

So in summary, this issue has been around a very long time, much longer than Flame or Stuxnet. There are a few very interesting conclusions I come to as I write this:

  1. Even without the Basic Constraints issue no collision attack would have been needed to make use of this original Licensing PKI .
  2. Despite the changes in the client behavior in Windows Vista in how Critical Extensions are handled, the weak cryptography made it practical to continue to use this PKI maliciously.
  3. Those who knew about this tried to keep it a secret as long as possible as it was very valuable.
  4. Despite developing a world-class security program over the last decade this issue managed to fly under the radar until now.

MSRC 2718704 and Terminal Services License Certificates

1 Reply

When looking at the certificates the Flame authors used to sign the malware in their Windows Update module one has to wonder what was different between a normal terminal services license certificate and the one they used.

Well getting at one of these certificates is a little awkward as they are kept in the registry vs. in the traditional certificate stores, thank goodness with a little due diligence you can find one.

   
   

 

So from this what can we learn? There are a few things:

  1. Every subordinate CA has the same name “Microsoft LSRA PA”
  2. Every terminal services license has the same name “Terminal Services LS”
  3. The license appears to have a random serial number (even the CA certs did not!)
  4. The license has a 512bit RSA key.
  5. The license is signed with MD5.
  6. No restrictions have been explicitly placed on this certificate.
  7. This license is good for 257 days.
  8. The CRL url is invalid.
  9. The Certificate Issuer url is invalid.
  10. It contains a proprietary extension (1.3.6.1.311.18)

As far as best practices go there are clearly a lot of problems here:

  1. Same name for every CA – the point of a CA is to identify an entity, if every CA has the same name this can’t really happen.
  2. Same name for every subject – while the certificates here are for authenticating entitlement to make a connection and not to authenticate an identity the subject should still be unique.
  3. Over entitled– The certificate contains no Extended Key Usage or other restrictions, as a result this certificate is only as restricted as its issuers issuer is, namely it is good for:
    • Code Signing (1.3.6.1.5.5.7.3.3)
    • License Server Verification (1.3.6.1.4.1.311.10.6.2)
  4. Weak keys – RSA 512 bit was factored in 1999, there should be no CA issuing 512 bit keys period especially in a PKI as powerful as this, especially for one that is good for code signing.
  5. Weak hash algorithm – MD5 has been known to be weak since 1993 and has had increasingly effective attacks published against it ever since — There should be no CA issuing MD5 signed certificates in 2012.
  6. Invalid URLs – This indicates that the CA was not actively managed, it also included internal only file references in the certificate also a no-no and a indicator of the same.
  7. Poorly thought out proprietary extension – The extension is not well thought out, the OID (1.3.6.1.311.18) is an arc for Hydra (Terminal Services) and not a unique identifier (or shouldn’t be at least) for a specific extension. This isn’t harmful but a signal the design didn’t go through much review.

The other items are odd, like the 257 day validity window and the proprietary extension but they are not problematic – at least on the surface.

Well that’s what we know so far.