Wanted: Network and System Specialist [Singapore]

Title: Network and System Specialist

Location: Singapore

Languages: English, Japanese is a plus.

 

Who we are

GlobalSign was formed in 1996 as one of the Internet’s original trust service providers (you probably know us as a Certificate Authority). Over the years we have issued millions of digital certificates that have been used to secure commerce and communication on the Internet. Our solutions take the pain out of using cryptography and help organizations solve complex problems with increased productivity and peace of mind.

 

Who we’re looking for

We are looking for an up-and-coming system administrator with a passion for security. For the right candidate this is an amazing opportunity to build and operate a high-availability secure system to power modern web applications

 

What they will do

  • Administration and maintenance of network and system infrastructure including: Maintain service availability, reliability, performance and facilitate incident response
    • Network equipment (Routers, Switches , Firewall / IPS).
    • Load balancers, proxy Servers and web servers.
    • Unix hosts (Redhat) and Databases (SQL).
  • Hardening and monitoring of security of hosts and network devices.
  • Ensure that the network is operating at optimum performance and with high availability by continually reviewing areas of improvement.Support disaster recovering and business continuity exercises.
    • Recommend the optimum or alternative solutions.
    • Monitor and maintain service availability.
    • Ensure procedures are documented.
    • Ensure compliance to policies and procedures.
    • Follow up on requests, deployments & incidents/issues (including escalation) and ensure proper and timely closure.
  • Ability to diagnose and troubleshoot performance, and reliability of all aspects of system.
  • Development and deployment of automation of common tasks.

 

Skills & Experience

  • At least two years’ experience in a similar role.
  • Ability to work in a team and individually.
  • Ability to multi-task and work in a fast pace environment
  • Prior experience maintaining up-to-date operation and procedure documentation.
  • Candidate able to work extended hours and weekends
  • Required to carry duty Mobile and response to urgent calls
  • Strong preference for applicants with experience with PKI and cryptographic key management.
  • Preference for experience using virtualization.
  • Preference for applicants with experience working in a high-availability environment with %99.999 up-time.

 

GlobalSign is an equal opportunity employer with locations all over the world. Aside from being a great place to work we offer an excellent benefit package that includes Health, Dental, 401k, Life Insurance, and a generous time off and holiday schedule.

All applicants will be considered without regard to race, color, religion, sex, national origin, age, marital or veteran status; medical condition, disability; or any other legally protected status.

 

Wanted: Software Engineer [Seattle/Manila]

Title: Software Engineer

Location: Greater Seattle Area and Manila

Languages: English

 

Who we are

GlobalSign was formed in 1996 as one of the Internet’s original trust service providers (you probably know us as a Certificate Authority). Over the years we have issued millions of digital certificates that have been used to secure commerce and communication on the Internet. Our solutions take the pain out of using cryptography and help organizations solve complex problems with increased productivity and peace of mind.

 

What we’re going to do

The web has changed a lot since 1996 but how we bootstrap trust on it has not changed much – we are going to fix that.

 

Who we’re looking for

We are building a small engineering team here in the Seattle area and another in Manila and need developers who feel comfortable on both the front- and back-end. People who love learning new stuff, self-starters — true creative problem solvers.

We want someone who feels knows behavior between and across browsers, best practices, and when and it and is not okay to use an OnClick event.

Someone who believes tested code is happy code, that when done right security can enable new scenarios, that APIs should be easy to consume and that a well thought out platform is significant part of a projects success.

 

Skills & Experience

  • HTML/CSS/Javascript (jQuery, AJAX, etc.) with a healthy interest in HTML5, REST and JSON.
  • Experience with one or more server-side languages and frameworks especially Java, Node.js and PHP.
  • Interest in developing highly interactive web applications with performance, scalability, and usability in mind.
  • Experience working with Linux, Apache, and relational databases. Bonus points for having written your own stored procedure before.
  • Experience developing software in a professional environment, including source control, bug tracking, unit testing
  • Understanding of security risks and secure software development.
  • Understanding of the similarities and differences across browsers (both young and old).
  • Love your version control (Git preferably).
  • Enjoys prototyping and iterating stuff.
  • Bonus points for speaking Japanese.

 

Required Qualifications

  • 2 years dynamic / scripting language programming, with background in Java and C++ programming preferred.
  • 1+ years experience in Systems Engineering / Administration with firm understanding of *nix architecture.
  • Bachelor’s degree in Computer Science, or equivalent experience. Engineering or related discipline highly recommended.
  • Awesomeness trumps all other requirements.

 

If this sounds like it could be you, send us a CV and some examples of the work that you’re most proud of.

 

GlobalSign is an equal opportunity employer with locations all over the world. Aside from being a great place to work we offer an excellent benefit package that includes Health, Dental, 401k, Life Insurance, and a generous time off and holiday schedule.

All applicants will be considered without regard to race, color, religion, sex, national origin, age, marital or veteran status; medical condition, disability; or any other legally protected status.

 

Wanted: Senior Software Engineer (Front-End) [Seattle]

Title: Senior Software Engineer (Front-End)

Location: Greater Seattle Area

Languages: English

 

Who we are

GlobalSign was formed in 1996 as one of the Internet’s original trust service providers (you probably know us as a Certificate Authority). Over the years we have issued millions of digital certificates that have been used to secure commerce and communication on the Internet. Our solutions take the pain out of using cryptography and help organizations solve complex problems with increased productivity and peace of mind.

 

What we’re going to do

The web has changed a lot since 1996 but how we bootstrap trust on it has not changed much – we are going to fix that.

 

Who we’re looking for

We are building a small engineering team here in the Seattle area and need a senior developer who is passionate about security and creating beautiful user experiences. In this role you’ll be able to use your experience to help us recruit new team members and build our engineering team to into a powerhouse.

We want someone who feels someone who feels comfortable on both the front- and back-end, who loves learning new stuff, who’s entrepreneurial, a technical leader, a true creative problem solver.

You’ll have experience with security technologies but your background will be focused on building experiences that bridge the gap between user needs and technology requirements. You’ll also have experience building fault-tolerant, high performance, and highly scalable systems.

 

Skills & Experience

  • HTML/CSS/Javascript (jQuery, AJAX, etc.) master with a healthy interest in HTML5, REST and JSON.
  • Experience with one or more server-side languages and frameworks especially Java, Node.js and PHP.
  • Experience designing highly interactive web applications with performance, scalability, usability, and security in mind.
  • Experience with relational database schema design and queries.
  • Experience developing software on Unix/Linux.
  • Love your version control (Git preferably).
  • Understanding of applied cryptographic concepts such as certificates, certificate chains, key management.
  • Understanding of security risks and secure software development.
  • Enjoys prototyping and iterating stuff.Bonus points for speaking Japanese.

 

Required Qualifications

  • 5 years dynamic / scripting language programming, with background in C/C++ systems programming preferred.
  • 2+ years experience in Systems Engineering / Administration with firm understanding of *nix architecture.
  • Strong Computer Science fundamentals (data structures and algorithms).
  • Bachelor’s degree in Computer Science, or equivalent experience. Engineering or related discipline highly recommended.
  • Awesomeness trumps all other requirements.

 

If this sounds like it could be you, send us a CV and some examples of the work that you’re most proud of.

 

GlobalSign is an equal opportunity employer with locations all over the world. Aside from being a great place to work we offer an excellent benefit package that includes Health, Dental, 401k, Life Insurance, and a generous time off and holiday schedule.

All applicants will be considered without regard to race, color, religion, sex, national origin, age, marital or veteran status; medical condition, disability; or any other legally protected status.

How to seed the CryptoAPI URL cache with an OCSP response

It is possible to “staple” an OCSP response into higher level protocols such as TLS. This concept has been supported in Windows since Windows VISTA, shortly after it was added to OpenSSL/Apache and soon it will also be in Nginx.

When “stapling” is used the subscriber (the web server in the TLS case) requests the status of his own certificate from the OCSP responder his CA operates to get a time valid OCSP response for his own certificate.

Since the OCSP response is signed by the CA it can be relayed by the web server for the duration in which that OCSP response is time valid to save the client of the web server (the relying party, aka the browser) the need to make an additional socket connection back to the CA.

This has both performance and privacy benefits.

You can apply the same concept to other PKI related protocols/applications as well, for example in a document signing application like Adobe Acrobat. In such an application the subscriber might sign the document, timestamp it using a timestamp protocol like RFC 3161 and then attach a time valid OCSP response to it so that the document is verifiable at a later date.

This scenario is important not only for its performance and privacy benefits but because it is a practical necessity because CAs do not typically maintain revocation information (OCSP responses and CRLs) for expired certificates.

By stapling the OCSP response to the signed and time-stamped document the relying party can verify the signature, the certificates and the revocation status of the certificates in the context of the timestamp that was attached to the document.

But how do you do that with CryptoAPI?

It’s actually pretty straight forward, as the relying party when you call CertGetCertificateChain to validate the associated certificate you need to:

  1. Once you have verified your timestamp is cryptographically valid and trusted, take the time it and pass it in as pTime.
  2. On the CERT_CONTEXT there is a CERT_INFO structure that contains an array of CERT_EXTENSION, here you create an extension of type CERT_OCSP_RESPONSE_PROP_ID and in there you put a basic signed OCSP response .

When CryptoAPI does the chain validation it will try to use the OCSP response you passed in, if it finds a problem with the provided response it may go online to get a new one that is “OK”.

This online behavior can be controlled by indicating to CertGetCertificateChain you do not want online revocation checking (see CERT_CHAIN_REVOCATION_CHECK_CACHE_ONLY).

Ryan

CryptoAPI, Revocation checking, OCSP and the Unknown certStatus

In CryptoAPI one can use the CertGetCertificateChain API to do the path building and basic chain validation, this validation may include revocation checking depending on which flags you pass via dwFlags; for example these flags control if revocation checking occurs, and if so, on which certificates:

  • CERT_CHAIN_REVOCATION_CHECK_END_CERT
  • CERT_CHAIN_REVOCATION_CHECK_CHAIN
  • CERT_CHAIN_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT

Typically you would specify CERT_CHAIN_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT which ensures the whole chain is checked (where possible – e.g. one shouldn’t bother asking the root if he considers himself revoked).

But in the context of OCSP what are the potential revocation related returns we might see?

  1. Revoked – I have received a signed response from the CA or have had policy pushed to me that tells me that this certificate is not to be trusted.
  2. Not Revoked – I have received a signed response from the CA that says this certificate was not revoked.
  3. Unknown – I have received signed response from the CA that says it doesn’t know anything about this certificate.
  4. Offline – I was unable to reach the responder to verify the status of the certificate.

Each of these cases are clearly represented in the API that is used by CertGetCertificateChain to perform the revocation check, this API is CertVerifyRevocation. The higher level CertGetCertificateChain however only has two possible returns: Revoked and Unknown outside the “not revoked” case.

One might assume the OCSP Unknown would get mapped into the Revoked state, this unfortunately is not the case, it is returned as unknown, as does the Offline error.

This means that if OCSP was used you cannot tell what the actual status was, this is especially problematic since IE and Chrome both default to modes where they ignore “Unknown” revocations due to concerns over Revocation responder performance and reliability.

It is possible to work around this platform behavior though — the problem is that it’s not documented anywhere, let’s take a quick stab at doing that here.

First since CertGetCertificateChain doesn’t tell us what method was used to do revocation checking we have to use heuristics to figure it out. Thankfully to enable OCSP stapling in higher level protocols like TLS the OCSP response is passed back to the caller if OCSP was used; to get that we need to know:

1. The CERT_CHAIN_ELEMENT in the returned CHAIN_CONTEXT points to the following revocation information:

  • PCERT_REVOCATION_INFO pRevocationInfo;
  • PCERT_REVOCATION_CRL_INFO   pCrlInfo;
  • PCCRL_CONTEXT           pBaseCrlContext;

2. For an OCSP response, the CRL_CONTEXT is specially created to contain the full OCSP response in the following critical extension szOID_PKIX_OCSP_BASIC_SIGNED_RESPONSE.

  • The presence of this extension indicates an OCSP response was used.
  • The Issuer name in the CRL is the name of the OCSP signer.
  • The signature algorithm is sha1NoSign.
  • ThisUpdate and NextUpdate contain the response validity period.

With this we can take the OCSP response from the CRL_CONTEXT and then look at the “ResponseData” within it, you will need to look within the “responses” here,  you will need to find the right “SingleResponse” based on its “CertID”.

NOTE: Some responders will return the status of multiple certificates in a response even if the status of only one was requested.

You now can determine what the responder said the status of the certificate was by inspecting “certStatus” element.

This is a fair amount of work unfortunately but it does enable you to do the right thing with authoritative “Unknown” responses.

Ryan

Serving OCSP on a CDN

So last week we moved our revocation repositories behind a CDN, this has a number of great benefits but it does have downsides though, for example.

  1. Cache misses result in a slower response – About a 110ms in my tests.
  2. OCSP clients that create POSTs get a slower response – This is because they are treated like a cache miss.

Our CDN provider mitigates much he first issue by having a pre-loader that ensures its cache is pre-populated based on request history.

Addressing the second issue requires the CDN provider to be aware of OCSP protocol semantics, specifically the fact that one can compute what a GET request would look like by simply Base64 encoding the binary body of the POST variation.

A CDN with knowledge of this can optimize out the POST derived cache miss, our CDN has done this the change has not yet propagated to all of their datacenters but where it has POST performs the same as a GET.

Hopefully this optimization will have propagated to all their datacenters by next week, when this logic is fully deployed the clients that generate POST based OCSP requests (without a nonce) will also have ~100ms response times.

I should probably ad that in our case almost all of the POST based OCSP requests we receive come from Firefox and do not contain a nonce, hopefully soon Firefox will move to using GET for requests without a nonce like other clients.

Using Monitis to monitor OCSP and CRL performance

Earlier I did a post on the performance of revocation repositories, in that I used the Monitis do some basic bench-marking for repository performance.

They have a cool feature that allows me to create a public monitoring portal on one of my own domains, I have created two:

  1. OCSP Performance
  2. CRL Performance

It also looks like CloudFlare (one of our CDN partners), the ones we are working with to get these great numbers have also done a blog post on what we are doing here – cool stuff!

Ryan

 

Revocation repositories, IPv6 support, message size, and performance

So the last few weeks I have spent a reasonable amount of time looking at performance and networking related problems associated with revocation repositories. I still have some additional work to do but I figured I would capture some of my findings in a quick post.

CRL repositories and IPv6
CA Host Supports V6
GlobalSign crl.globalsign.com Yes
Comodo crl.comodoca.com No/Sometimes
CyberTrust sureseries-crl.cybertrust.ne.jp No
Symantec/GeoTrust EVSSL-crl.geotrust.com No
Symantec/VeriSign EVIntl-crl.verisign.com No
DigiCert crl3.digicert.com No
StartSSL crl.startssl.com No
TrustWave crl.securetrust.com No
TrustCenter crl.ix.tcclass3.tcuniversal-i.trustcenter.de No
GoDaddy crl.godaddy.com No
Entrust crl.entrust.net No
Certum crl.certum.pl No

* IPv6 Capability was checked via http://ipv6-test.com/validate.php

 

OCSP repositories and IPv6
CA Host Supports V6
GlobalSign ocsp.globalsign.com Yes
COMODO ocsp.comodoca.com No
GoDaddy ocsp.godaddy.com No
DigiCert ocsp.digicert.com No
StartCom ocsp.startssl.com No
TrustCenter ocsp.tcuniversal-I.trustcenter.de No
TrustWave ocsp.trustwave.com No
Enstrust ocsp.entrust.net No
Symantec/GeoTrust ocsp.geotrust.com No
Symantec/VeriSign ocsp.verisign.com No
CyberTrust sureseries-ocsp.cybertrust.ne.jp No
Certum ocsp.certum.pl No

* IPv6 Capability was checked via http://ipv6-test.com/validate.php

 

CRL Size
CA CRL Size # Entries
Certum http://crl.certum.pl/evca.crl 862 7
Symantec/Geotrust http://EVSSL-crl.geotrust.com/crls/gtextvalca.crl 720 12
Cybertrust http://sureseries-crl.cybertrust.ne.jp/SureServer/2021_ev/cdp.crl 1742 17
GlobalSign http://crl.globalsign.com/gs/gsextendvalg2.crl 1114 18
TrustWave http://crl.securetrust.com/STCA.crl 2158 65
StartSSL http://crl.startssl.com/crt4-crl.crl 2339 90
DigiCert http://crl3.digicert.com/evca1-g1.crl 5393 100
GoDaddy http://crl.godaddy.com/gds3-37.crl 6848 153
Comodo http://crl.comodoca.com/COMODOExtendedValidationSecureServerCA.crl 12959 351
TrustCenter http://crl.ix.tcclass3.tcuniversal-i.trustcenter.de/crl/v2/tc_class3_L1_CA_IX.crl 16138 407
Entrust http://crl.entrust.net/level1e.crl 40995 1005
Symantec/VeriSign http://EVIntl-crl.verisign.com/EVIntl2006.crl 207922 5972
* CRL count was determined using “curl url>crl & openssl crl -in crl -inform der -text | grep -c “Serial”
* Size was determined using the Content-Length header, “curl –verbose –url *”

 

OCSP response Size
CA Host Size
Comodo ocsp.comodoca.com 471
Digicert ocsp.digicert.com 471
GlobalSign ocsp2.globalsign.com 1491
CyberTrust sureseries-ocsp.cybertrust.ne.jp 1588
StartSSL ocsp.startssl.com 1596
Symantec/Verisign ocsp.verisign.com 1739
GoDaddy ocsp.godaddy.com 1923
Entrust ocsp.entrust.net 1939
Certum ocsp.certum.pl 2113
Trustwave ocsp.trustwave.com 2519
TrustCenter ocsp.tcuniversal-I.trustcenter.de 3135
Symantec/Geotrust ocsp.geotrust.com 3346

* Size was determined using the Content-Length header, “curl –verbose –url *”

 

OCSP Response Performance
CA Host Average (ms)
GlobalSign ocsp2.globalsign.com

95

DigiCert ocsp.digicert.com

156

GoDaddy ocsp.godaddy.com

163

COMODO ocsp.comodoca.com

202

StartCom ocsp.startssl.com

280

Enstrust ocsp.entrust.net

293

Symantec/GeoTrust ocsp.geotrust.com

293

Symantec/VeriSign ocsp.verisign.com

295

TrustWave ocsp.trustwave.com

297

TrustCenter ocsp.tcuniversal-I.trustcenter.de

389

CyberTrust sureseries-ocsp.cybertrust.ne.jp

421

Certum ocsp.certum.pl

568

* Measuring with Monitis points in US-MID, US-EST-US-WST, Canada, Denmark, Netherlands, Germany, Spain, UK, Sweden, China, Austrailia, Signapore, Japan were used.

 

CRL Repository Performance
CA Host Average (ms)
GlobalSign crl.globalsign.com 101
DigiCert crl3.digicert.com 102
Entrust crl.entrust.net 120
Comodo crl.comodoca.com 178
GoDaddy crl.godaddy.com 186
StartCom crl.startssl.com 267
TrustWave crl.securetrust.com 272
Symantec/GeoTrust EVSSL-crl.geotrust.com 298
Symantec/Verisign EVIntl-crl.verisign.com 298
TrustCenter crl.ix.tcclass3.tcuniversal-i.trustcenter.de 371
Certum crl.certum.pl 426
CyberTrust sureseries-crl.cybertrust.ne.jp 454
* Measuring with Monitis points in US-MID, US-EST-US-WST, Canada, Denmark, Netherlands, Germany, Spain, UK, Sweden, China, Austrailia, Signapore, Japan were used.

 

A revised look at the new Windows Update SSL configuration

The other day I did a quick post about how SSL was configure for Windows Update, I thought I would double back and see if they have made any changes and it looks like they have.

From looking at the SSL Labs results I notice a few changes:

  1. The servers are now indicating a cipher suite preference.
  2. The servers are now putting TLS suites above the SSL suites.
  3. The servers no longer support TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA.
Since they made some changes since the post we can assume the goal was to improve things, I think they have but it doesn’t entirely make sense to me the choices they have made, for example:
  1. Why continue to support SSL 2.0 when no client that only supports SSL 2.0 has the WU client software on it.
  2. Why include more than one SSL cipher suite, if the goal is to enable the use of SSL 3.0 and the only clients to this server are based on SCHANNEL and CryptoAPI a single suite would be enough.
  3. If your going to support the RC4 cipher suites, why not prioritize them above the other suites so that they are resistant to BEAST.
  4. Why remove TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, I am guessing its performance but that is surprising since its not that much more expensive.

We also see a few changes when browsing to https://www.update.microsoft.com, unlike before when we browse from a client that doesn’t have a WU agent available on it (like XP /w IE6 and no SP) we no longer get instructions on how to get patched we now get a blank page.

Anyhow thats what I see today.

Ryan

Setting HTTP headers with OpenSSL and the OCSP test client

So I have been doing some OCSP performance bench-marking the last few weeks, one of the things we noticed while doing this is that OpenSSL does not set the HOST header when sending requests.

This is a problem when interacting with a OCSP responder that is on a shared host (a virtual host), without it the web-server doesn’t know which instance to send the request to. You can work around this by using the “-header” command, for example:

openssl ocsp -noverify -no_nonce -respout ocspglobalsignca.resp -reqout ocspglobalsignca.req -issuer globalsigng2.cer -cert globalsign.com.cer -url “http://ocsp2.globalsign.com/gsextendvalg2” -header “HOST'” “ocsp2.globalsign.com”

You need to ensure the HOST header is set consistently with the host contained in the -url switch but it works.

Ryan