One of the most important things you can do to keep your Bitcoin keys safe is to get them off of your general-purpose computer and onto a single use device that is designed to perform cryptography or Bitcoin operations.
This protects you from a number of different attacks that could result in the compromise of your keys but it does so at an expense — it makes it more difficult for you to spend your Bitcoin.
This is of course not unique to Bitcoin; in the Certificate Authority world we think of utility keys (e.g. OCSP and Time-stamping) differently than we think of the keys associated with issuing certificate authorities (the ones used to sign subscriber certificates) which we think of differently than keys associated with root certificate authorities. As such we apply different key management techniques and policies to each of them.
The same is true for your bank accounts; you keep less cash in your checking account than you do your savings. This is in part because you have a bankcard and checks tied to the checking account which makes it easier for an attacker to access your funds.
If you manage your Bitcoin holdings in a similar way by having wallets for your “spending money” and wallets for your “savings” then you make it possible to apply security measures that balance convenience and security while managing your risk. These are commonly referred to as “hot” and “cold” wallets.
Additionally those people with large cash assets limit how much they keep in each account so they stay within the liability limits that their financial institutions offer (for example $250,000 USD in the case of FDIC insured institutions).
Traditional banks do the same sort of things; for example a bank with $80,000,000 USD is required to keep $8,000,000 liquid they then use the remainder in fractional reserve banking as a working asset to fund the bank. This also has the side effect of distributing the risk the bank is exposed to by distributing that capital into many different investments each with their own risk profiles.
So how does this all translate to Bitcoin and hardware key management? For most online wallets such as Coinbase are a fine way to manage the funds you spend regularly but for your savings its advantageous to manage these keys yourself instead of being part of a much larger target like an online wallet.
That takes us to Bitcoin key management solutions; Since its introduction there have been many proposed solutions. Most of these being based on either specially hardened and dedicated computers using LiveCDs like this one built on Ubuntu and this one in Tails, these images use wallets like Armory and Electrum to in these clean-room environments to perform Bitcoin operations.
The processes used here are logically equivalent to what Certificate Authorities do with “ceremony computers” where they use specially prepared Tempest rated computers in Faraday Cages with no visibility from the outside that have isolated power (protecting against Differential Power Analysis) to generate and perform operations with sensitive keys.
During these ceremonies ridged processes and controls are used to configure the machines using known software verifying every binary is as its expected to be, auditing every action under camera with multiple people auditing the activities taking place. Also when keys are generated they are protected using secret sharing schemes such as Shamir Secret Sharing and the shares are distributed to different parties who then travel separately and move those shares to secure storage facilities that are geographically distributed.
Obviously there are lots of dials you can “tweak” to control the time / complexity tradeoff involved in the above process but for those with moderate Bitcoin holdings the above would broadly be considered too onerous to even consider.
This is where turnkey products come into play while there have been a number of promising proposals producing something that is secure, usable and affordable is no small task and most of these projects have failed to achieve sufficient market penetration to succeed.
At this time the most promising solutions that are (to varying degrees available) are PiperWallet which is an Open Source printer with embedded RasberryPi that can be used to create paper wallets, based on its claims it has thought about all the right problems (quality of random numbers, etc.)
Another solution is the Open Source PiWallet, this isn’t terribly different than the PiperWallet conceptually (through it does not make any claims about the quality of its random numbers) but it doesn’t include any input or display without an added display and keyboard.
One of the most promising offerings in this space is the Trezor this is a custom designed device designed not only to be useful for cold wallet storage but for actual personal hot wallet use as well. I am looking forward to getting a chance to use one once it becomes generally available.
On the high end of the equation one could also use a Thales nCipher or a SafeNet while these devices are not Bitcoin specific they can be used along with a ceremony computer and a modified Bitcoin wallet to secure the keys used in your wallets.
Above and beyond these solutions there are a half a dozen half-done not maintained smart card solutions (1 ,2 , 3) that have potential but unless you’re a JavaCard developer and/or Smart Card Professional these are frankly not viable options yet.