Tag Archives: Supply Chain

Accountability and Transparency in Modern Systems

Over the last several decades we have seen the rate of technological innovation greatly accelerate. A key enabler of that acceleration has been the move to cloud computing which has made it possible for hardware, software, and services to be shared. This significantly reduced both the capital and time necessary to adopt and operate the infrastructure and services built on these platforms.

This migration started by enabling existing software to run using dedicated computers and networks owned and operated by someone else. As these computers got faster and the tools to share the physical hardware and networks were built, the cost of technological innovation reduced significantly. This is what democratized modern startup entrepreneurship as it made it cost-effective for individuals and small businesses to gain access to the resources once only available to the largest companies.

This flipped technological innovation on its head. It used to be that government and big businesses were the exclusive sources of technological innovation because they were the ones who could afford to buy technology. The lowering of the cost of innovation is what gave us the consumer startups we have today. This drew the attention of large companies to this emerging market and led to the creation of the modern smartphone which was fundamental to creating the market opportunity we see in consumer startups. This was a scale opportunity that was fundamentally different than the prior government and enterprise models of innovation.

As enterprises saw the rate of innovation and agility this new model provided, it became clear that they too needed to embrace this model in their businesses. It is this reality that led to the creation of Salesforce, the first Software As a Service, and AWS, the first to market with a modern Cloud Service Provider. It was these offerings that gave us Software As a Service (SaaS), Platform As a Service (PaaS), and what we think of as modern cloud infrastructure.

At first, these enterprises only moved greenfield or very isolated projects to the cloud but as the benefits of the new model became irrefutable and the capabilities of these offerings were enriched in ways that were impractical to replicate in their environments, they started moving more business-critical offerings. We can see this trend continues today, a recent survey found that 55% of IT organizations are now looking at ways to reduce their on-premise spending. This will lead to many legacy systems being replaced with more modern, scalable, agile, and secure solutions.

That same survey found that digital transformation and security are the two biggest reasons for this shift. This is no surprise when we look at how capital efficient modern businesses are relative to those based on legacy IT and manual processes, or how vulnerable legacy IT systems are to modern attacks. 

This does beg the question, what is next?. I believe that two trends are emerging. The first being the democratization of compliance for modern systems and the second being the shift in expectations of what does it mean to be “secure”.

If we look at the first trend, the democratization of compliance, we see the internet becoming balkanized through regulation and governments seeking to get more control over what people do on the internet. Increased regulation makes it significantly harder for new entrants to compete, which in turn helps entrench the incumbents who can often eat the engineering and compliance costs associated with the regulations. When you think about this in the context of the global economy in which the internet exists, an economy made up of 195 independent sovereign countries, the compliance burden becomes untenable.

Modern Cloud Service Providers can make a significant dent in this by making it possible for those who build on them to meet many of these compliance obligations as a byproduct of adopting their platforms.

In the near term, this will likely be focused on the production of the artifacts and audit reports that are needed to meet an organization’s current compliance requirements but if we project out, it will surely evolve to include services for legal identity verification, content moderation, and other areas of regulatory oversight. A decade from now I believe we will see systems being built on these platforms in such a way that they will be continually compliant producing the artifacts necessary to pass audit as a natural byproduct of the way they work. 

This will in turn make it easier to demonstrate compliance and create new opportunities such as auditors continually monitoring an organization for its compliance with guidelines rather than just doing annual point-in-time assessments as is done today.

This has also led to companies like Coalition building offerings that let customers augment existing systems with the artifacts to demonstrate conformance with security best practices are being met so that insurance companies can offer more affordable risk-based insurance policies.

As we look at the second trend, the redefinition of what it means to be secure, we can see consumers becoming more aware of security risks and as a result, their expectations around the sovereignty of their data and the confidentiality of their information evolving. 

One response to this realization is the idea of decentralization. The thesis here is arguably is that there can be no sovereignty as long as there is centralization. In practice, most of these decentralized systems are in-fact quite centralized. While there are many examples of this, one of the more visible has been the DAO hard fork which was done to recover stolen funds or the simple fact that 65% of Bitcoin mining happens in China. Additionally, for the most part, the properties that enable sovereignty typically come from the use of verifiable data-structures and cryptography and not decentralization. That is not to say these systems do not have a place, I would argue that their success and durability so far at least suggests there is “a there, there” but I would also say that, at least currently, they do not yet live up to their full promise. 

Another response to this is the consumer adoption of End-To-End encryption in messaging applications (even iMessage is end-to-end encrypted!) and by extension to that problem the verifiability of the systems that implement these schemes. 

The best example here is probably Signal, they spent time designing security and privacy into their messaging protocol and implementing systems from the beginning, modeling its design on modern threats and decades of learning about what does, and does not work. This approach led to the protocol that they defined being adopted by many of their competitors, including WhatsApp, Facebook Messenger, Skype, and Google Allo.

Signal is also a great example of the verifiability property, in particular, the work they have done with Contact Discovery is exciting. What they have done with this feature is first to minimize what information they need to deliver the capability in the hope to limit future abuse. Secondly, they leveraged technologies like SGX, which is an example of a Confidential Compute, that enables them to demonstrate what they are doing with the information they do collect. This introduces transparency and accountability which both are important ingredients to earning trust.

The use of hardware security as a key component of the security boundary has already found its way from consumer phones, laptops, and tablets to the cloud. For example, Google Cloud‘s Shielded VMs and Azure Trusted Launch use hardware to provide verifiable integrity to VM instances to make it possible to detect VMs compromised by boot- or kernel-level malware or rootkits similar to how Apple does with the iPhone. We also now see AMD Sev and SGX seeing broader deployment in the larger Cloud Service Providers (I will be the first to admit these technologies have room to grow if they are to live up to their promises but they are promising none the less).

With this foundation, the industry is starting to look at how they can bring similar levels of transparency and accountability into applications and ecosystems too. One of the projects that have demonstrated that doing this can have a big impact is Certificate Transparency. As a result of the investments in deploying Certificate Transparency, the internet is now materially more secure than it was before and this is a direct result of introducing accountability into an opaque ecosystem based on blind trust.

Another example in this space is the Golang Checksum Database where verifiable data-structures like Merkle Trees are being used to introduce accountability into the software supply chain as a means to mitigate risks for those who rely on the Golang ecosystem. 

For many problems in the security space, you can solve from one of two philosophical bases. You can either create privileged systems only visible to a few that you hope aren’t corruptible or you can build democratizing transparency into the system as a check on corruption.

Dino A. Dai Zovi

While the earlier examples are using combinations of hardware, cryptography, and verifiable data-structures to deliver on these properties, other examples take a more humble approach. For example, Google Cloud’s Access Transparency uses privilege separation, audit logs, and workflows to provide the fundamental ability to track business justifications for access to systems and data. The existence of these systems is further validation that the trend of verifiability is emerging.

So what should you take away from this post? I suppose there are four key messages:

  1. The definition of security in modern Cloud services is continuing to be influenced by the consumer space which is leading to the concepts of verifiability, accountability, data sovereignty, and confidentiality becoming table stakes.
  2. Globalization and regulation are going to accelerate the adoption of these technologies and patterns as they will ultimately become necessary to meet regulatory expectations.
  3. Increasingly verifiable data structures, cryptography, and hardware security capabilities are being used to make all of this possible.
  4. These trends will lead to the democratization of compliance to the many regulatory schemes that exist in the world.

I believe when we look back, these trends will have significantly changed the way we build systems and a new generation of businesses will emerge enabling these shifts to take place.

Software Supply Chain Risk Mitigation

Increasingly we are seeing attacks against what is now commonly referred to as the software supply chain.

One of the more notable examples in the last few months was from the Nodejs package management ecosystem [1]. In this case, an attacker convinced the owner of a popular but unmaintained Node package to transfer ownership to them. The attacker than crafted a version of the package that unsuccessfully attacked Copay, a bitcoin wallet platform.

This is just one example of this class of attack, insider attacks of the software supply chain are also becoming more prevalent. When looking at this risk it holistically it is also important to realize that as deployments move to the Cloud the lines between software and services also blur.

Though, not specifically an example of a Cloud deployment issue, in 2015 there was a public story of how some Facebooks employees have the ability to log into users accounts without the target user’s knowledge [2]. This insider risk variant of the supply chain exists in the Cloud in a number of different areas.

Probably the most notable being in the container images provided by their Cloud provider. It is conceivable that a Cloud provider could be compelled by government to build images that would attack a specific or set of customers as part of an investigation, or that an employee would do so under compulsion or in service of personal interests.

This is not a new risk, in fact, management of internal and external dependencies has always been core to building secure systems. What has changed is that in the rush to the Cloud and Open Source users have adopted the tools and resources these cloud providers have built to make this migration easier without fully understanding and managing this risk that they have assumed in doing so.

In response to this reality, Cloud providers are starting to provide tools to help mitigate this risk, some such examples include:

  • Providing audit records of employee access to customer data and services,
  • Building solutions to provide hardware-based trusted execution environments that provide some level of protection from cloud providers.
  • Offering hardware key management solutions provided by third-parties to protect sensitive key material,
  • Cryptographically signing the binaries and images that are published so that their distribution is controlled and tampering post-production can be detected.

Despite these advancements, there is still a long way to go to mitigate these risks in a holistic fashion.

One effort in this area I am actively involved in is in the adoption of the concept of Binary Transparency. This can be thought of as an evolution of legacy code signing models. In these solutions, a publisher places a cryptographic signature using a private key associated with a public certificate of some sort that is either directly trusted based on package origin and signature (such as with GPG signatures) or is authenticated based on the legal identity of the publisher of the package (as is the case with Authenticode).

These solutions, while valuable, help you authenticate a package but they do not provide you the tools to understand the history of that package. As a result, these publishers can produce packages either accidentally or on purpose that are malicious in nature that is signed with their “trusted keys” and it is not detectable until it is too late.

As an example of this risk, you only need to look at RealTek, over the years numerous times their code signing key has been compromised and used to produce malware, some of it targeted such as in the case of Stuxnet [3].

Binary Transparency addresses this risk in a few ways. At its core Binary Transparency can be thought of as an append-only ledger listing of all versions of a given binary, each of these versions having a pointer to a content addressable store where that binary is available.

This design enables the runtime that will execute the binary to do a few things that were not possible, It can, for example, ensure it is running the most recent version of a binary and to only run the binary when it, and some number of previous revisions are publicly discoverable. This also enables the relying parties of the published binaries and images to comp it can inspect all versions and potentially diff those versions to understand the differences.

When this technique is combined with the concept of reproducible builds, as is provided by Go [4] and a community of these append-only logs and auditors of those logs you can get strong assurances that:

  • You are running the same version as everyone else,
  • That the binary you are running is reproducible from the source you can review,
  • The binary are running has not neen modified since it was published,
  • That you, and others, will not run binaries or images that have not been made publicly available for inspection.

A system with these properties disincentivizes the attacker from executing these attacks as it significantly increases the probability of being caught and helps bound the impact of any compromise.

Importantly, by doing these things, it makes it possible to increase the trust in the Cloud offering because it minimizes the amount of trust the user must put into the Cloud provider to remain honest.

A recent project that implements these concepts is the Go Module Transparency project [5] [6].

Over time we will see these same techniques applied to other areas [7] [8] of the software supply chain, and with that trend, users of open source packages, automatic update systems, and the Cloud will be able to have increased peace of mind that their external dependencies are truly delivering on their promises.


  • [1] Node.js Event-Stream Hack Exposes Supply Chain Security Risks
  • [2] Facebook Engineers Can Access Your Account Without A Password
  • [3] STUXNET Malware Targets SCADA Systems
  • [4] REPRODUCING GO BINARIES BYTE-BY-BYTE
  • [5] Proposal: Secure the Public Go Module Ecosystem
  • [6] Transparent Logs for Skeptical Clients
  • [7] Firefox Security/Binary Transparency
  • [8] Contour: A Practical System for Binary Transparency