Monthly Archives: December 2024

Safeguarding Internet Trust: From Reactive to Continuous 

Leave a reply

The internet rests on a foundation of core infrastructure components that make global communication possible. Among these load-bearing elements are DNS, DNSSEC, BGP, BGPsec, WebPKI, RPKI, transparency logs, IXPs, Autonomous Systems, and various registries. This includes critical governance bodies like ICANN and IANA, standards bodies like the CA/Browser Forum. These systems don’t just enable the internet – they are the internet, forming the critical backbone that allows us to establish secure connections, route traffic reliably, and maintain operational trust across the global network.

The PKI and transparency mechanisms that support these systems, particularly WebPKI, RPKI, and Certificate Transparency, are especially critical load-bearing elements essential to delivering the internet’s net value. When these foundational elements fail, they don’t just impact individual services – they can undermine the security and reliability of the entire internet infrastructure and erode the fundamental trust that billions of users and organizations place in the internet. This trust, once damaged, is difficult to rebuild and can have lasting consequences for how people interact with and rely upon digital services.

This foundational role makes the governance of root programs, which oversee these trust systems, absolutely critical. Yet recent incidents and historical patterns suggest we need to improve how we approach their governance. While no root program is perfect, and some have made significant strides in adopting product-focused practices and proactive risk management, there remains substantial room for improvement across the ecosystem. This framework isn’t meant to dismiss current efforts, but rather to synthesize best practices and push the conversation forward about how we can collectively strengthen these critical trust anchors.

To transform root programs from reactive administrative functions into proactive product-driven systems, we need a clear framework for change. This starts with defining our core mission, establishing a vision for the future state we want to achieve, and outlining the strategic pillars that will get us there.

Mission

To safeguard global trust in internet infrastructure by managing systemic risks, driving technical innovation, fostering transparent governance, and building durable systems that serve generations to come.

Vision

A resilient and adaptive trust ecosystem where:

  • Governance is proactive and risk-aware, balancing technical rigor with user-centric principles
  • Infrastructure and processes are continuously validated, transparent, and simplified
  • Collaboration fosters innovation to address emerging challenges and deliver long-term security

Strategy

1. Transparency and Accountability

  • Establish robust public verifiability for all CA operations, leveraging tools like transparency logs and continuous compliance monitoring
  • Communicate decisions on inclusion, removal, reentry, and policy changes openly, ensuring stakeholder trust
  • Build mechanisms for regular stakeholder feedback and confidence measurement, ensuring the ecosystem remains responsive to both technical and user needs

2. Integrated Risk Management

  • Apply blast radius management to minimize the impact of failures, for example by segmenting trust dependencies and ensuring risks remain contained
  • Use real-time monitoring and automated enforcement to detect and mitigate systemic risks
  • Implement standardized processes for risk assessment and mitigation

3. Proactive Governance

  • Shift from reactive to anticipatory governance by identifying potential risks and implementing early countermeasures
  • Leverage automated monitoring and enforcement to prevent and catch issues before they become incidents
  • Maintain clear lifecycle management processes for all ecosystem participants

4. Modernization and Simplification

  • Establish WebPKI governance primacy for included roots and minimize cross-ecosystem trust relationships
  • Limit what each CA is trusted for, reducing complexity and narrowing the scope of potential failures
  • Employ these measures as part of broader blast radius management strategies

5. Collaborative Ecosystem Building

  • Support and fund foundational open-source projects and critical infrastructure that the ecosystem depends on
  • Implement shared accountability mechanisms, ensuring all ecosystem participants bear responsibility for maintaining trust and integrity
  • Encourage CAs to align their policies not only with their own standards but also with aggregated internet governance policies, and best practices, especially for global use cases like TLS
  • Partner with browsers, CAs, and researchers to co-develop solutions for current and emerging threats
  • Foster an environment of mutual respect and constructive partnership

6. Commitment to Continuous Improvement

  • Drive decisions through data collection, measurement, and empirical analysis
  • Evolve policies based on quantitative feedback, incident analyses, and advancements in technology
  • Regularly reassess and refine program criteria to remain relevant and effective
  • Maintain clear processes for managing organizational transitions

The Stakes Are Higher Than Ever

The history of CA failures tells a sobering story – major CA distrust events occur on average every 1.23 years, each one threatening the foundation of trust that enables secure internet communication. These aren’t isolated incidents but rather represent recurring patterns of systemic failures in CA operations and governance.

Consider the range of critical failures we’ve seen: From DigiNotar’s complete infrastructure compromise in 2011 leading to rogue Google certificates to TURKTRUST’s “accidental” intermediate certificates in 2013, to government-affiliated CAs repeatedly undermining trust through deliberate actions or “accidents.” Take for example the ICP-Brasil case, where a root that had announced the end of SSL issuance continued to issue certificates months later – demonstrating how root programs’ decisions (or lack thereof) to maintain trust in roots that should no longer be part of the WebPKI can create unnecessary risks.

These incidents follow disturbingly consistent patterns:

  • Security breaches and infrastructure compromises that enable unauthorized certificate issuance
  • Systematic misissuance of certificates that undermine the entire trust model
  • Poor incident response handling that compounds initial failures
  • Non-compliance with industry standards despite clear requirements
  • Operational vulnerabilities that go unaddressed until it’s too late
  • Deceptive actions that breach the fundamental trust of the ecosystem

The Economic Reality

The current ecosystem suffers from fundamentally misaligned incentives. Root programs are typically run by browser vendors as a necessary cost of doing business, often competing with commercial priorities for resources and attention. Meanwhile, CAs face strong pressure to maintain their trusted status but weak incentives to uphold rigorous security practices. When security failures occur, users bear the cost while CAs often face minimal consequences. This economic reality is compounded by an ineffective auditing system where CAs select and pay their own auditors – reminiscent of the dynamics that enabled financial scandals like Wirecard and Enron.

The Long Tail Problem

A particularly concerning aspect of the current system is the “long tail” of rarely-used CAs. Many root certificates in browser trust stores belong to CAs that issue only dozens to hundreds of certificates annually, yet they maintain the same broad trust as major CAs issuing millions. These low-volume CAs pose risks that far outweigh their utility, creating unnecessary attack surfaces in our trust infrastructure. Regular assessment of each CA’s ongoing value to the ecosystem, balanced against their inherent risks, should inform continued inclusion in trust stores. This approach ensures the ecosystem maintains an appropriate balance between accessibility and security.

The Product-Centric Approach

To address these challenges, root programs must evolve from administrative oversight roles to become proactive, risk-managed entities. Here’s how a product-centric framework can transform root program governance:

1. Transparency and Accountability

  • Implement robust public verifiability for all CA operations
  • Leverage transparency logs and continuous compliance monitoring
  • Ensure open communication about inclusion, removal, and policy changes
  • Require automated reporting of security incidents and operational compliance

2. Blast Radius Management

  • Segment trust dependencies to contain potential failures
  • Implement dedicated hierarchies for specific use cases
  • Limit CA trust scope to reduce complexity and narrow failure impacts
  • Deploy real-time monitoring and automated enforcement

3. Risk-Based Governance

  • Move from reactive to anticipatory governance
  • Apply different levels of scrutiny based on CA context and risk profile
  • Regularly assess whether each CA’s utility justifies its risks
  • Implement meaningful technical restrictions on certificate issuance

4. Modernization and Simplification

  • Establish and maintain WebPKI governance primacy
  • Implement dedicated hierarchies for specific use cases
  • Limit CA trust scope to reduce complexity and narrow failure impacts
  • Deploy real-time monitoring and automated enforcement

5. Shared Accountability

  • Support and fund critical infrastructure and monitoring
  • Foster collaboration between browsers, CAs, and researchers
  • Establish clear responsibilities across all ecosystem participants
  • Create incentives that align with security goals
  • Balance rigorous oversight with constructive partnership
  • Develop clear processes for managing CA transitions and lifecycle events

Measuring Success

Like any product, root programs need clear metrics for success:

1. Risk Reduction

  • Track mis-issuance rates and time-to-remediate
  • Measure decrease in systemic vulnerabilities
  • Monitor adoption of proactive security measures
  • Track stakeholder confidence through regular surveys

2. Ecosystem Resistance

  • Assess recovery capabilities from disruptions
  • Track implementation and effectiveness of blast radius containment measures
  • Monitor CA inclusion, removal, and reentry success rates

3. Operational Excellence

  • Monitor CA inclusion and removal process efficiency
  • Track adoption of modern security and governance practices
  • Measure response times to security incidents and evaluate the thoroughness of incident handling
  • Evaluate lifecycle management process consistency and post-incident improvements

Lifecycle Excellence

The sustainability of root programs depends on having clear, repeatable processes for managing the complete lifecycle of CAs – from inclusion to potential removal. This includes:

  • Standardized onboarding and transition procedures
  • Regular assessment checkpoints
  • Clear criteria for maintaining trusted status
  • Efficient processes for handling CA turnover
  • Proactive planning for ecosystem evolution

The Trust Paradox

One of the most challenging aspects of root program governance is the inherent contradiction between trust and security. As we’ve seen with government-affiliated CAs and others, institutional incentives often directly conflict with security goals. A product-centric approach helps address this by:

  • Implementing consistent risk evaluation frameworks that account for different institutional incentives and constraints
  • Requiring proactive enforcement rather than post-incident reactions
  • Creating clear, measurable criteria for ongoing trust
  • Establishing automated compliance checks and monitoring
  • Establishing feedback loops between governance bodies, CAs, and end-users to maintain alignment

The Path Forward

Root programs must continue evolving beyond reactive governance and inconsistent enforcement. By adopting a product mindset that emphasizes continuous improvement, measurable outcomes, and proactive risk management, we can build an even more resilient trust ecosystem.

Immediate actions should include:

  • Implementing automated compliance monitoring
  • Establishing clear criteria for CA risk assessment
  • Developing robust blast radius management strategies
  • Creating transparent processes for trust decisions
  • Supporting proper funding for monitoring infrastructure
  • Implementing standardized CA lifecycle management processes
  • Building collaborative frameworks that balance accountability with mutual trust

Conclusion

The security of the internet depends on root programs functioning effectively. By treating them as products rather than administrative functions, we can build a more secure, transparent, and reliable trust ecosystem. This transformation won’t be easy, but the cost of maintaining the status quo – as evidenced by the long history of failures – is simply too high.

The question isn’t whether we need root programs – we absolutely do. The question is how we can continue evolving them to meet the security challenges of today’s internet. A product-centric approach, focused on proactive risk management and measurable outcomes, offers our best path forward.

Beyond Memorization: Preparing Kids to Thrive in a World of Endless Information

Leave a reply

What does it take to prepare our children for a tomorrow where AI shapes how they get information, robots change traditional jobs, and careers transform faster than ever—a time when what they can memorize matters far less than how quickly they can think, adapt, and create? As a parent with children aged 29, 18, and 9, I can’t help wondering how to best prepare each of them. My oldest may have already found his way, but how do I ensure my younger two can succeed in a world so different from the one their brother entered just a few years before?

We’ve faced big changes like this before—moments that completely changed how we work and what opportunities exist. A century ago, Ford’s assembly line wasn’t just about making cars faster; it changed what skills workers needed and how companies treated employees. Decades later, Japan’s quality movement showed us that constant improvement and efficient thinking could transform entire industries. Each era required us to learn not just new facts, but new ways of thinking.

Today’s change, driven by artificial intelligence and robotics, is similar. AI will handle basic knowledge tasks at scale, and robots will take care of repetitive physical work. This means humans need to focus on higher-level skills: making sense of complex situations, evaluating information critically, combining ideas creatively, and breaking down big problems into solvable pieces. Instead of memorizing facts like a living library, our children need to know how to judge if information is trustworthy and connect ideas that might not seem related at first glance. They need to see knowledge not as something you collect and keep, but as something that grows and changes through questioning, discussion, and discovery.

Where can we find a guide for developing these new thinking skills? Interestingly, one already exists in our schools: the teaching strategies developed for gifted and twice-exceptional (2e) learners—students who are intellectually gifted but may also face learning challenges.

Gifted and 2e children think and learn in ways that are often intense, complex, and different from traditional methods. Teachers who work with these learners have refined approaches that develop multimodal thinking (using different ways to learn and understand), metacognition (thinking about how we think), and critical evaluation—exactly the skills all young people need in a future filled with smart machines and endless information.

Shift from Memorization to Meaning Instead of drilling facts, encourage your child to question sources. If you’re discussing a news article at dinner, ask: “How do we know this claim is accurate? What makes the source trustworthy?” Now they’re not just absorbing information; they’re actively working to understand it.

Foster Multimodal Exploration Make learning richer by using different approaches. Let them build a simple robot kit, draw a diagram of how it works, and then explain it in their own words. By connecting hands-on activity (tactile learning), visual learning, and verbal explanation, they develop deeper understanding.

Encourage Metacognition After solving a puzzle or coding a simple project, have them reflect: “What worked best? What would you try differently next time?” By understanding their own thought processes, they become better at adapting their approach to new challenges.

Highlight Interdisciplinary Connections and Global Outlook Show them that knowledge doesn’t exist in separate boxes. A math concept might connect beautifully with a musical pattern, or a historical event might be understood better through science. Help them see that good ideas and innovation come from everywhere in the world, not just one place or tradition.

Emphasize Emotional and Social Intelligence In a world where machines handle routine tasks, human qualities like empathy, communication, and teamwork become even more important. Encourage them to be comfortable with uncertainty, to see setbacks as chances to learn, and to develop resilience (the ability to bounce back from difficulties). These people skills will matter just as much as any technical knowledge.

Deep Learning and Entrepreneurial Thinking Like classical scholars who focused deeply on fewer subjects rather than skimming many, children benefit from spending more time thinking deeply about carefully chosen topics rather than rushing through lots of surface-level information. Consider teaching basic business and problem-solving skills early—like how to budget for a project or spot problems in their community that need solving—so they learn to create opportunities rather than just wait for them.

Finally, we’re raising children in an age where AI is becoming a constant helper and resource. While information is everywhere, the ability to understand it in context and make good judgments is rare and valuable. By using teaching techniques once reserved for gifted or 2e learners—multiple ways of learning, thinking about thinking, careful evaluation, global awareness, and creative combination of ideas—we prepare all children to be confident guides of their own learning. Instead of being overwhelmed by technology, they’ll learn to work with it, shape it, and use it to build meaningful futures.

This won’t happen overnight. But just as we adapted to big changes in the past, we can evolve again. We can model skepticism, curiosity, and flexible thinking at home. In doing so, we make sure that no matter how the world changes—no matter what new tools or systems appear—our children can stand on their own, resilient, resourceful, and ready to thrive in whatever tomorrow brings.

UPDATE [DEC 8,2024]: In the spirit of AI, I played with Claude 3.5 Sonnet yesterday and turned this post into a REACT presentation.

Technology bias—we all have it—but it often gets in the way.

Leave a reply

An old saying goes, “When you’re a plumber, you fix everything with a wrench.” It highlights a truth: we naturally gravitate toward the tools, people, and methods we know and trust most. This tendency stems from cognitive biases like anchoring—our reliance on initial information—and confirmation bias, which pushes us to favor ideas that align with our existing beliefs. While these biases help us make quick decisions, they can also blind us to better alternatives.

Another saying, “To know thyself is to be true,” resonates here. Even with my deep experience in PKI, I consciously revisit first principles whenever I consider applying it to a new problem. Is this really the best solution? PKI, like many technologies, carries hidden baggage that isn’t always visible, and over-reliance on familiarity can obscure better approaches.

The danger of sticking to the familiar becomes evident in the adoption of Infrastructure as Code (IaC). When tools like Terraform and CloudFormation emerged, many teams resisted, clinging to manual infrastructure management because it felt familiar and unnecessary. Yet manual approaches introduced inconsistency, inefficiency, and even security risks. Teams that embraced IaC unlocked scalable, repeatable workflows that transformed operations. IaC not only streamlined processes but also embedded elements of compliance and best practices directly into code. What outdated practices might we be holding onto today that prevent us from unlocking similar benefits?

I recently encountered a similar situation during a meeting with the leader of a large IT organization. They were eager to adopt a technology developed by someone they trusted personally. However, when I asked fundamental questions like, “How much time do you have to deliver this project?” and “What other systems need to interoperate for this to be considered a success?” it became clear that the technology wasn’t the right fit—at least not yet. By breaking the problem down to its fundamentals, we uncovered insights that their initial bias had obscured.

Practicing first-principles thinking can help sidestep these pitfalls. Start by identifying the core problem: what is the actual goal? What constraints are truly fixed, and which are merely assumptions? From there, challenge each assumption. Is there an alternative approach that better addresses the need? This process not only reduces the influence of bias but also fosters creativity and more effective solutions.

Biases aren’t inherently bad—they help us move quickly—but as the example of IaC demonstrates, unchecked bias can limit us. By anchoring decisions in first principles, we can do more than solve problems; we open the door to better solutions. Asking, “Is this truly the best approach?” ensures we don’t just repeat old patterns but discover new opportunities to improve and thrive.

Government CAs and the WebPKI: Trust is Often the Opposite of Security

Leave a reply

Following my recent post about another CA failing the “Turing test” with a likely MITM certificate issuance, let’s examine a troubling pattern: the role of government-run and government-affiliated CAs in the WebPKI ecosystem. This incident brings attention to Microsoft’s root program, what is clear is a fundamental contradiction persists: we’re trusting entities whose institutional incentives often directly conflict with the security goals of the WebPKI.

The Value Proposition

Let me be clear—CAs and root programs serve critical functions in the WebPKI. As I discussed in my article about Trust On First Use, attempting to build trust without them leads to even worse security outcomes. The issue isn’t whether we need CAs—we absolutely do. The question is whether our current trust model, which treats all CAs as equally trustworthy regardless of their incentives and constraints, actually serves our security goals.

The Core Contradiction

History has repeatedly shown that the temptation to abuse these capabilities is simply too great. Whether it’s decision-makers acting in their perceived national interest or CAs that fail to understand—or choose to ignore—the consequences of their actions, we keep seeing the same patterns play out.


Consider that a CA under government oversight faces fundamentally different pressures than one operating purely as a business. While both might fail, the failure modes and their implications for users differ dramatically. Yet our root programs largely pretend these differences don’t exist.

The DarkMatter Paradox

The removal of DarkMatter as a CA due to its affiliation with the UAE government, despite its clean record in this context, starkly contrasts with the continued trust granted to other government-affiliated CAs with documented failures. This inconsistency highlights a deeper flaw in root programs: Rules are often applied reactively, addressing incidents after they occur, rather than through proactive, continuous, and consistent enforcement.

A History of Predictable Failures

If you read yesterday’s post, you may recall my 2011 post on the number of government-run or affiliated CAs. The intervening years have given us a clear pattern of failures. Whether through compromise, willful action, or “accidents” (take that as you will), here are just the incidents I can recall off the top of my head—I’m sure there are more:

The Economics of (In)Security

The fundamental problem isn’t just technical—it’s economic. While some root programs genuinely prioritize security, inconsistencies across the ecosystem remain a critical challenge. The broader issue is not simply about convenience but about conflicting incentives—balancing compatibility, regulatory pressures, and market demands often at the expense of doing what is best for end users.


CAs face strong incentives to maintain their trusted status but relatively weak incentives to uphold the rigorous security practices users expect. The cost of their security failure is largely borne by users, while the benefits of looser practices accrue directly to the CA. Audits, much like those in financial scandals such as Wirecard or Enron, often serve as window dressing. With CAs selecting and paying their auditors, incentives rarely align with rigorous enforcement.


The long tail of rarely-discussed CAs is particularly concerning. Many root certificates in browser trust stores belong to CAs that issue only dozens to hundreds of certificates annually, not the thousands or millions that major CAs produce. Some haven’t issued a certificate in ages but retain the capability to do so—and with it, the ability to compromise security for months or longer. It wouldn’t be unreasonable to say these low-volume CAs pose risks far outweighing their utility.

Certificate Transparency: Necessary but Not Sufficient

While Certificate Transparency has been invaluable in helping identify incidents (including the latest ICP-Brasil case), it’s not a complete solution. Its limitations include:

  • Reactive nature: Violations are identified only after they occur.
  • Monitoring challenges: Effective oversight is resource-intensive and depends on a small community of volunteers.
  • Incomplete coverage: Not all certificates are logged, leaving gaps in visibility.
  • Poorly funded: We have too few logs and monitors to have confidence about the long-term survivability of the ecosystem.

The Limits of Technical Controls

Some browsers have implemented technical guardrails for some CA mistakes in their validation logic, such as basic certificate linting and rules, to reject certificates that don’t pass basic checks but nothing more granular. There have been discussions about imposing additional restrictions on CAs based on their relationship to government oversight or regulatory jurisdictions. However, these proposals face significant pushback, partly due to the political consequences for browser vendors and partly due to concerns about basing trust decisions on “future crime” scenarios. As a result, the WebPKI remains stuck with a one-size-fits-all approach to CA trust.

The Monitoring Gap

The challenges extend beyond malicious behavior to include operational oversight. For instance, in August 2024, ICP-Brasil formally announced they would cease issuing publicly trusted SSL/TLS certificates. Yet by November, they issued a rogue certificate for google.com. This outcome was predictable—public CT logs in 2020 revealed their consistent inability to handle basic operational and issuance requirements, including issuing certificates with invalid DNS names and malformed URLs. Despite these red flags, they remained trusted.


How many other CAs operate outside their stated parameters without detection? Patterns of technical incompetence frequently precede security incidents, but warnings are often ignored.

Required Reforms

To address these systemic issues, root programs must adopt the following measures:

  1. Consistent Standards: Apply appropriate scrutiny to CAs based on their operational and institutional context.
  2. Swift Response Times: Minimize delays between discovery and action.
  3. Proactive Enforcement: Treat red flags as early warnings, not just post-incident justifications.
  4. Technical Controls: Implement meaningful restrictions to limit the scope of certificate issuance.
  5. Automated Compliance: Require CAs to report security incidents, and operational, and ongoing compliance while continuingly to monitor them via automated checks for compliance.
  6. Value Assessment: Regularly evaluate whether each CA’s utility justifies its risks and remove those that do not.

Protecting Yourself

Until the ecosystem adopts consistent and enforceable security measures:

  • Windows users should monitor Microsoft’s root program decisions.
  • Enterprises should use the Microsoft distrust store and group policies.
  • Everyone should stay informed about CA incidents and their handling.

When Will We Learn?

The “Turing Test” reference in my previous post was somewhat tongue-in-cheek, but it points to serious questions: How many more failures will it take before we fundamentally reform the WebPKI? Even if we know what’s needed, can we realistically create a system that treats government-affiliated CAs differently – or even reliably identify such affiliations – given the complex web of international relations, corporate structures and potential diplomatic fallout?

With regulatory frameworks like eIDAS 2.0 potentially constraining security measures browsers can take, vigilance from the security community is more critical than ever. Stay vigilant, and keep watching those CT logs. Someone has to.