How well-intentioned automation traps people in frustrating loops, and what we can do to stop it.

My wife is from Belarus. On one of my first visits there, I had my first real exposure to what extreme bureaucracy looked like.

Each time I visited a new city, if I stayed more than a certain number of days, I had to register with the police. The process could take an entire day and involved going to a bank to deposit money into the police branch’s account, then returning with a receipt.

One time, we tried to withdraw the remaining cash and close a bank account. We spent the entire day waiting in line after line, at one bank location after another. In the end, we gave up because the opportunity cost was greater than the amount of money we were trying to reclaim.

Need to pay for passport photos? You could not pay in cash, I assume due to fear of fraud and graft; you had to go to the bank, transfer money to the photo shop, and bring back a receipt to prove it.

What struck me was that I was the only one who found this painful. Everyone else accepted it as normal. Endless lines, paperwork, and procedural steps that seemed arbitrary and counterproductive.

So why am I writing about this? This morning I was reflecting on recent experiences changing flights and helping my parents with their Comcast subscription. Over and over, I ran into automation that was supposed to make things easier but actually made things worse.

I tried to change a flight from London to Seattle on Delta. Since it was a codeshare with Virgin, the website couldn’t handle it. I called the support line and got pushed through a phone tree that aggressively tried to send me back to the website. The site still didn’t work. I called back and asked to speak with someone and was routed to a virtual assistant that did nothing but run keyword searches on the help site. Eventually, I got connected to a lower-tier agent who told me the $350 fare difference I saw online wasn’t correct and that it would be $3,000. I pushed back until I reached someone who could actually help. They made the change. The entire process took nearly two hours.

Then there was Comcast. My aging parents have been living with me, and I’ve been helping with their bills. I noticed their TV and internet service had crept up to $350 per month. It was the result of expired deals, supposed discounts that added phone lines they never used, and a long list of tactics designed to get people to pay more for services they didn’t want. Fixing it took well over an hour, and once again I had to fight through automation before talking to someone who would do anything.

Not all automation becomes bureaucracy. My USAA mobile app lets me deposit checks instantly, transfer money in two taps, and reach a human agent with a button press when something goes wrong. It avoids the bureaucracy trap because it was designed around what I actually need to do, with seamless escalation when the automation isn’t enough.

There’s a saying that comes to mind: don’t attribute to malice what can be explained by ignorance. The people who built these systems were probably trying to help. But they were judged by what they shipped and time saved on support calls, not by whether their systems improved user experience.

So what does this mean? When we build systems like these, we need to start by deciding how we will define and measure success. That question should come at the beginning, not the end. It needs to shape how the system is designed, not just how it is reported.

Too often, we optimize for metrics that are easy to measure, like time on call or tickets closed, rather than the experience we’re actually trying to create. Instead, consider measuring success by how empowered customers feel, not how fast they hang up. Once the system is live, we have to come back and test our assumptions. That means checking whether it actually helps users, not just whether it saves time or reduces support volume. One way to do this is to regularly review customer satisfaction and compare it to the experience we intended to create. If it isn’t working, we need to change how the system behaves and what we measure.

This is especially important as we start building with AI. These systems can develop unexpected behaviors. Take Air Canada’s chatbot, which confidently told a customer he could buy a full-price ticket to his grandmother’s funeral and apply for a bereavement discount within 90 days after travel. This was completely wrong. When the customer tried to get the promised refund, the airline refused and even argued the chatbot was a “separate legal entity responsible for its own actions.” Unlike a phone tree that just frustrates you, the AI gave authoritative-sounding but fabricated policy information. The airline probably measured success by how many conversations the AI handled without escalating to humans, not realizing that customers who got wrong answers often just give up rather than keep fighting.

What we choose to measure and how fast we respond when something goes wrong matters more than ever. Once these systems are deployed, they don’t just carry our assumptions forward. They reinforce them. They make it harder to see when the original design was flawed, because the automation itself becomes the norm.

The goal should always be to reduce friction and make life easier for real people, not just to make things more efficient for the teams who built the system. The best systems I’ve used made it easy to talk to a human when I needed to, and didn’t treat automation as a wall to hide behind.

If we lose sight of that, we risk recreating the same kind of bureaucracy I saw years ago, only now it will be faster, more rigid, and much harder to argue with.

I’ve been building with computer vision and ML since before it was cool, and I use these tools daily. When my middle child announced they were majoring in computer engineering, I didn’t panic about automation taking their job. I encouraged it.

But something strange is happening in the world of software development that very few seem to be talking about. AI has created a paradox: building software has never been more accessible, yet the risks of ownership have never been higher.

This pattern isn’t isolated to software development. I’ve been tracking similar dynamics across compliance, skill markets, and organizational structures. In each domain, AI is creating the same fundamental shift: execution becomes liquid while orchestration becomes critical. The specific risks change, but the underlying forces remain consistent.

When Building Feels Free

Aaron Levie from Box recently laid out the case for where AI-generated software makes sense. His argument is nuanced and grounded in Geoffrey Moore’s framework of “core” versus “context” activities. Most people don’t actually want to build custom software, he argues, because they’re fine with what already works and “the effort to customize is greater than the ROI they’d experience.” Taking responsibility for managing what you build is “often not worth it” since when something breaks, “you’re on your own to figure out what happened and fix it.”

More fundamentally, custom development is simply less efficient for most business problems. Context activities like payroll, IT tickets, and basic workflows are things “you have to do just to run your organization, but a customer never really notices.” You want to spend minimal time maintaining these systems because no matter how well you execute them, customers rarely see the difference.

The real opportunity, Levie argues, lies elsewhere – integration work between existing systems, custom optimizations on top of standard platforms for organizational edge cases, and addressing the long tail of core business needs that have never been properly served. The result will be “an order of magnitude more software in the world,” but only where the ROI justifies customization.

Levie’s right, but he’s missing something crucial. AI isn’t just making certain types of custom development more viable. It’s fundamentally changing what ownership means. The same technology that makes building feel effortless is simultaneously making you more liable for what your systems do, while making evaluation nearly impossible for non-experts.

When Your Code Becomes Your Contract

Air Canada learned this the hard way when their chatbot promised a bereavement discount that didn’t exist in their actual policy. When their customer tried to claim it, Air Canada argued the chatbot was a “separate legal entity” responsible for its own actions.

The Canadian tribunal’s response was swift and unforgiving. They called Air Canada’s defense “remarkable” and ordered them to pay. The message was clear: you own what you deploy, regardless of how it was created.

This isn’t just a one-off, global regulations are tightening the screws on software accountability across every jurisdiction. The EU’s NIS2 directive creates real liability for cybersecurity incidents, with fines up to $10.8 million (€10 million) or 2% of global turnover. SEC rules now require public companies to disclose material incidents within four business days. GDPR has already demonstrated how quickly software liability can scale. Meta faced a $1.3 billion (€1.2 billion) fine, and Amazon got hit with $806 million (€746 million).

While these are not all AI examples, one thing is clear. When your AI system makes a promise, you’re bound by it. When it makes a mistake that costs someone money, that’s your liability. The technical complexity of building software has decreased, but the legal complexity of owning it has exploded into a ticking time bomb.

AI’s Hidden Danger

Here’s where the paradox gets dangerous. The same AI that makes building feel effortless makes evaluation nearly impossible for non-experts. How do you test a marketing analytics system if you don’t understand attribution modeling? How do you validate an HR screening tool if you can’t recognize algorithmic bias?

This creates what I call “automation asymmetry” – the same dynamic I’ve observed in compliance and audit workflows. AI empowers the builders to create sophisticated, polished systems while leaving evaluators struggling with manual review processes. The gap between what AI can help you build and what human judgment can effectively assess is widening rapidly.

As a product leader, I constantly weigh whether we can create enough value to justify engineering, opportunity costs, and maintenance costs. AI makes this calculation treacherous. A feature that takes a day to build with AI might create months of hidden maintenance burden that I can’t predict upfront. The speed of development is now disconnected from the cost of ownership.

Unlike traditional software bugs that fail obviously, AI systems can exhibit “specification gaming.” They appear to work perfectly while learning fundamentally wrong patterns.

This is Goodhart’s Law in action. When a measure becomes a target, it ceases to be a good measure. The AI system learns to optimize for your evaluation criteria rather than the real-world performance you actually care about.

Picture an HR screening system that correctly identifies qualified candidates in testing but starts filtering out good applicants based on subtle biases in the training data in the foundation model you built on. This isn’t a bug you can catch with normal testing. It requires understanding algorithmic bias that most organizations lack.

Or consider data leakage risks. AI systems can inadvertently memorize and leak personal information from their training data, but detecting this requires privacy testing that most organizations never think to perform. By the time you discover your customer service bot is occasionally revealing other users’ details, you’re facing GDPR violations and broken customer trust.

Imagine a bank that “validates” its loan applications analysis by testing on the same templates used for training. They celebrate when it passes these tests, not understanding that this proves nothing about real-world performance. Or consider a logistics company that builds an AI route optimization system. It works perfectly in testing, reducing fuel costs by 15%. But after deployment, it makes decisions that look efficient on paper while ignoring practical realities. It routes through construction zones, sends drivers to nonexistent addresses, and optimizes for distance while ignoring peak traffic patterns.

Many ownership challenges plague all custom software development. Technical debt, security risks, staff turnover, and integration brittleness. But AI makes evaluating these risks much harder while making development feel deceptively simple. Traditional software obviously fails. AI software can fail silently and catastrophically.

Why Even Unlimited Resources Fail

Want proof that ownership is the real challenge? Look at government websites. These organizations have essentially unlimited budgets, can hire the best contractors, and have national security imperatives. They still can’t keep basic digital infrastructure running.

The Social Security Administration’s technical support runs Monday through Friday, 7:30 AM to 4:00 PM Eastern. For a website. In 2025. Login.gov schedules multi-hour maintenance windows for essential services. Georgetown Law Library tracked government URLs from 2007 and watched half of them die by 2013. Healthcare.gov cost $2.1 billion and barely worked at launch.

These aren’t technical failures. They’re ownership failures. Many government projects falter because they’re handed off to contractors, leaving no one truly accountable yet the agencies remain liable for what gets deployed. The same organizations that can build nuclear weapons and land rovers on Mars can’t keep websites running reliably, precisely because ownership responsibilities can’t be outsourced even when development is.

“But wait,” you might think, “commercial software companies fail too. What about when vendors go bankrupt and leave customers stranded?”

This objection actually proves the point. When Theranos collapsed, their enterprise customers lost the service but weren’t held liable for fraudulent blood tests. When a SaaS company fails, customers face transition costs and data loss, but they don’t inherit responsibility for what the software did during operation.

Compare that to custom system failure. When your AI medical analysis tool makes a misdiagnosis, you don’t just lose the service. You’re liable for the harm it caused. The failure modes are fundamentally different.

If even professional software companies with dedicated teams and specialized expertise sometimes fail catastrophically, what makes a non-software organization think they can manage those same risks more effectively? If unlimited resources can’t solve the ownership problem, what makes us think AI-generated code will?

Traditional ownership costs haven’t disappeared. They’ve become economically untenable for most organizations. Technical debt still compounds. People still leave, taking institutional knowledge with them. Security vulnerabilities still emerge. Integration points still break when external services change their APIs.

AI makes this trap seductive because initial development feels almost free. But you haven’t eliminated ownership costs. You’ve deferred them while adding unpredictable behavior to manage.

Consider a typical scenario: imagine a marketing agency that builds a custom client reporting system using AI to generate insights from campaign data. It works flawlessly for months until an API change breaks everything. With the original developer gone, they spend weeks and thousands of dollars getting a contractor to understand the AI-generated code well enough to fix it.

These businesses thought they were buying software. They were actually signing up to become software companies.

The New Decision Framework

This transformation demands a fundamental shift in how we think about build versus buy decisions. The core question is no longer about execution capability; it’s about orchestration capacity. Can you design, evaluate, and govern these systems responsibly over the long term?

You should build custom software when the capability creates genuine competitive differentiation, when you have the institutional expertise to properly evaluate and maintain the system, when long-term ownership costs are justified by strategic value, and when existing solutions genuinely don’t address your specific needs.

You should buy commercial software when the functionality is context work that customers don’t notice, when you lack domain expertise to properly validate the system’s outputs, when ownership responsibilities exceed what you can realistically handle, or when proven solutions already exist with institutional backing.

Commercial software providers aren’t just offering risk transfer. They’re developing structural advantages that individual companies can’t match. Salesforce can justify employing full-time specialists in GDPR, SOX, HIPAA, and emerging AI regulations because those costs spread across 150,000+ customers. A 50-person consulting firm faces the same regulatory requirements but can’t justify even a part-time compliance role.

This reflects Conway’s Law in reverse: instead of organizations shipping their org chart, the most successful software companies are designing their org charts around the complexities of responsible software ownership.

Mastering the Paradox

The AI revolution isn’t killing software development companies it’s fundamentally changing what ownership means and repricing the entire market. Building has become easier, but being responsible for what you build has become exponentially harder.

This follows the same pattern I’ve tracked across domains: AI creates automation asymmetry, where execution capabilities become liquid while orchestration and evaluation remain stubbornly complex. Whether in compliance audits, skill markets, or software ownership, the organizations that thrive are those that recognize this shift and invest in orchestration capacity rather than just execution capability.

Advanced AI development tools will eventually solve some of these challenges with better validation frameworks and automated maintenance capabilities. We’ll likely see agentic AI automating much of our monotonous security, support, and maintenance work in the future also. These systems could help organizations build the connective tissue they lack automated monitoring, intelligent debugging, self-updating documentation, and predictive maintenance. But we’re not there yet, and even future tools will require expertise to use effectively.

This doesn’t mean you should never build custom software. It means you need to think differently about what you’re signing up for. Every line of AI-generated code comes with a lifetime warranty that you have to honor.

The question isn’t whether AI can help you build something faster and cheaper. It’s whether you can afford to own it responsibly in a world where software liability is real, evaluation is harder, and the consequences of getting it wrong are higher than ever.

Understanding this paradox is crucial for anyone making build-versus-buy decisions in the AI era. The tools are more powerful than ever, but mastering this new reality means embracing orchestration over execution. Those who recognize this shift and build the institutional capacity to govern AI systems responsibly will define the next wave of competitive advantage.

You’re not just building software. You’re signing up for a lifetime of accountability.

Part of the “AI Skill Liquidity” series

Early in my career, I was a security engineer, in this role we approached problems with a distinctive mindset. We look at a new system and immediately start threat modeling. What could go wrong? Where are the vulnerabilities? How might an attacker exploit this? Then we’d systematically build defenses, design monitoring systems, and create incident response procedures.

Later at Microsoft, I realized that good lawyers operate almost identically. They’re security engineers for text. When reviewing a contract, they’re threat modeling potential disputes. When structuring a transaction, they’re identifying legal vulnerabilities and designing defenses. When arguing a case, they’re building systems to withstand attack from opposing counsel. Of course, not all legal work requires this depth of analysis but the most valuable does.

This realization first drew me to Washington State’s legal apprenticeship program. The idea of learning legal “security engineering” through hands-on mentorship rather than accumulating law school debt appealed to me. I never pursued it, but I remained fascinated by sophisticated legal reasoning, regularly listening to Advisory Opinions and other legal podcasts where excellent legal minds dissect complex problems.

Just as I’ve written about skill liquidity transforming compliance and software development, the same forces are reshaping legal practice. AI is injecting liquidity into what was once an extremely illiquid skill market, where expertise was scarce, slowly accumulated, and tightly guarded. But here’s what’s different, current legal AI treats lawyers like document processors when they’re actually security engineers, and this fundamental misunderstanding is creating unexpected problems.

The Security Engineer’s Dilemma

Legal skills have historically been among the most illiquid in professional markets, exactly the kind of expertise scarcity that AI disruption targets. A single fabricated case citation can end careers. Imagine a security engineer whose monitoring system had a 99% accuracy rate, but the 1% false negatives included critical breaches that were mishandled so badly that the company’s reputation was irreparably damaged. Reputation operates on decades-long timescales because clients hire lawyers based on their track record of successfully defending against legal threats.

Just as I described in software development, AI injecting liquidity into skill markets creates a repricing rather than elimination. Where legal expertise was once scarce and slowly accumulated, AI makes certain capabilities more accessible while reshaping where the real value lies.

Traditional legal training worked like security engineer mentorship. Junior lawyers learned threat modeling by working on real cases under senior guidance. They’d review contracts while learning to spot potential vulnerabilities, draft briefs while understanding how opposing counsel might attack their arguments, structure deals while considering regulatory risks. Quality control and knowledge transfer happened simultaneously, seniors reviewing junior work would catch errors while teaching systematic risk assessment.

AI is disrupting this model in ways that would terrify any security team lead. Document review, research, and drafting that once provided junior lawyers with hands-on threat modeling experience are being automated. The tasks that taught pattern recognition, learning to spot the subtle contract clause that creates liability exposure, recognizing the factual detail that undermines a legal argument, are disappearing.

This creates the same middle tier squeeze I explored in software development, acute pressure between increasingly capable juniors and hyper-productive seniors. Junior lawyers become more capable with AI assistance while partners extend their span of control through AI tools, leaving mid-level associates caught in a compressed middle where their traditional role as the “throughput engine” gets automated away.

Here’s the economic problem, when AI saves 20 hours on document review, partners face a choice between investing those hours in unpaid training or billing them elsewhere. The math strongly favors billing. Fixed-fee arrangements make this worse, junior lawyers become cost centers rather than revenue generators during their learning phase.

The Governance Crisis

Current legal AI focuses on document creation, research assistance, and drafting support, essentially better word processors and search engines. While impressive, they’ve created a governance burden that’s consuming the time that should be spent teaching threat modeling skills.

This mirrors what I wrote about in compliance “automation asymmetry” where sophisticated AI-generated documents overwhelm human review capacity. Just as automated compliance artifacts can mask underlying issues with perfect formatting, AI legal documents create a veneer of competence that requires more, not less, senior oversight.

Every AI-generated document requires human verification. Partners must create AI policies, review procedures, and verification systems. The American Bar Association found lawyer concerns about AI accuracy increased from 58% in 2023 to 75% in 2025, exposure to these tools has heightened rather than reduced professional anxiety. Firms now maintain an average of 18 different AI solutions, each requiring its own governance framework.

This is like asking security engineers to spend their time verifying automated log reports instead of designing security architectures. Quality control has shifted from collaborative threat assessment to bureaucratic oversight. Instead of senior lawyers working through junior analysis while explaining legal reasoning, we have senior lawyers checking AI output for fabricated cases and subtle errors.

The teaching moments are gone. The efficient combination of quality control and knowledge transfer that characterized traditional review has been broken into separate activities. Senior expertise gets consumed by managing AI rather than developing human threat modeling capabilities.

There’s a deeper concern too. Security engineers know that over-reliance on automated tools can weaken situational awareness and pattern recognition. Legal reasoning requires the same kind of layered understanding, knowing not just what the law says, but how different doctrines interact, how factual variations affect outcomes, how strategic considerations shape arguments. AI can provide correct answers without fostering the threat modeling instincts that distinguish excellent lawyers.

The problem isn’t that we have AI in legal practice, it’s that we have the wrong kind of AI.

Building Better Security Tools

The fundamental problem is architectural. Current legal AI treats legal work as document processing when it’s actually systematic threat assessment. Most legal AI focuses on output, drafting contracts, researching case law, generating briefs. This misses the intellectual core, the systematic risk analysis frameworks that constitute legal reasoning.

Good lawyers, like security engineers, rely on systematic approaches. Constitutional analysis follows specific threat models for government overreach. Contract law has systematic frameworks for identifying formation vulnerabilities, performance risks, and breach scenarios. Tort analysis uses systematic negligence assessment patterns. These frameworks require internalization through guided practice that current AI disrupts.

But imagine different AI, tools designed for threat modeling rather than document creation. Instead of generating contract language, AI that helps identify potential vulnerabilities in proposed terms. Instead of researching cases, AI that systematically maps the legal threat landscape for a particular situation. Instead of drafting briefs, AI that helps build comprehensive defensive arguments while teaching the reasoning patterns that make them effective.

This would change governance entirely. Instead of verifying AI-generated content, lawyers would verify AI-enhanced threat assessments. Systems that show their analytical work (explaining why certain contract clauses create liability exposure, how different factual scenarios affect legal outcomes) enable both quality control and learning.

Security engineers don’t just need better log parsing tools; they need better threat modeling frameworks. Lawyers face the same challenge. The 19th-century apprenticeship model worked because it focused on developing systematic risk assessment capabilities through guided practice with real threats.

The Pattern Continues

This completes a progression I’ve traced across professions, and the pattern is remarkably consistent. In software development, execution capabilities are being liquified, but systems architects who understand complex threat models gain value. In compliance, process expertise is being liquified, but systematic thinkers who can model regulatory interactions across domains have advantages.

In law, legal reasoning itself could be liquified, but the outcome depends on whether we develop AI that enhances threat modeling capabilities rather than just automating document production. The sophisticated legal analysis I love hearing on Advisory Opinions represents systematic risk assessment applied to complex problems. This is exactly the kind of security engineering thinking that creates real value.

The pattern across all three domains is clear, as AI makes execution more liquid, value shifts toward orchestration. In software, orchestrators who build AI-augmented workflows and internal platforms create structural advantages. In compliance, orchestrators who design intelligent systems for continuous assurance gain leverage over reactive, manual approaches.

Current legal AI accidentally creates governance overhead that eliminates mentorship. But reasoning-focused AI could enhance both efficiency and competence development. Instead of making lawyers better document processors, we could make them better security engineers, orchestrators who design systematic threat assessment capabilities rather than just executors who handle individual risks.

The choice matters because society depends on lawyers who can systematically identify legal threats and build robust defenses. Current AI accidentally undermines this by turning lawyers into document reviewers instead of security architects.

The Washington apprenticeship program that first caught my attention represents something important, learning through guided practice with real threats rather than theoretical study. The future may not eliminate apprenticeship but transform it, the first generation learning legal threat modeling through AI designed to build rather than replace systematic reasoning capabilities.

When I listen to Advisory Opinions, I’m hearing security engineers for text working through complex threat assessments. That’s the kind of thinking we need more of, not less. We can build AI that enhances rather than replaces it.