How well-intentioned automation traps people in frustrating loops, and what we can do to stop it.

My wife is from Belarus. On one of my first visits there, I had my first real exposure to what extreme bureaucracy looked like.

Each time I visited a new city, if I stayed more than a certain number of days, I had to register with the police. The process could take an entire day and involved going to a bank to deposit money into the police branch’s account, then returning with a receipt.

One time, we tried to withdraw the remaining cash and close a bank account. We spent the entire day waiting in line after line, at one bank location after another. In the end, we gave up because the opportunity cost was greater than the amount of money we were trying to reclaim.

Need to pay for passport photos? You could not pay in cash, I assume due to fear of fraud and graft; you had to go to the bank, transfer money to the photo shop, and bring back a receipt to prove it.

What struck me was that I was the only one who found this painful. Everyone else accepted it as normal. Endless lines, paperwork, and procedural steps that seemed arbitrary and counterproductive.

So why am I writing about this? This morning I was reflecting on recent experiences changing flights and helping my parents with their Comcast subscription. Over and over, I ran into automation that was supposed to make things easier but actually made things worse.

I tried to change a flight from London to Seattle on Delta. Since it was a codeshare with Virgin, the website couldn’t handle it. I called the support line and got pushed through a phone tree that aggressively tried to send me back to the website. The site still didn’t work. I called back and asked to speak with someone and was routed to a virtual assistant that did nothing but run keyword searches on the help site. Eventually, I got connected to a lower-tier agent who told me the $350 fare difference I saw online wasn’t correct and that it would be $3,000. I pushed back until I reached someone who could actually help. They made the change. The entire process took nearly two hours.

Then there was Comcast. My aging parents have been living with me, and I’ve been helping with their bills. I noticed their TV and internet service had crept up to $350 per month. It was the result of expired deals, supposed discounts that added phone lines they never used, and a long list of tactics designed to get people to pay more for services they didn’t want. Fixing it took well over an hour, and once again I had to fight through automation before talking to someone who would do anything.

Not all automation becomes bureaucracy. My USAA mobile app lets me deposit checks instantly, transfer money in two taps, and reach a human agent with a button press when something goes wrong. It avoids the bureaucracy trap because it was designed around what I actually need to do, with seamless escalation when the automation isn’t enough.

There’s a saying that comes to mind: don’t attribute to malice what can be explained by ignorance. The people who built these systems were probably trying to help. But they were judged by what they shipped and time saved on support calls, not by whether their systems improved user experience.

So what does this mean? When we build systems like these, we need to start by deciding how we will define and measure success. That question should come at the beginning, not the end. It needs to shape how the system is designed, not just how it is reported.

Too often, we optimize for metrics that are easy to measure, like time on call or tickets closed, rather than the experience we’re actually trying to create. Instead, consider measuring success by how empowered customers feel, not how fast they hang up. Once the system is live, we have to come back and test our assumptions. That means checking whether it actually helps users, not just whether it saves time or reduces support volume. One way to do this is to regularly review customer satisfaction and compare it to the experience we intended to create. If it isn’t working, we need to change how the system behaves and what we measure.

This is especially important as we start building with AI. These systems can develop unexpected behaviors. Take Air Canada’s chatbot, which confidently told a customer he could buy a full-price ticket to his grandmother’s funeral and apply for a bereavement discount within 90 days after travel. This was completely wrong. When the customer tried to get the promised refund, the airline refused and even argued the chatbot was a “separate legal entity responsible for its own actions.” Unlike a phone tree that just frustrates you, the AI gave authoritative-sounding but fabricated policy information. The airline probably measured success by how many conversations the AI handled without escalating to humans, not realizing that customers who got wrong answers often just give up rather than keep fighting.

What we choose to measure and how fast we respond when something goes wrong matters more than ever. Once these systems are deployed, they don’t just carry our assumptions forward. They reinforce them. They make it harder to see when the original design was flawed, because the automation itself becomes the norm.

The goal should always be to reduce friction and make life easier for real people, not just to make things more efficient for the teams who built the system. The best systems I’ve used made it easy to talk to a human when I needed to, and didn’t treat automation as a wall to hide behind.

If we lose sight of that, we risk recreating the same kind of bureaucracy I saw years ago, only now it will be faster, more rigid, and much harder to argue with.