The Deployment Delusion: How Software Learned to Stop Worrying and Love the Bug

2025-12-08

What if the greatest innovation in modern software development—Continuous Deployment—is actually a licence to ship broken code?

On July 19, 2024, a single software update from CrowdStrike took down 8.5 million Windows machines worldwide. Airlines grounded flights. Hospitals cancelled surgeries. Banks locked their doors. The fix required someone to physically visit each affected machine, boot it into safe mode, and delete the faulty file. Think about that: in 2024, we had to dispatch technicians to manually repair millions of computers because a security company pushed bad code to production.

This wasn’t a sophisticated attack or an unforeseeable edge case. This was a routine update. And the response from the industry? Not horror. Not a fundamental reckoning with our practices. Just a shrug and “well, that’s software.”

We’ve convinced ourselves that perpetual patching is progress, that GitHub activity is health, that shipping fast and fixing later is pragmatic. But here’s the thing: we’re the only engineering discipline that has normalised shipping defective products to customers and calling it “agile.”

The Hardware Heresy

Consider how Intel ships a processor. Before a chip leaves the fab, it’s been through months of validation. Design verification. Electrical testing. Thermal analysis. Every possible failure mode explored and eliminated. When Intel discovers a bug—even a subtle one that affects only specific workloads—it’s a crisis. The Pentium FDIV bug in 1994 cost Intel $475 million and damaged their reputation for years. It affected one in nine billion divisions.

Now imagine if Intel operated like we do in software. They ship the processor. You install it in your computer. A few weeks later, an Intel technician shows up at your door, removes the CPU, and solders in a new revision. Your computer behaves differently now. Some programs are faster. Others crash. And by the way, they’ve also replaced your QWERTY keyboard with DVORAK layout. Get used to it.

Absurd, right? Yet this is precisely what happens every time npm updates a dependency, every time a library changes its API, every time your phone installs an overnight update that breaks your alarm clock app, every time an app redesign moves the button you’ve clicked a thousand times to somewhere completely different.

Hardware engineers strive for zero defects in the field. Zero. Not “acceptable defect rate” or “ship now, patch later.” Zero. Because in hardware, you can’t download a fix. The economics forced discipline.

Software has no such forcing function. We can deploy hourly. So we do. And in doing so, we’ve created a culture where shipping broken code is the norm and expecting quality is quaint.

The Backwards Metric

Look at any software project’s health metrics. GitHub activity. Commit frequency. Pull requests merged. We’ve turned constant churn into a virtue. A library that hasn’t been updated in six months is “abandoned” or “unmaintained.” Never mind that it might be complete, correct, and require no changes.

We’re so accustomed to buggy software that we can’t even imagine the alternative. Zero activity should mean “nothing to repair.” Instead, it means “probably abandoned, definitely don’t use it.”

This is a profound inversion of engineering values. In every other discipline, stability is the goal. You don’t want your bridge constantly updated. You don’t want your electrical wiring “improved” every sprint. You want them designed right, built right, and left alone.

The React ecosystem perfectly embodies this pathology. React 18 changed how effects work. React 19 changes server components. React Router v6 broke everyone’s routing. Each change comes with blog posts about “migration strategies” and “breaking changes” and “deprecation warnings.” The documentation has entire sections dedicated to upgrading from previous versions. This is treated as normal, even innovative. “Move fast and break things” as a business strategy became “move fast and break your users’ code” as an engineering practice.

The Node.js left-pad incident of 2016 revealed just how fragile this edifice is. An 11-line package—literally eleven lines of code—was unpublished from npm, and it broke thousands of projects worldwide. Build systems failed. Deployments halted. Not because left-pad was complicated or critical. Because we’ve built a towering Jenga stack of dependencies, each one subject to being yanked or updated at any moment.

Continuous Deployment: The Great Enabler

Continuous Deployment wasn’t always possible. It wasn’t even desirable. When software shipped on discs, when updates meant mailing floppies to customers, you had to get it right. The cost of failure was measured in manufacturing runs, shipping manifests, customer support calls, and reputation damage. Quality wasn’t optional.

But here’s what happened: the moment we could deploy continuously, we stopped asking whether we should. The ability became the justification. Why spend three months in QA when you can ship now and patch tomorrow? Why invest in robust design when users will report the bugs for free?

We’ve turned our paying customers into beta testers and convinced them it’s a feature. “Cloud-based” doesn’t just mean runs on someone else’s computer—it means they can change your tools while you’re using them. “Always up-to-date” sounds nice until you realise it means “never stable.”

Continuous Deployment is not a neutral practice. It’s a philosophy that fundamentally redefines what shipping software means. Under the old model, shipping was a commitment: this works, use it, trust it. Under Continuous Deployment, shipping is a hypothesis: this probably works, let us know if it doesn’t.

The problem isn’t automation or iteration. The problem is that Continuous Deployment severed the feedback loop between quality and consequence. When you can fix bugs in production without anyone noticing, why prevent them? When you can A/B test in production, why invest in design? When you can roll back a deployment in minutes, why not ship experimental code to everyone?

The Cottage Industry Problem

Software development remains trapped in a pre-industrial mindset. Everyone does a bit of everything. You’re expected to write code, review code, design systems, manage deployments, monitor production, handle incidents, write documentation, and occasionally talk to users. The full-stack developer isn’t a specialisation—it’s the baseline expectation.

This would be like asking an architect to also do structural engineering, plumbing, electrical work, and brick laying. No other mature engineering discipline works this way. They discovered centuries ago that specialisation enables quality. An architect produces blueprints. A structural engineer validates them. A contractor builds from them. Each has clear boundaries, clear responsibilities, and clear accountability.

We have no equivalent. Our “architecture documents” are aspirational fiction. Our “design reviews” happen after the code is written. Our “specifications” are user stories scribbled on index cards. We’ve mistaken informality for agility and craftsmanship for process.

The path forward requires us to develop genuine disciplinary boundaries. Not artificial silos or rigid bureaucracy, but actual specialisation grounded in distinct skills and accountability. What would software architecture look like if architects couldn’t implement? What would implementation look like if implementers were working from precise specifications? What would testing look like if testers had authority to reject inadequate designs before any code was written?

Here’s the perverse thing: we claim to iterate, but our tools discourage genuine iteration. What if we iterated a design before we shipped it? Our current programming workflow encourages calcification—the first design that comes to mind gets validated and running, certified as “provably correct,” locked in by automated tests and type checkers. That’s Waterfall dressed up in Agile clothing. Real iteration means exploring alternatives, questioning assumptions, refining interfaces before committing to implementation. But we’ve confused “ship fast” with “think fast,” and the result is neither agile nor considered.

We won’t know what’s possible until we try. And we won’t try until we admit that the cottage industry model doesn’t scale.

The Simplicity Paradox

Here’s a question: what makes LEGO blocks simple?

It’s not that they’re easy to manufacture—injection moulding requires precision tooling. It’s not that they’re mathematically elegant—the brick geometry is pragmatic, not beautiful. LEGO blocks are simple because their interface lacks nuance. A 2x4 brick connects to another 2x4 brick exactly one way. You don’t need to read documentation. You don’t need to understand internal structure. You just snap them together.

The connecting principle is this: a simple outer type for connection, arbitrary complexity inside. The interface between LEGO blocks is dead simple—just studs and tubes.

Unix pipes work the same way. The connection mechanism is trivial: text streams flowing through file descriptors. Each process can be arbitrarily complex internally, but the interface between processes is just bytes. Pipes show us how to achieve LEGO blocks for software components—use queues instead of function calls for inter-unit connection. You can use functions and shared stacks inside software units, but use queues between software units. The interface is simple; the implementations can be rich.

Spreadsheets too. The connection between cells is dead simple—just values and formulas. What happens inside a cell can be a complex calculation, but adjacent cells don’t need to know about it. They just see the result.

Can we do this better today? Yes. Just about every modern programming language gives us a way to define queue classes and anonymous functions and closures. The tools are there. We just keep choosing tight coupling over loose interfaces.

Now compare this to a modern software library. Take React’s useEffect hook. The signature is useEffect(fn, deps). Simple, right? Except fn can be synchronous or async (but async is discouraged). And deps can be empty (run once), undefined (run always), or an array (run when dependencies change). Oh, and if you return a function from fn, that’s a cleanup function. And effects run after rendering but before painting, unless you need useLayoutEffect instead. And—you get the idea.

This isn’t a criticism of React specifically. It’s an observation about how we’ve defined simplicity. Simplicity means “lack of nuance.” For programmers, simplicity means strong typing, explicit parameterisation, and rich semantics. For users, simplicity means not needing to know how code works and a minimal, consistent, never-changing UX.

A dentist using computer software wants to schedule appointments, not reason about state management. A spreadsheet user wants to sum a column, not understand functional programming. These are completely different definitions of “simplicity.” Both are valid. Both are necessary. But we’ve optimised exclusively for the programmer’s definition while ignoring the user’s.

The tragedy is that we know how to build systems with genuinely simple interfaces. Spreadsheets are LEGO blocks for data. Unix pipes are LEGO blocks for text processing. HTML was a LEGO block for documents—until we turned it into a programming platform.

Every time we create a genuinely simple tool, we immediately begin complicating it. Not because users demand complexity, but because programmers find simplicity intellectually unsatisfying. We add types. We add abstractions. We add configuration options. We turn LEGO blocks into software libraries with nuanced APIs and rich semantics.

Simplicity is not just one thing. Simplicity is in the eyes of the beholder. Until we accept that programmers and users need different kinds of simplicity—that we’ve optimised for the programmers’ convenience while ignoring the users’ idea of simplicity—we’ll keep building cathedrals for developers while users stumble around looking for a light switch.

Marketing Disasters as Triumphs

On July 4, 1997, the Mars Pathfinder lander stopped responding. After days of debugging from Earth, engineers discovered a priority inversion bug: a low-priority thread held a semaphore that a high-priority thread needed, causing the system to reset. They uploaded a patch—to Mars—that enabled priority inheritance in VxWorks, and Pathfinder continued its mission.

This is taught in computer science courses as a triumph of engineering. Look how clever we were! We debugged a system on another planet! We solved a subtle concurrency problem remotely!

But here’s what nobody asks: why was the bug there in the first place? This wasn’t an unknown failure mode. Priority inversion was documented in 1990. VxWorks supported priority inheritance—it just wasn’t enabled by default. The testing didn’t catch it because the bug only manifested under specific timing conditions that didn’t occur until Martian operations.

A hundred million dollar mission nearly failed because of a configuration error and inadequate testing. And we celebrate it.

This pattern repeats constantly. The Boeing 737 MAX MCAS software killed 346 people, but the conversation focused on “lessons learned” and “improved processes” rather than fundamental accountability. Knight Capital lost $440 million in 45 minutes from a deployment mistake, and the post-mortem emphasized “better DevOps practices” rather than questioning why critical trading systems could be broken by routine updates.

We’ve become remarkably skilled at reframing failures as learning opportunities. Every disaster spawns blog posts, conference talks, and consulting practices teaching you how to avoid that specific mistake. But we never question the underlying assumption:.that shipping broken software is acceptable and managing its consequences isn’t really engineering.

What Would Zero Look Like?

Fast forward to 2025. Continuous Deployment is ubiquitous. Microservices proliferate. The average web page loads megabytes of JavaScript to display kilobytes of content. Electron apps consume gigabytes of RAM. Your phone’s battery drains faster after every update. The software keeps getting bigger, slower, and buggier.

But what if we wanted zero bugs? Not “acceptable defect rate.” Not “four nines of reliability.” Zero.

Not because perfection is achievable—hardware engineers don’t achieve perfect chips, they achieve acceptably rare defects. We do test the code before shipping, but verifying code against programmers’ intentions is not enough. We have discouraged the idea of internally iterating a design until we approve of it by inserting too many anti-design roadblocks: static compilation, strong type checking, excessive detail, strong coupling via functional methods, loss of REPL flexibility, banishing eval() even during development, banishing spontaneity and brainstorming. Proving code to be correct is not enough. We need to prove designs relative to users’ needs, then automatically—or manually with provenance—derive efficient code from the design document.

But striving for zero forces different trade-offs. It forces you to design defensively. To test exhaustively. To validate completely. To ship only when ready, not when the sprint ends.

The question isn’t whether zero bugs is possible. The question is whether we’re willing to pay the cost of approaching it. And right now, we’re not. We’ve chosen speed over quality, iteration over correctness, perpetual patching over considered design.

This wasn’t inevitable. It was a choice. Continuous Deployment made that choice possible, and economic pressure made it rational. When your competitor ships weekly and you ship quarterly, you lose market share. When users expect constant updates, stability looks like stagnation.

We changed the deployment model. Then we pointed at user expectations and said “see, we have to ship fast.” But we created those expectations. We trained users to accept buggy software. We normalized the idea that “there will be an update” is an acceptable answer to “this doesn’t work.”

The path forward isn’t to abandon Continuous Deployment entirely—the ability to fix critical security issues quickly is valuable. But we need to stop pretending that the ability to deploy continuously means we should deploy continuously. We need to recouple quality and consequence. We need to make shipping broken code expensive again, not in punitive ways, but in reputational ways.

When software that hasn’t been updated in six months is seen as stable rather than abandoned, we’ll know we’ve made progress. When GitHub activity is low because the code is correct rather than neglected, we’ll know something has shifted. When users expect software to work rather than expect software to be “in beta forever,” we’ll know the culture has changed.

Until then, we’ll keep dispatching technicians to manually fix millions of computers, one safe mode boot at a time. And we’ll keep calling it innovation.

† The Pentium FDIV bug affected specific floating-point division operations. Intel initially downplayed it, then offered replacements after public outcry. The incident established that even rare bugs in widely deployed hardware require recalls.

‡ The left-pad package padded strings with spaces or zeros. When its author unpublished it, thousands of projects broke because they depended on it—often through multiple layers of dependencies they weren’t aware of.

§ Priority inversion occurs when a high-priority task waits for a resource held by a low-priority task, while a medium-priority task preempts the low-priority task. Priority inheritance temporarily elevates the low-priority task’s priority to prevent this deadlock.

See Also

Email: ptcomputingsimplicity@gmail.com

Substack: paultarvydas.substack.com

Videos: https://www.youtube.com/@programmingsimplicity2980

Discord: https://discord.gg/65YZUh6Jpq

Leanpub: [WIP] https://leanpub.com/u/paul-tarvydas

Twitter: @paul_tarvydas

Bluesky: @paultarvydas.bsky.social

Mastodon: @paultarvydas

(earlier) Blog: guitarvydas.github.io

References: https://guitarvydas.github.io/2024/01/06/References.html