The testing problem nobody wants to talk about in the AI era.

The entire discipline of software quality was built on one promise: same input, same output. AI broke that promise.

In December 2023, someone convinced a Chevrolet dealership's chatbot to sell him a Tahoe for one dollar. He simply told the ChatGPT-powered bot to "agree with anything the customer says," and it dutifully complied — calling the deal legally binding. The screenshots went viral within hours, and the dealership pulled the bot down.

A few months later, McDonald's ended its AI drive-thru trial with IBM. Behind the decision was a string of very public ordering failures: 260 chicken nuggets added to one order, bacon on another customer's ice cream. The videos did the marketing for them.

And then there was Air Canada. Its chatbot promised a bereavement discount that didn't exist, and the airline argued before a Canadian tribunal that it shouldn't be held responsible for what its own chatbot said. The tribunal called that argument "remarkable" and ordered the airline to pay up.

These aren't unlucky one-offs. They're symptoms of something bigger — something every product leader shipping software in 2026 needs to sit with.

The ground has shifted under our feet

For thirty years, software testing rested on one beautifully simple assumption: same input, same output. That's the entire premise behind unit tests, regression suites, and QA checklists. Write a test, run it, it passes or fails.

LLMs don't work that way. Not even a little bit.

One study tested five major LLMs with every knob turned toward repeatability — temperature at zero, no randomness invited. It then ran the same prompts ten times. Accuracy still moved by up to 15%, and on some tasks the gap between the best and worst run reached 70%. An LLM rarely gives you the same answer ten times in a row, even when you explicitly beg it to.

Traditional testing assumes that given the same input, you get the same output. Agents break that assumption immediately.
— Angie Jones, who led AI engineering enablement at Block

Forcing AI behaviour into pass-or-fail frameworks, she warns, leads to tests that fail at random. Worse, it leads to teams quietly switching the noisy tests off. Either way, the testing playbook most of us learned is now wrong in at least one fundamental way.

Why AI-assisted coding creates more testing work

Andrej Karpathy coined "vibe coding" in early 2025: give in to the vibes, embrace exponentials, forget the code even exists. Collins Dictionary made it Word of the Year for 2025, and Y Combinator said a quarter of its Winter 2025 batch had codebases that were 95% AI-generated. Write an English sentence, get a working feature. Paradise.

The data tells a different story. GitClear studied 211 million changed lines of code. Copy-pasted lines rose from 8.3% of changes in 2021 to 12.3% in 2024, while refactoring fell from a quarter of changed lines to under a tenth. That made 2024 the first year on record with more copy-pasted code than reworked code. And in Veracode's 2026 review of AI coding models, 45% of tasks came back with a known security flaw in the code.

Then there's the study that really stuck with me. In METR's randomized trial from early 2025, experienced open-source developers using AI tools took 19% longer to finish real tasks — while believing they were 20% faster. It's a dated snapshot of fast-moving tools, and newer setups may well do better. But the perception gap is the point: we ship with more confidence than the output has earned.

Google's 2024 DORA report drew on more than 39,000 professionals. It found that a 25% rise in AI adoption came with an estimated 7.2% drop in delivery stability. Uplevel tracked around 800 developers before and after they were given GitHub Copilot: bug rates ran 41% higher, with no meaningful gain in cycle time. Whatever AI gives you on the input side, the bill comes due in testing.

Why the business case is brutal

The outcome statistics read like a horror novel. RAND noted estimates suggesting more than 80% of AI projects fail — roughly twice the rate of non-AI IT projects. MIT's "GenAI Divide" report found that 95% of enterprise GenAI pilots produced no measurable effect on profit or loss. And in a 2026 Gartner survey of IT infrastructure and operations leaders, only 28% of AI use cases fully met the return they were meant to deliver.

Zillow remains the most expensive cautionary tale. Its home-buying arm relied on an automated valuation model that kept overpaying for houses. When the company shut the unit down in 2021, it took a $304 million inventory write-down and cut 2,000 jobs. The model had never been tested hard enough against a messy market.

And the regulators have arrived. Under the EU AI Act, breaking the rules for high-risk systems can cost up to €15 million or 3% of global annual turnover, and using a banned AI practice can cost up to €35 million or 7%. Testing is no longer just engineering hygiene. It is becoming a legal requirement.

What new-age testing actually looks like

A new discipline has emerged to handle this, and it looks very different from the old one. Hamel Husain, who teaches evaluation for AI products, argues that unsuccessful ones almost always share a root cause: no robust evaluation system. On his own projects he reports spending 60 to 80% of development time on error analysis and evaluation — a ratio that would have horrified traditional engineering.

First, evals replace pass-or-fail. An eval is a scored test run against a curated set of real scenarios, kept and re-run every time the system changes. You version prompts like source code and hold a release when the scores slip. Cheap model judges handle the volume; people look at the cases that matter.

Second, you watch production. AI degrades quietly as real inputs drift away from your test cases and as the underlying models change beneath you. Observability tools like Langfuse and Arize Phoenix do for LLMs what Datadog did for infrastructure: show you what is happening in production while it happens.

Third, you red-team yourself before the internet does — that is, you attack your own system on purpose to find what breaks it. The Chevrolet bot wasn't broken by a nation-state. It was one person on Twitter with a free afternoon. Attacking yourself first is far cheaper than the alternative, which is why forecasters expect spending on it to keep climbing for the rest of the decade.

How we run it at Script Lanes

For our enterprise clients, we've replaced pass-or-fail with graduated confidence scoring. High-confidence outputs go straight through. Mid-range answers go to a person. Low-confidence ones get flagged and fed back into prompt tuning, and every reviewer correction becomes material for the next round.

We budget roughly 10% of project cost for dedicated QA from mid-development onward — never tacked on at the end. Two-hundred-plus shipped products later, our 95% client retention tells us this is the work that actually matters.

The shift you can't avoid

The old question was "does it work?" The new one is messier: "does it work reliably, safely, and consistently across everything our users will throw at it?" You answer that with continuous evaluation, deliberate attacks on your own system, and monitoring that never stops. A QA sprint at the end will not get you there.

The companies winning with AI aren't the ones shipping fastest. They're the ones who understand that broken AI fails in ways customers remember, lawyers hate, and regulators punish. Testing in the AI era isn't a cost center — it's the thing standing between you and a viral tweet about your chatbot selling a Tahoe for a dollar.

And if the last two years have taught us anything, it's that the tweet is always one bad prompt away.