The Dashboard Was Telling the Truth

Every character and company in this piece is a composite, assembled from two decades of incidents, conversations with colleagues on both sides of the argument, and the rather loud debate about AI and code quality currently running on social media. Nobody in this story is a villain. The failure pattern is real, and you have probably shipped it.

Every Monday at 09:40, the same ticket arrived.

Not the same number. The same ticket. A retail partner of a mid-sized software vendor would report that a handful of order status updates from the weekend had never reached their warehouse. Some orders shipped twice. Some never shipped. A support engineer would open the logs, find event delivered: 200 OK for every one of them, and escalate.

Development would look, find nothing wrong, bump a timeout, add a log line, close the ticket. The following Monday, 09:40.

The module that was, by every measure, fine

The vendor sold an order management platform to retailers. One feature was simple: when an order changed state, notify the retailer’s other systems through a webhook. Post a JSON payload to a URL.

The code that did this was excellent by every measure the industry has taught us to apply. Clean interfaces, dependency injection, 94% test coverage. The static analysis suite showed a wall of green and a maintainability rating of A. Every merge request needed two approvals and got them. Manual QA before each release. The team lead had built the module three years earlier and had mentored everyone who touched it since. He would tell you, sincerely, that it was the most solid part of the product.

He was right, by the definition of solid he had been given. I want to spend a moment on why that definition deserved his trust, because the rest of this story does not make sense otherwise.

Why the dashboard earned its trust

Anyone who wrote software before static analysis, before mandatory reviews, before coverage gates, remembers what it was like. Formatting wars in every review. Null dereferences in production that a linter would have caught in seconds. Modules nobody dared touch because there were no tests at all. Releases that broke on Friday and stayed broken until Monday.

The conventions this team followed were the industry’s answer to that era, and they worked. They turned software delivery from a craft that depended on whoever happened to be in the room into something a business could plan around. They let teams grow, let people go on holiday, let new hires contribute in their first week. Every green badge on that dashboard represents a class of failure that used to be routine and is now rare.

So when the team lead looked at the dashboard and said “this is solid”, he was not being complacent. He was reading the instruments that had kept him and his team out of trouble for twenty years. Trusting them was the professional thing to do. It is what we have all been trained to do, and for good reason.

The trouble is only that instruments measure what they were built to measure.

The person who was not supposed to look

The vendor also had an operations engineer. He ran the servers, the network, the monitoring. He had been doing it since before anyone said “DevOps”, and had spent a lot of nights watching systems fail in ways their authors never anticipated.

A year earlier he had started working with AI coding agents on his own projects. Not the autocomplete kind. The kind you give a specification to, that drafts a plan you argue with, that produces code you read line by line. He had built a routine around it: write the spec himself, let one agent draft the plan, have a second agent from another provider challenge it, fix the disagreements, implement, review every diff, insist on comments dense enough that a junior could maintain the result. Decades of side projects he never had time for were suddenly real.

He had no standing to touch the webhook module. But the partner kept being pointed at his monitoring dashboards, and he had the kind of curiosity that does not respect org charts.

So one evening he gave an agent the webhook code and the partner’s integration contract, and instead of asking “is this code good”, he asked a different question: the partner says events are missing, the logs say they were delivered, explain how both can be true, and tell me what this implementation does not do that the contract requires.

What it did not do

The answer ran to a page. Stripped down, it was this.

The code built a payload, issued a POST, and if the call returned without throwing, logged delivered. That was the whole algorithm.

It did not retry. A connection reset or a partner-side 502 during a deploy meant one attempt and the event was gone, and depending on where the exception surfaced, the log said delivered anyway.

It did not send an idempotency key, so when the partner’s load balancer replayed a request during a failover, the warehouse received “shipped” twice and shipped twice.

It did not read the response. The partner’s contract specified that a successful delivery returns a receipt identifier the sender must store and reference in the next event for that order. The code checked for a 2xx and discarded the body.

And the tests, all 94% of them, tested a mocked client that returned 200. They verified, thoroughly, that the code did what the code did. None tested what the contract said it should do, because the contract had been read once, quickly, three years ago, under a deadline.

None of this is unusual. Most of us have written exactly this client. “Send the webhook” was the story. Nobody wrote the story for “handle the partner being down for ninety seconds”, because nobody was thinking about the partner being down. The story was done. The tests were green. The gates passed. The process worked precisely as designed.

Two definitions of done

The operations engineer opened a merge request with the missing pieces: retries with backoff, a dead-letter store, an idempotency key, receipt handling, honest logging. The description said in its first line that this was a proposal meant to help, not a demand. He had pushed back on the agent’s design in two places and had a second agent review the result before a human ever saw it.

The team lead declined it the same day.

His reasons were all reasonable. The module had passed every gate for three years. The tests were green. The people who reviewed it understood the framework. An operations engineer with an AI tool restructuring application logic was, in most companies, a sign that something had gone wrong with boundaries. And, said with real feeling: this is how AI wrecks codebases, plausible code from people who do not understand what they are changing.

It is the same argument you will find in a hundred threads this week. He made it better than most, because he had a concrete merge request to point at.

And here is the thing I did not appreciate until later: he was tired. Not of this ticket. Of the year. Every feed he opened told him his profession was ending, his methods were obsolete, his juniors would be replaced, and the people saying it loudest had shipped the least. Then a colleague from another department handed him a machine-assisted rewrite of his best module. From where he stood, the feed had walked into the office.

What was actually on the table was not AI versus craft. It was two definitions of done that had never been compared. His: compiles, lints, covered, reviewed, shipped. The other: does what the contract says, under failure, and tells the truth when it cannot. Both are legitimate. The first is the one our tools can check. The second is the one our customers experience. For twenty years, the gap between them was filled by whoever had the experience and the detachment to read the spec and the code in the same sitting, and that person was rare and busy and usually emotionally invested in the code. So the gap mostly went unfilled, and the Monday ticket arrived.

What changed, and what did not

What changed is not that a machine became smarter than the team lead. It did not. What changed is that a tireless, unsentimental reader with no sunk cost and no affection for the original design became cheap enough to put on a two-month-old support ticket. It read the contract. It read the code. It listed the gaps. Then a human with twenty-five years of watching things fail decided which gaps mattered and how to close them.

What did not change is everything the team lead’s fear was pointing at. The same agent, asked a lazier question by a person who does not read the answer, will produce a confident, well-formatted, fire-and-forget webhook client and call it done. The tool is indifferent. What made the difference was the workshop around it: who wrote the question, who challenged the plan, who read the diff, who put their name on it. That workshop is not an AI thing. It is the same discipline the team lead had been practising for two decades, pointed at a different layer.

Which is why the ending of this story is not that one of them won.

What they did next

The partner’s own integration engineer sent a four-page email a few days later that reached, independently, the same conclusions about the receipt identifier and the retries. That took the question out of the realm of opinion.

The two of them sat down. The operations engineer walked through how the proposal had been produced, the spec, the adversarial review, the parts he had overridden. The team lead walked through what was wrong with it by his team’s standards: naming, error-handling idiom, a retry strategy that would have hammered the partner under sustained outage. Both were right. The version that shipped was written by the team, in the team’s style, using the proposal as a map. The tests that got added were contract tests, against a recorded partner response, and they failed the first time they ran against the old code.

The team lead has since started asking one question at the end of design reviews, before the gates, before the coverage number: what does this not do that the person on the other end expects? Sometimes he asks it of an agent. More often he asks it of the room. The dashboards are still green. They were never wrong. They were just answering a question nobody was asking any more, and now there is a second question.

If you recognise any of this

The conventions you inherited are good. Keep them. They are doing a job that used to cost you weekends.

But if there is a ticket that keeps coming back, take the module it points at and the document that says what that module is supposed to do, and ask, in these words, what the implementation does not do that the document requires. Ask a colleague from another team. Ask an agent. Ask both and compare. Read the answer the way you would read a new hire’s first review: with respect and scepticism in equal measure.

And if someone from outside your team shows up with an answer, remember that the feed is not in the room with you. They are. The feed says your craft is obsolete. The colleague is saying your craft is exactly what is needed to turn a page of machine output into something that ships. Those are not the same message, even when they arrive the same week.