Writing · · 5 min read
Your incident dashboard is lying to you
Most operational reporting is a confident summary of inconsistent data. Here is the ten-minute test that tells you whether yours is.
Every operations leader I have met has a dashboard. Most of them have a category on that dashboard called Other, and in a surprising number of cases Other is the largest slice.
That is not a reporting problem. It is the reporting working correctly, faithfully summarizing data that was never consistent to begin with.
The categorization happens at the worst possible moment
Think about when an incident gets classified. Someone is on a call, the phones are lighting up, and a form requires a category before the ticket can be saved. The person filling it in has the least information they will ever have about this incident — they are perhaps four minutes into something that will take six hours to understand — and they have precisely zero incentive to get the taxonomy right. They have an incentive to make the form go away.
So they pick the first plausible option. Or Other. And that value, chosen in the first four minutes by someone under pressure, is never revisited after the incident is understood. It becomes permanent. It goes into the dashboard. It gets presented to executives as insight.
Three things compound it:
Free-text titles. "MyChart down," "Patients can't log in," "Portal issue," and "LOGIN OUTAGE — URGENT" are four descriptions of one event. Any analysis that groups by title treats them as four unrelated problems.
Taxonomy drift. A category list written three years ago has since been interpreted differently by six teams. Nobody agrees on where "authentication" ends and "identity" begins, so the boundary moves depending on who is typing. Neither team is wrong. The taxonomy simply never said.
Classification at open, not at close. The single highest-leverage fix, and almost nobody does it, because it requires going back to a ticket after the adrenaline has worn off.
The ten-minute test
Do not take my word for any of this. Run this instead:
- Pull your last fifty resolved incidents.
- Strip the existing category.
- Have two experienced people categorize them independently, using your current taxonomy, without conferring.
- Measure how often they agree.
If agreement is above 85 percent, your data is sound and you should trust your dashboard. Between 70 and 85, you have drift worth correcting. Below 70 percent, your operational reporting is not measuring your operation. It is measuring how your staff feel about a dropdown menu.
Do not assume your team is the exception. That category was chosen in the first four minutes, under pressure, by someone whose incentive was to save the ticket rather than to be precise. Given how it was produced, high agreement would be the accident. Drift is the default.
Standardize before you analyze
The instinct after a bad result is to buy better analytics. This is exactly backwards, and it is expensive. Sophistication applied to inconsistent data produces confident wrong answers faster. You do not have an analysis problem. You have a normalization problem, and it sits upstream of every dashboard you own.
What actually works, in order:
Normalize retroactively before you change any process. Take twelve months of historical tickets and re-classify them against a taxonomy you trust. This is unglamorous and it is the whole job. It also immediately tells you what your real top five issues are, which is usually not what the dashboard said.
Move classification to close, not open. Let the person who resolved the incident — the only person who actually knows what it was — set the final category. Treat the open-time category as a routing hint with a short shelf life, not a fact.
Write the taxonomy down with boundary cases. Not a list of category names. A document that says explicitly: a failed login caused by an expired certificate is Infrastructure, not Authentication, and here is why. Ambiguity in a taxonomy is not resolved by good intentions. It is resolved by examples.
Measure agreement, not just volume. Re-run the fifty-ticket test quarterly. It is the only metric that tells you whether the rest of your metrics mean anything.
Where AI is genuinely useful, and where it isn't
This is the part most organizations get inverted. The popular first AI project in operations is a chatbot that answers user questions. The valuable first AI project is normalization: reading a messy free-text ticket title and proposing a consistent, structured classification against a taxonomy you define.
That task is narrow, checkable, and enormously valuable. A human can audit a hundred of its suggestions in twenty minutes and know precisely how much to trust it. It has a clear right answer. It produces a compounding asset — a clean historical record — rather than a novelty.
It is also deeply boring, which is why it gets skipped in favor of the chatbot, and why the chatbot then gets built on top of data nobody has cleaned.
The order matters. Standardize, then analyze, then automate. Reversing that sequence is how organizations end up with an expensive AI layer producing articulate summaries of numbers that were never true.
If you run the fifty-ticket test, I would genuinely like to know what number you got.