We’ve hit the phase where companies are deploying AI in production, but still can’t answer one basic question: Who’s accountable when it goes wrong?
We’ve seen real-world failures. A model trained on historical spending data underflags patients of color for preventive care. A recommendation engine overorders inventory by $2 million because it misread seasonal trends. A loan application gets denied for a reason nobody can explain, and the customer is threatening legal action.
And every time, the finger-pointing starts. Is it on the data science team? The business? The vendor? Compliance? Who owns the decision the model made?
If your answer is “we’ll figure it out when it happens,” you’re not ready.
I had a version of this conversation last month with a head of risk at a large regional bank. They’d deployed a credit decisioning model six months earlier. It was performing well by most measures. Default rates were stable, approval rates were up slightly, processing time was down.
Then they got a complaint from a small business owner whose loan application was rejected. Nothing unusual there. What was unusual: the business owner was persistent. He wanted to know exactly why his application was denied. Not a generic explanation. The actual reason.
They couldn’t tell him.
The model had flagged something in his business cash flow patterns that correlated with elevated risk. But nobody could explain what that something was or why it mattered. The data science team could point to the features the model weighted heavily. The business team could explain the general risk framework. But nobody could connect the dots in a way that made sense to a human being.
Legal got involved. Compliance got nervous. The business owner’s attorney started using phrases like “algorithmic discrimination” and “arbitrary and capricious.”
The bank settled. Not because the model was wrong, but because they couldn’t defend the decision it made. The accountability structure wasn’t clear. Nobody knew whose job it was to be able to explain the model’s reasoning in plain language.
That’s the problem we’re seeing everywhere. AI gets treated like it’s different from other business systems. Special. Complicated. Technical. So accountability gets left vague with the assumption that smart people will figure it out.
But AI decisions have consequences. Real ones. They approve loans or deny them. They flag transactions as fraudulent or let them through. They decide which insurance claims get paid immediately and which ones get reviewed by humans. They influence medical diagnoses, hiring decisions, parole recommendations.
If you can’t clearly articulate who’s accountable for those decisions, you’re not governing AI. You’re gambling with it.
AI governance isn’t a formality. It’s operational discipline. If you don’t know who owns the output, who monitors for drift, or who escalates when things break, you don’t have governance. You have risk accumulating quietly in production.
The clients who get this right treat accountability like a precondition. It’s defined before deployment, documented clearly, and aligned across legal, risk, and business. They don’t deploy a model until everyone agrees on who owns what happens next.
Here’s what that looks like in practice. A property and casualty insurer we worked with built a claims triage model. Before it went live, they established explicit accountability:
The data science team owned model performance. If accuracy dropped below defined thresholds, they owned investigating why and proposing fixes.
The claims operations team owned business outcomes. If the model was creating operational problems, customer complaints, or unexpected costs, they owned escalating that and deciding whether to adjust model parameters or pull it back.
The legal and compliance team owned regulatory exposure. They defined what kinds of decisions the model was allowed to make autonomously and which ones required human review. They owned the explanation framework for customers who questioned decisions.
The IT operations team owned infrastructure and monitoring. If the model failed technically, went down, or stopped processing claims, they owned detecting that and restoring service.
Clear lines. No ambiguity. Everyone knew their role before the first claim got processed.
And here’s what happened: three months after launch, the model started flagging an unusual number of water damage claims as high-risk. The operations team noticed customer complaints were up. They escalated to data science. Turned out the model was reacting to an unseasonably wet spring. It was trained on historical patterns and hadn’t seen this kind of weather event before.
Not a model failure. An edge case that required judgment.
Because accountability was clear, the issue got identified fast, escalated to the right people, and resolved before it became a major problem. They adjusted the model parameters for weather-related claims and added monitoring for similar anomalies.
That’s what good governance looks like. Not preventing every issue, but having the structure in place to catch issues fast and respond decisively.
Now contrast that with what I see more often. Model gets built. It passes validation. Legal signs off on a memo. It gets deployed. Everyone assumes someone else is watching it.
Six months later, performance has degraded 15% and nobody noticed because nobody was checking. Or the model is making decisions that technically comply with policy but feel wrong to customers, and nobody knows whose job it is to care about that. Or there’s a regulatory inquiry and three different teams all thought someone else was handling compliance documentation.
That’s not governance. That’s hope.
AI doesn’t remove accountability. It multiplies the need for it. Every decision an AI system makes is a decision your organization is responsible for. The fact that a model made it instead of a human doesn’t change that.
So before you deploy your next AI model, answer these questions:
Who owns monitoring model performance in production? Not just during the pilot. Every day, every week, for as long as that model is making decisions.
Who owns investigating when something goes wrong? When accuracy drops, when customer complaints spike, when a decision doesn’t make sense—who gets the call?
Who owns explaining the model’s decisions? To customers, to regulators, to your own executives when they ask uncomfortable questions.
Who owns deciding when to pull a model back? When do you stop trusting it? Who has the authority to make that call?
Who owns keeping the model current? Data drifts. Business conditions change. Who’s responsible for knowing when the model needs retraining?
If you can’t answer all five questions with names and clear responsibilities, you’re not ready to deploy.
And here’s the hard part: this can’t be one person. AI accountability requires cross-functional alignment. Technical, business, legal, and risk all have legitimate stakes. The failure mode isn’t when everyone’s involved. It’s when nobody clearly owns the final call.
The best governance structures I’ve seen assign a primary owner for each accountability area, but with clear escalation paths and communication protocols. Data science might own performance monitoring, but they escalate to business operations when intervention is needed. Legal might own regulatory compliance, but they coordinate with data science on documentation.
It’s a matrix, but it’s a clear matrix. Everyone knows their lane and how to work across lanes when needed.
This is harder than building the model. It requires organizational discipline. It requires executives who understand that deploying AI isn’t just a technical milestone, it’s an operational commitment.
But it’s not optional. Not anymore. The regulatory environment is tightening. Customer expectations are rising. The cost of getting this wrong is climbing.
So treat accountability like what it is: the foundation of responsible AI deployment. Build it first. Make it explicit. Test it before you need it.
Because when something goes wrong—and eventually something will—the question won’t be whether your model was sophisticated enough. It’ll be whether anyone was actually in charge.
