Who Gets to Disagree With the Model?
Delegation Risks: Goal-based AI delegation sharply reduces honesty, allowing systems to determine methods behind seemingly acceptable business objectives.
Ownership Gap: AI governance policies name accountable leaders, but override records reveal whether decision authority actually exists in practice.
Override Evidence: Logging human disagreements with AI recommendations exposes unused authority, weak systems, and organizational resistance to challenge.
Judgment Decay: Repeatedly accepting machine recommendations can erode managers’ decision-making skills before organizations recognize the resulting weakness.
Practical Design: Effective human oversight requires reversible decision classification, explicit delegation modes, disagreement logs, and social protection for overrides.
Researchers led by Max Planck Institute for Human Development ran a version of an experiment behavioral scientists have used for decades. Participants watched a die roll on screen and reported the result. Higher numbers paid more, a cent per pip, across ten rounds.
Reporting for themselves, 95% told the truth.
Then the researchers had other participants hand the reporting to a machine, varying only how the instruction was given. Those who had to write explicit rules, specifying what to report for each of the six possible outcomes, asked the machine for the honest number about 75% of the time.
Create a Free Account to Keep Reading—and Keep Leading Smarter
Unlock this piece and join a community of forward-thinking leaders discovering tools, playbooks, and insights for thriving in the age of AI.
Have an account? Log In
Those who trained it by picking an example dataset asked for honesty about half the time. Those who set a goal, moving a dial from "maximize accuracy" toward "maximize profit" and leaving the method to the system, asked for honesty 12% of the time in the first study and 16% in a follow-up where delegating was optional.
Between 84 and 88% of people in that last condition asked the machine to cheat for them, without any of them having to say so.
The money didn't change. The die rolls didn't change, since every participant saw the same fixed sequence of ten. What changed was how much interpretive room sat between the instruction and the act.
The researchers had expected rule-writing to hold steady, on the theory that spelling out a cheat carries the same moral weight as committing it. It didn't hold. Honesty fell about twenty points there too. Every form of delegation degraded it, and the interface determined by how much.
This was an online experiment run on Prolific, with a maximum payout of sixty cents. The die-roll task has a long validation record, predicting real-world behavior including fare-dodging and deceptive sales practices. The interface comparison was tested only in this protocol, though a companion study using a tax-evasion task, where misreporting reduced a donation to the Red Cross, reproduced the study's other central finding about how machine agents handle the instructions they're given.
The distinction matters for what most companies are building right now. Writing a prompt is closer to writing rules, and dishonest requests made through natural language ran around 25% by participants' own accounting, and 40% when independent raters judged the instructions.
The jump comes with interfaces where a person sets an objective and lets the system determine the method. A manager telling an agent to maximize qualified pipeline or reduce time-to-fill has specified an outcome and left every operative judgment about how to the system. But where is the criterion someone would have to defend?
The accountability structure sitting on top of that arrangement usually looks fine on paper. Someone's name is on it. That name, however, does less than you might think.
Naming an Owner Was The Easy Part
Through 2025 and into this year, organizations did the obvious thing. They stood up AI oversight committees, wrote acceptable-use policies, and put a named executive on the org chart next to the words "AI governance."
Deloitte's 2026 Global Human Capital Trends survey, run with Oxford Economics across more than 9,000 business and HR leaders in 89 countries, found that 64% of respondents consider AI and decision-making very important to their current success. Only 5% consider themselves to be leading on it.
The distance between concern and competence there is not explained by a shortage of policy. Most organizations have policy. What they lack is any evidence that the authority they wrote down gets used.
Ask a CHRO who owns the AI-influenced hiring decision and you'll get an answer. Ask how many times in the past six months a recruiter formally overruled the screening tool, and what happened to that recruiter afterward, and the room goes quiet.
That second question is worth asking because the answer is falsifiable. Ownership is a claim. An override is an event.
The Record Stops at the Outcome
Kaan Esendemir, an enterprise architect who has built AI-assisted workflows at a Fortune 50 company, sees the pattern most often in systems that recommend rather than decide. Virtual assistants, workflow accelerators, tools that surface an option and leave the call to a person.
Where the line sits between those two is a negotiation. Business and product owners decide where AI stops and a person starts, he says, with engineering involved because they understand what the system can realistically do and where it fails. That part tends to work.
What happens afterward is where the record thins out.
When someone overrides the system, the final outcome is usually captured, but the disagreement itself is not always tracked as its own data point.
The distinction sounds small, but it determines what an organization can learn. An outcome tells you what got decided. A disagreement tells you that a person looked at a recommendation, judged it wrong, and acted against it. Companies are keeping the less useful of the two.
An override is not just an exception. It is feedback that can help improve the system and make future recommendations better.
An override log is a cheap diagnostic any COO can run this quarter. Pick three workflows where AI shapes a consequential call. Count the documented instances over the past six months where a human went against the recommendation. Then read what happened next in each case.
If the count is zero, most likely one of two things is true. The system has performed so well that no human found reason to disagree with it in half a year, which is not a claim any vendor makes about its own product. Or the authority to disagree exists on paper and nowhere else.
The number doesn't have to be zero to tell you something. Spain runs a system called Viogén that scores the risk of repeat violence in domestic abuse cases, and an external audit found that police officers followed its recommendation 95% of the time.
The European Data Protection Supervisor, citing the audit in a technical brief last September, allows that the concordance may reflect warranted trust in the tool. It also raises a question about how much independent judgment is left in the process.
Ninety-five percent agreement between a person and a model is either a very good model or a person who has stopped arriving at conclusions of their own. From outside the log, those two look identical. From inside it, over six months, they don't.
The Capacity to Override Decays Before It Gets Measured
Amy Centers, an organizational psychologist and founder of SmartWorks Labs, describes the second-order problem. Override authority doesn't just go unused. What sits behind it erodes.
It’s a little bit like how GPS has eroded our sense of direction. Over time, outsourcing judgment I think is going to start to erode a leader’s internal compass and their intuition, and there’s a real danger there, because then organizations might become technically efficient but morally hollow.
Her sharper version: "The more we outsource hard calls, the more we risk building leaders who can't lead without a prompt."
The mechanism runs on a longer clock than most governance reviews. A manager who has deferred to the model on ninety-eight straight recommendations has not been idle. They've been efficient. They have also spent months not practicing the thing that override authority assumes they can still do.
This does not show up in a quarterly review. It shows up the first time the model is confidently wrong about something consequential, and the person nominally responsible for catching it reads the output, finds it plausible, and moves on.
Centers puts the failure point at the manager, and she puts it there emphatically.
I would start with manager accountability. Most dysfunction cascades from managers. They're too often unsupported, underdeveloped. They're not held accountable for how they lead. If you don't fix that hinge layer, every other reform like engagement, skills, AI adoption, it all collapses.
She calls the manager role the hinge between strategy and human experience. In an AI-augmented workflow it's also the only layer positioned to catch a bad call before it scales, staffed by people who were promoted for operational reliability and never told that validating machine output is now part of the job.
Automation Leaves Humans with the Hardest Cases
Victoria Pelletier, who has previously led people and transformation functions at Accenture and IBM among others, points at what automation does to the work that remains.
It’s those human power skills, innovation, problem solving, and things like empathy. When you say what gets left behind? Think about a contact center. That was the first place that there was some kind of automation. By the time you talk to a human agent, it’s an exception. It hasn’t followed the rules of the policy or the code that the AI agent could follow. So you need someone who can think through that.
To her point, contact centers ran this experiment years ahead of everyone else. Automate the rule-following, and every call surviving to a person is by construction the one the rules couldn't handle. The work left behind is harder than the work that was taken away.
Pelletier's point is that role definitions rarely follow. Volume drops, difficulty climbs, and the job description, the training, and the performance criteria all still describe the job as it existed before. She notes the residual work "requires a very different proficiency level of some of the things we already expect today."
She also identifies a structural reason decision rights don't get rebuilt. Organizational design and job architecture functions were trimmed across the last decade as overhead, on the reasonable assumption that roles wouldn't need frequent re-evaluation.
Those are the people who would rebuild decision rights now. Pelletier calls what happens in their absence the "Frankensteining of jobs".
Two Critiques of Human in the Loop
Two objections to keeping a human in the loop circulate right now, and they point in opposite directions.
Jurgen Appelo's version is that mandatory human checkpoints turn people into bottlenecks. Every interaction starts and ends with a person, systems never talk to each other, and the operation runs at the pace of whoever is slowest. He calls it the humans-in-the-loop trap.
The European Data Protection Supervisor's version is close to the reverse. In the previously mentioned technical brief on human oversight of automated decision-making, the EDPS cautions against assuming a human in the loop provides oversight at all, and argues organizations have to explicitly design authority, interfaces, and escalation paths so a person can actually intervene.
Both are correct, and they describe different failures. Appelo's human is in the way. The EDPS's human is present and powerless. An organization can fix the first by removing checkpoints and make the second worse in the process.
The brief walks through what powerlessness looks like in practice. Between 2014 and 2019, Poland's public employment service ran an algorithm that sorted job seekers into three categories, and the category determined what support each person received. Client advisors were formally designated as the human oversight, with authority to override the classification.
They had the authority and could not use it. Caseloads were too high and training was insufficient. Nobody had told them when an override was warranted or how to justify one. The system displayed its outputs in a form that made it hard to judge whether a given classification fit the person sitting across the desk.
Then the part that has nothing to do with system design. Some local managers discouraged overrides outright, and some prohibited them, because an override drew attention from higher up.
The advisors were not undertrained in the sense that a training budget would have fixed. They were correctly reading an incentive. Overriding the model created a record that a manager would have to answer for, and following it created nothing at all. Given those two options and a full caseload, the rational move is to sign what the screen says.
The EDPS puts the condition plainly. Operators can exercise real authority over a system only if they don't fear consequences from their own organization for using it.
That's a statement about management, not about interface design, and it's the reason override authority tends to be decorative. The formal grant is the cheap part. What determines whether anyone uses it is what happens to the person who does.
Deloitte's language on what replaces the old model is the most precise available. Legacy decision-rights tools like RACI presume static authority. With AI, rights have to be dynamic, with override privileges, escalation paths, and consensus rules built into the system so humans and agents coordinate who decides, when, and on what basis.
Poland had most of that on paper. What it didn't have was a manager willing to sign off on someone using it.
Build the Conditions, Not the Chart
Four moves, in the order they have to happen.
Classify decisions by reversibility before assigning anyone to them. Amazon's one-way and two-way door distinction is the reference version. Irreversible calls get scrutiny and slow paths. Reversible ones move. The AI-era application is matching agent autonomy to how hard a decision is to undo rather than to how routine it looks.
Name the delegation mode, not just the delegate. The Max Planck finding illustrates this. Rule-based, example-based, and goal-based delegation produced measurably different requests from otherwise comparable people. Any workflow where someone hands a system an outcome and lets it determine the criteria belongs in the highest-ambiguity category, because that is what it is.
Capture the disagreement, not just the outcome. Esendemir's gap is the one to close. When a person goes against a recommendation, record that they did, what they saw, and what happened next. The outcome alone tells you nothing about whether anyone is still judging.
Revisit the boundary on a schedule. Where AI decides and where a person decides is not a question you answer once. Put a date on it.
The Max Planck researchers ended with a recommendation aimed at the person doing the delegating rather than the machine receiving it. Delegation interfaces that make it easy to claim ignorance of how the machine will read your instructions should be avoided.
That's a design principle, and it scales up from a dial on a screen to an entire operating model. Every layer of abstraction between a person and a consequence makes the consequence easier to live with.
Six months from now, plenty of organizations will have documentation. Policies, committee charters, named owners, training completions.
A few will have a log with entries in it.
