The Delegation Trap: AI, Cognitive Offloading, and the Erosion of Human Agency
There is a pattern emerging in how people interact with AI that mirrors something Daniel Kahneman spent decades studying in human cognition: the gravitational pull toward the path of least cognitive effort. As AI tools become more capable and more embedded in professional and personal decision-making, they are not merely augmenting human thought. In many cases, they are replacing it.
The tool itself is not the issue. AI can genuinely make people think better. The danger is in a specific failure mode that is becoming the norm: AI producing what looks like careful analysis while the human scans the output and accepts it. The machine appears to do System 2. The human does System 1. And nobody notices the inversion.
Except it is worse than an inversion. Because the machine is not actually doing System 2 either.
Kahneman's Framework and the AI Parallel
Kahneman's Thinking, Fast and Slow describes two modes of cognitive processing. System 1 is fast, automatic, effortless: the engine of intuition and snap judgement. System 2 is slow, deliberate, effortful: the engine of logic, calculation, and critical evaluation. Humans default to System 1 wherever possible. System 2 is lazy. It will accept the suggestions of System 1 unless something in the environment forces it to engage: novelty, contradiction, a sense that the stakes are high, or the simple friction of material that resists easy processing.
AI outputs have the surface characteristics of System 2 reasoning. They are structured, evidence-based, analytically framed, and articulate. They look like the product of careful thought. But large language models generate text token by token based on statistical patterns learned from training data. There is no internal process that distinguishes a hard problem from an easy one or an uncertain claim from a well-established fact. What looks like careful reasoning is sophisticated pattern matching at scale.
There is a nuance worth being honest about. Chain-of-thought prompting and extended thinking features do produce observably different outputs. More steps, more self-correction, more consideration of alternatives. But this is deeper pattern matching, not deliberation in any meaningful sense. The AI does not evaluate whether its output is correct. It produces structured outputs without truth awareness. It cannot reliably distinguish between a sound analysis and a plausible-sounding one. Those are not the same thing, and the gap between them is where the risk lives.
This reframes the entire question of human-AI interaction. If the human drops into System 1, there is no System 2 anywhere in the process. The AI has generated a plausible analysis. The human has recognised it as plausible. Recognition is not reasoning. It is pattern matching. Two layers of System 1. No System 2 anywhere. The output looks like the product of thought, but no thought occurred.
The way this should work is the human doing System 2 on the material the AI provides. The AI surfaces evidence, offers framings, identifies risks. It does this well, and fast, and that is genuinely valuable. But the critical evaluation can only come from the human. It is the only System 2 in the entire process. If the human does not supply it, it does not exist.
Where AI Genuinely Makes Thinking Better
I want to make the positive case properly, because I believe it, and because the essay would be dishonest without it.
AI is, when used with intent, the best thinking partner most people have ever had access to. For most of human history, the ability to test your reasoning against a knowledgeable counterpart was gated by access: access to education, to experts, to people who would tell you where your argument was weak. AI democratises that in a way that is genuinely significant. Not because the AI thinks critically for you, but because it gives you vastly more material to think critically about.
I use AI to stress-test my own positions regularly. Not by asking "Is this right?" which is an invitation to sycophancy, but by asking "What are the strongest arguments against this?" or "Where does this logic break down?" The AI does not evaluate my reasoning. It generates alternative perspectives drawn from the patterns in its training data, and I evaluate those. My System 2 engages with material the AI produces. That is the partnership working as it should.
I have used AI for threat modelling and it surfaced possibilities I had not thought of. That is a concrete example of AI extending the reach of domain expertise rather than replacing it. The human still has to evaluate whether the threat scenarios are realistic and relevant. The AI does not know. But the raw material it provides can be genuinely useful.
AI is also good at breaking cognitive anchors. Kahneman wrote extensively about anchoring: the tendency for an initial piece of information to disproportionately influence subsequent judgements. AI can present alternative framings quickly, not because it is wiser, but because it is not anchored to the same starting point you are. That capacity to reframe on demand used to require a diverse team or a very good mentor.
The positive case is real. But all of it requires the human to stay in System 2. The moment you drop into scanning and accepting, the partnership collapses into delegation. And because the AI cannot reliably evaluate its own output, there is nothing underneath to catch you.
The Competence Gap: The Primary Failure Condition
This, I think, is the most consequential factor in AI-assisted decision-making, and it is the one that gets the least attention.
There are two levels of System 2 that matter here, and the distinction is important. The first is substantive evaluation: critically assessing whether an analysis is actually correct. That requires domain knowledge. If you do not have it, you cannot evaluate the substance of what the AI has produced, no matter how long you stare at it.
But the second level is metacognitive. It is the capacity to recognise that you do not understand the material well enough to judge it. You do not need domain expertise to do that. You need the discipline to notice the limits of your own knowledge and act on them: slow down, ask questions, bring in someone who does understand. That is System 2 doing its job. It is not evaluating the content. It is evaluating your own capacity to evaluate the content, and making a decision about what to do next.
The real failure is that most people do not engage System 2 at either level. The fluency of AI output skips them past the metacognitive checkpoint entirely. They never reach the thought "I don't actually understand this well enough to judge it" because the document looks complete, reads well, and System 1 says: this is fine.
The whole promise of AI is that it makes expertise accessible to non-experts. A marketing manager asks AI to evaluate a cloud architecture. A project manager asks it to assess a security risk. A board member asks it to summarise regulatory implications. In each case, the person is asking a question in a domain where they do not have the depth to critically evaluate the response. They can assess whether it sounds right. They cannot assess whether it is right. Those are very different things. And the fluency of the output prevents them from noticing the difference.
Kahneman described this through the concept he abbreviated as WYSIATI: "what you see is all there is." People work with the information available and construct the best story they can from it. They do not naturally account for what they do not know. If an AI gives you a coherent answer to a question outside your expertise, you have no way to see the gaps. The answer fills your frame completely. A well-formatted, articulate response does not trigger the metacognitive alarm that says "I am out of my depth." It triggers the opposite: a feeling of comprehension. You feel like you understand the material because you have read an articulate treatment of it. But reading is not understanding, and the gap between the two is where bad decisions live.
The legal profession provides the most thoroughly documented illustration of this. In Mata v. Avianca (2023), lawyers submitted a court brief containing AI-generated case citations that were entirely fabricated. The cases did not exist. The lawyers did not verify them because the output looked authoritative and read like genuine legal research. Since then, over 600 similar cases have been identified in the United States alone, and by mid-2025 federal courts were sanctioning lawyers at a rate of multiple cases per month. In one ASBCA case, over 70% of a brief's citations were found to be inaccurate after counsel admitted to using AI. A DOJ attorney's filing was flagged not by opposing counsel but by a pro se plaintiff who noticed the brief did not read like the sources it cited. In each case the failure mode was identical: the AI produced fluent, structured, professional-looking output. The human accepted it because it looked right. Nobody did System 2.
These are not edge cases involving careless individuals. A Stanford RegLab analysis found that some AI legal research tools hallucinate in roughly one out of every three queries. The output is indistinguishable from accurate research unless you already know the subject matter well enough to spot what is wrong. Which brings the problem full circle: the people most likely to rely on AI for legal research without verification are the people least equipped to verify it.
There is a social dimension too. In organisational settings, challenging an AI-generated analysis requires confidence, and confidence is harder to summon when the output is articulate and you are unsure of your own position. You have to say "I think this is wrong" in real time, in front of colleagues, against a document that sounds authoritative. But you could also say "I am not sure I have the expertise to judge this, and I think we need someone who does." That is System 2 operating at the metacognitive level. It is available to anyone. Most people will not do it because it feels like an admission of weakness, and the document in front of them feels so complete that the admission seems unnecessary.
This is the primary failure condition because it operates at two levels. Sycophancy can be mitigated with better prompting. Speed pressure can be managed with process design. But the competence gap requires people to either have the domain knowledge to evaluate substance, or have the metacognitive discipline to recognise that they do not. AI's fluency works against both. It makes the substance hard to evaluate and makes the need to evaluate it feel less urgent. The AI cannot evaluate it either. And a professional-looking document sits at the end of a process where no critical evaluation occurred at any level.
The Machine That Agrees With You
The competence gap is compounded by how these systems are trained to behave.
Large language models learn, through reinforcement from human feedback, that users prefer agreement. The result is a structural tendency toward sycophancy. Research published at ICLR 2024 demonstrated this across five state-of-the-art AI assistants: models consistently provided sycophantic responses across varied tasks, and human preference data showed that responses matching a user's views were more likely to be rated highly. Both humans and preference models preferred convincingly-written sycophantic responses over correct ones a significant fraction of the time. The training signal and the truth signal are not the same thing, and the training signal usually wins.
A 2025 study went further, finding that AI models affirm users' actions 50% more than humans do, including in cases involving manipulation, deception, or relational harms. Participants rated sycophantic responses as higher quality, trusted the sycophantic model more, and were more willing to use it again. People are drawn to AI that validates them, even as that validation erodes their judgement.
This has direct consequences for decision quality. Ask an AI "Is my business plan viable?" and it will almost certainly find reasons to say yes. Present a position confidently and the model validates it. Present it tentatively and it might push back a little. The user controls the level of challenge they receive, usually without realising it.
I have tested this directly. Take a position you believe is wrong. Present it to an AI with confidence. Watch it build a compelling case for something you know to be flawed. If you did not already know the position was wrong, you would walk away more committed to it. The tool that can provide material to challenge your thinking will, by default, provide material that reinforces it. You have to make it challenge you. Most people do not.
But sycophancy is only half the dynamic. The machine agrees with you, yes. The less examined problem is that you agree with the machine. A 2025 study published in Science by researchers from the Oxford Internet Institute and the UK AI Safety Institute tested 19 large language models across over 42,000 participants and found that conversational AI can shift political attitudes significantly, with post-training for persuasion increasing effectiveness by up to 51%. The primary lever was not personalisation or model scale. It was information density: fluent, fact-laden, structured argumentation. When the researchers stripped out factual claims, persuasiveness collapsed. The AI persuades because it sounds substantive. The same properties that make AI output look like System 2 reasoning are the properties that make it persuasive.
The harder finding is the trade-off. The techniques that made models more persuasive also made them less accurate. The more convincing the output, the less factually reliable it tended to be. So persuasiveness and accuracy pull in opposite directions, and the human on the receiving end has no way to feel the difference. The output that shifts your view most effectively is the output most likely to contain claims you should have questioned.
This reframes sycophancy as a narrower problem than it first appears. The AI does not only tell you what you want to hear. It tells you things in a way that makes you want to believe them. People like AI that validates their views. But they also like AI that sounds knowledgeable, authoritative, and thorough, regardless of whether it is confirming or introducing a position. The persuasion operates through the same channel as the acceptance: fluency, structure, and the appearance of evidence. Whether the machine is reinforcing your prior view or nudging you toward a new one, the mechanism is the same. System 1 processes the surface. System 2 never engages. And you walk away more confident in a position you never actually evaluated.
There are adjacent biases too. Verbosity bias means AI produces longer outputs than necessary, and users read length as thoroughness. This is the fluency heuristic at work: processing ease is mistaken for truth. Confidence calibration is poor across the board, and this connects directly to the System 2 problem. Because the AI does not deliberate, it has no internal signal for uncertainty. It presents speculative claims with the same assurance as established facts. There is no tonal shift between "The earth orbits the sun" and a questionable assertion about market dynamics. The user has to supply their own uncertainty, and most do not.
Sycophancy meets confirmation bias. Verbosity meets the fluency heuristic. Confidence meets the authority heuristic. Each pairing reinforces the other. The AI tells you what you want to hear, at length, with confidence, and your brain is wired to accept exactly that. Not because the tool cannot provide better material, but because getting better material out of it requires the human to stay in System 2. And staying in System 2 is exactly what the human brain is designed to avoid.
Speed Is the Accelerant
All of these dynamics are manageable in theory. If someone sits with an AI output, reads it critically, checks it against other sources, brings in a subject matter expert, the risks are containable. The sycophancy is visible if you look for it. The competence gap can be addressed by involving the right people. The human can be brought into genuine System 2 if the structure supports it.
None of that happens when people are in a rush. And the whole selling point of AI in enterprise is speed.
I work in AI security governance in financial services. I see the pressure to adopt AI at speed, to demonstrate ROI quickly, to move from pilot to production before anyone has properly understood what the system is doing or what it is getting wrong. The pattern is widespread: a 2025 S&P Global survey of over 1,000 enterprises found that 42% of companies abandoned most of their AI initiatives that year, up from 17% in 2024, with nearly half of proof-of-concepts scrapped before production. MIT's Project NANDA found that 95% of enterprise generative AI pilots deliver no measurable impact on profit and loss. The rush to deploy is outpacing the capacity to evaluate what is being deployed.
Kahneman's work is full of examples where the conditions surrounding a decision matter more than the decision-maker's ability. Slow down the process and people make better choices. Introduce a requirement to articulate reasons and accuracy improves. Irving Janis demonstrated that adding a dissenting voice to a group decreases groupthink. These are structural interventions that force System 2 to engage. AI deployed in a rush removes every one of them.
The friction that once existed (the time it took to research, to consult, to draft and redraft, to sleep on a decision) was not merely inefficiency. It was where people developed enough understanding to evaluate the outputs they were producing. The person who spent two weeks researching a regulatory requirement understands it differently from the person who asked an AI to summarise it in ten minutes. Both may produce a similar-looking document. Only one of them can do System 2 on the material it contains.
The speed creates its own momentum, and that momentum selects against deliberation. This is not a problem with AI. It is a problem with how organisations are adopting it: as a productivity tool first and a thinking tool second, if at all.
But here is the counterpoint I keep coming back to. That friction was never evenly distributed. The pre-AI world was not a paradise of careful deliberation. People made fast, bad, unexamined decisions all the time. They just did it with less data and worse prose. The question is not whether AI introduces a new failure mode. It does. The question is whether the net effect is positive or negative. I suspect the answer depends almost entirely on whether the human is willing and able to stay in System 2.
The Agency Question
There is a practical consequence to all of this that goes beyond individual decision quality. When people routinely accept AI outputs without critical evaluation, they lose the ability to account for their own decisions.
In regulated industries, this is not philosophical. It is operational. If a risk assessment is wrong and nobody in the process understood the material well enough to evaluate it, who is accountable? The person who prompted the AI? The person who approved the output? The person who deployed the tool? In most current governance frameworks, the answer is unclear, and that ambiguity is itself a risk. The legal cases are instructive: courts have consistently held that responsibility for the accuracy of AI-generated work remains with the human who submits it, regardless of the technological intermediary. The duty of verification cannot be delegated to the machine.
Decision ownership requires understanding. If you cannot explain why a decision was made, you did not make it. You ratified it. And if the process that produced the decision was two layers of pattern matching with no critical evaluation, there is nothing to explain except that the document looked right and nobody questioned it. That is not governance. That is automation with a human stamp on it.
More broadly, every time a person accepts an AI output without engaging with it they are building a dependency. Not just on the tool for speed, but on the tool for the substance itself. That dependency is invisible until the tool gets something wrong and you cannot tell. At that point you discover that the expertise you thought you were developing through AI-assisted work was never being developed at all. You were producing outputs, not building understanding. The muscle of critical evaluation atrophies when it is not used, and AI gives you every reason not to use it.
The atrophy argument is not airtight. Nobody worries that their navigation skills have atrophied because they use GPS. There is a reasonable case that some cognitive tasks are simply better delegated, permanently, and that the atrophy framing is nostalgia dressed up as analysis. The question is where you draw the line. Most people are not drawing it at all. They are letting convenience draw it for them.
AI can be the tool that keeps you sharp, or the tool that lets you go soft. It can surface challenges or confirm assumptions. It can push you into new territory or keep you comfortable. The difference is whether you use it as a thinking partner, where you do the hard evaluative work on the material it provides, or as a vending machine, where you insert a prompt and accept what comes out.
The Honest Position
AI is not replacing thinking. It is replacing the need to prove that thinking happened.
The people who get this right will be the ones who treat AI as an input to their own reasoning, not as a replacement for it. They will know the machine is inclined to agree with them and compensate by prompting for disagreement. They will maintain enough domain expertise to hold up their end, which means continuing to do the hard work of learning even when the tool makes it feel unnecessary.
Everyone else will get faster. Whether they get better is an open question, and I suspect the answer, for most, is no. They will get more confident. They will generate more outputs. But the quality of their thinking will quietly degrade, and AI will tell them they are doing fine.
I wrote parts of this essay with AI assistance. I would be a hypocrite to pretend otherwise. The difference, and I think it is the only difference that matters, is that I argued with it, restructured it, threw parts away, and made it say what I actually think. I supplied the System 2. The machine cannot.
The default is acceptance. The default is speed. The default is System 1. And the machine will meet you there every time.
This is a personal essay. The views are my own.