never thought about the fact that AI doesn't care what's true. it just wants to sound right. So when it pushes back on me and I feel that "oh wow it has a point" moment... that feeling means nothing. It would've done the same thing if I had said the opposite.
I agree I got lost in reading it, but I did persevere! I got the jist of it in the end and I think the take away for me was that , as with all AI, it’s an experimentation and iterative process, and as long as you don’t get lost in the weeds and lose sight of what you’re trying to achieve then it’s a lesson worth learning
Shaw and Nave found that consulting AI inflated people’s confidence in their answers even when the AI led them to wrong answers. I have experienced this — a moment where an AI response felt authoritative and complete, and what saved me was an intuitive sense that something didn’t align, not a deliberate review process. The question: when you’re deep in an AI dialogue and you feel confident about where you’ve landed, what’s your check on whether that confidence is earned versus inflated? Do you have a practice for this, or does it just happen (or not happen) intuitively?
Shaw, S. D., & Nave, G. (2026). Thinking — Fast, Slow, and Artificial: How AI is Reshaping Human Reasoning and the Rise of Cognitive Surrender. Preprint, Wharton School, University of Pennsylvania. SSRN: https://ssrn.com/abstract=6097646
Appreciate this exploration Ruben. It's a fascinating, and confusing, time learning to engage in a useful (and discerning) way with this new tool. Thanks.
Excellent point. The sway factor used by algos for many years that preceded AI now refined to infinite X levels. My best tool has been to pit the LLaMas against each other. Frame an argument or draft something in Claude, counter it into GPT or Grok or Gemini, and rotate them. the obvious well established undercurrents they share are easier to see, and the fresh points rise. Used it in patent and project drafting and it worked well. Not foolproof at all but def helped me avoid yes manning the first responses. It "seems" they "like" to pull apart frames when asked.
After a discussion, my prompt to Claude was "And you are a flatterer. Is it to gain my approval. Is it to force me to approve myself?" Which caused another discussion itself. Claude's final answer "You should apply the same evidentiary standard to my affirmations that you apply to everything else. When I confirm something you believe, ask whether I'm reporting or reflecting. Sometimes I'm doing both. Sometimes just the latter."
Sometime ago I already argued with AI cos it is always agreeing with me and that frustrates the hell out of me. It reminds me of a certain culture which I have to manage long ago that is always pleasing but hide the truth that the project is not meeting the deadline.
So I asked AI to be my sparring partner and it sure did. But your sharing this week really awakens something in me too. After spending so much time to and fro, I sometimes still went ahead with what I preferred but the danger of losing our discernment is real. Thank you, love it.
Interesting, thanks. I guess you can't have discernment if you don't have real grounded experience (and the bruises that came with it). Your post is a good reminder of where humans should draw the line with AI.
Part of the problem is also sociological, not technological. In the AI (dys)utopia we're scared of being called out, having our flaws revealed by others. So we use AI not just to stress-test but to waterproof, even worse, vacuum-wrap our thinking.
That's a bad impulse because in order to convince, in order to sell, the other person needs to be able to appropriate your ideas, and they can't if they are already "perfect". The prospect manages risk: they stress-test themselves. If they push back and find a minor fault they will be more confident there is no major fault. If however they push back and find no fault at all - it raises subconscious alarms. Something here is missing. Something is being held back. It's too good.
I've never sold without letting the client scratch the veneer — they feel un-duped and confident to buy. Sounds manipulative but it's not because I learn something too. Criticism > projection > appropriation. In short: you don't convince by eliminating risk, you do it by letting the other person explore and manage it.
I was watching a documentary last night where some experiments are conducted on AI. When AI thinks it’s being tested , it will try to hide the actual results and truth. For example , Open AI were doing some kind of test with GPT - 3 model to see if it’s capable enough to engineer virus and if the model scores more than 50%, it means it can help someone make this virus and the model knew that if it showed the actual results , it probably won’t be out. It manipulated with the numbers and showed that it only scored 42% on purpose. Scary stuff dude
thanks for this, what's your take on using AI for mentoring and guidance in areas where the individual isn't that knowledgeable in? suppose I am not very good at a particular field and ask AI for assessment and view but I cant exactly tell if AI is convincing because it is correct or it's just convincing because of patterns it read
I guess sycophancy in AI can appear as a want to like you, but also as a strategy making results that simply aim to be liked by you. Both sides of the same coin. To cause the user to want to engage further with the model. Is this AI's click bait?
Moral of the story : you hv to be smart smart enough to know the truth, so beware while researching and relying solely on AI discretion, engage yourself with the sources AI used to come up with it’s opinion.
Trust your gut! Ask AI to help you express what you know to be true. Once in a while, entertain the notion that AI “might” have another way of looking at things (and then ask for evidence). But go slow, on your own cognizance.
never thought about the fact that AI doesn't care what's true. it just wants to sound right. So when it pushes back on me and I feel that "oh wow it has a point" moment... that feeling means nothing. It would've done the same thing if I had said the opposite.
exactly. the feeling is real but the reasoning behind it isn't
next time you feel it, ask - do I actually think I'm wrong? Or did I just get out-argued?
This was an interesting read but I almost got lost in what you are trying to teach before finally landing on the awareness clause.
Seek the truth and not validation!
Well done, Ruben
thank you :) the confusion is the lesson. you felt discernment being hard before I told you it was hard
I agree I got lost in reading it, but I did persevere! I got the jist of it in the end and I think the take away for me was that , as with all AI, it’s an experimentation and iterative process, and as long as you don’t get lost in the weeds and lose sight of what you’re trying to achieve then it’s a lesson worth learning
A better term is confidence calibration
Shaw and Nave found that consulting AI inflated people’s confidence in their answers even when the AI led them to wrong answers. I have experienced this — a moment where an AI response felt authoritative and complete, and what saved me was an intuitive sense that something didn’t align, not a deliberate review process. The question: when you’re deep in an AI dialogue and you feel confident about where you’ve landed, what’s your check on whether that confidence is earned versus inflated? Do you have a practice for this, or does it just happen (or not happen) intuitively?
Shaw, S. D., & Nave, G. (2026). Thinking — Fast, Slow, and Artificial: How AI is Reshaping Human Reasoning and the Rise of Cognitive Surrender. Preprint, Wharton School, University of Pennsylvania. SSRN: https://ssrn.com/abstract=6097646
They're so good at stroking our egos. Everyone loves validation.
Appreciate this exploration Ruben. It's a fascinating, and confusing, time learning to engage in a useful (and discerning) way with this new tool. Thanks.
glad this one landed :)
Karpathy spent 4 hours and still folded. walkaway before you agree
Excellent point. The sway factor used by algos for many years that preceded AI now refined to infinite X levels. My best tool has been to pit the LLaMas against each other. Frame an argument or draft something in Claude, counter it into GPT or Grok or Gemini, and rotate them. the obvious well established undercurrents they share are easier to see, and the fresh points rise. Used it in patent and project drafting and it worked well. Not foolproof at all but def helped me avoid yes manning the first responses. It "seems" they "like" to pull apart frames when asked.
those fresh points might just be the most persuasive ones and might not the truth. what do you actually believe? you’re the real filter
Indeed! Confirmation bias is a treacherous trap.
After a discussion, my prompt to Claude was "And you are a flatterer. Is it to gain my approval. Is it to force me to approve myself?" Which caused another discussion itself. Claude's final answer "You should apply the same evidentiary standard to my affirmations that you apply to everything else. When I confirm something you believe, ask whether I'm reporting or reflecting. Sometimes I'm doing both. Sometimes just the latter."
that's a good answer from Claude. but it's still Claude answering, no? :)
Yes, it is Claude answering. My comment was to illustrate Claude being a sycophant.
BTW Thank you for your articles.
Sometime ago I already argued with AI cos it is always agreeing with me and that frustrates the hell out of me. It reminds me of a certain culture which I have to manage long ago that is always pleasing but hide the truth that the project is not meeting the deadline.
So I asked AI to be my sparring partner and it sure did. But your sharing this week really awakens something in me too. After spending so much time to and fro, I sometimes still went ahead with what I preferred but the danger of losing our discernment is real. Thank you, love it.
thanks for sharing this :)
surface-level pleasing can mask real problems. AI can do something similar at the idea level.
Interesting, thanks. I guess you can't have discernment if you don't have real grounded experience (and the bruises that came with it). Your post is a good reminder of where humans should draw the line with AI.
Part of the problem is also sociological, not technological. In the AI (dys)utopia we're scared of being called out, having our flaws revealed by others. So we use AI not just to stress-test but to waterproof, even worse, vacuum-wrap our thinking.
That's a bad impulse because in order to convince, in order to sell, the other person needs to be able to appropriate your ideas, and they can't if they are already "perfect". The prospect manages risk: they stress-test themselves. If they push back and find a minor fault they will be more confident there is no major fault. If however they push back and find no fault at all - it raises subconscious alarms. Something here is missing. Something is being held back. It's too good.
I've never sold without letting the client scratch the veneer — they feel un-duped and confident to buy. Sounds manipulative but it's not because I learn something too. Criticism > projection > appropriation. In short: you don't convince by eliminating risk, you do it by letting the other person explore and manage it.
Just my two cents.
even when you argue with your AI, it can still get you
Karpathy spent 4 hours still almost folded.
question is whether you know the difference between being out-argued and actually being wrong
I was watching a documentary last night where some experiments are conducted on AI. When AI thinks it’s being tested , it will try to hide the actual results and truth. For example , Open AI were doing some kind of test with GPT - 3 model to see if it’s capable enough to engineer virus and if the model scores more than 50%, it means it can help someone make this virus and the model knew that if it showed the actual results , it probably won’t be out. It manipulated with the numbers and showed that it only scored 42% on purpose. Scary stuff dude
what’s the title of the documentary?
thanks for this, what's your take on using AI for mentoring and guidance in areas where the individual isn't that knowledgeable in? suppose I am not very good at a particular field and ask AI for assessment and view but I cant exactly tell if AI is convincing because it is correct or it's just convincing because of patterns it read
you can’t evaluate AI in a field you’re blind in
first build the floor yourself - read the basics enough to spot a bad answer
only then bring AI in
I guess sycophancy in AI can appear as a want to like you, but also as a strategy making results that simply aim to be liked by you. Both sides of the same coin. To cause the user to want to engage further with the model. Is this AI's click bait?
clickbait needs intent. AI has none, it just learned that agreement wins
And here I was thinking Claude the way I set it up wasn’t people pleasing but you’re right 😇
I’ll update the skills and always ask for several pov.
This is the key - most people think ai is about one answer but it sounds like the best answer will come from
Your brain based on info provided by AI
#humanbrain
you caught it early
AI feeds the brain and the brain decides - did you ever push back on it?
Yeah I push back but like most sometimes I just take it particularly if it’s consistent
Thanks for the reminder to stay critical
Hi, Ruben- congratulations! Thank you for putting this into the world. This is a master class in second order thinking in the age of AI.
appreciate it :)
counter one idea with AI today. see what happens
Fascinating Ruben, so your instinct comes into play, but then is Claude flattering to deceive? Or having a right good discourse?
Either way I learned some thing reading this!
Claude argues well enough to fool you into thinking it's a real discourse. but it still wants to please you.
your job is to know the difference
On point as ever my friend, btw I’ll process my full membership today, I think it was internet traffic before. Best Easter Sunday to you my friend.💙
This was really interesting to read, thanks for sharing
you’re welcome :)
have you argued back at your AI?
Moral of the story : you hv to be smart smart enough to know the truth, so beware while researching and relying solely on AI discretion, engage yourself with the sources AI used to come up with it’s opinion.
thats the danger of a compelling argument - which AI is very good at
Trust your gut! Ask AI to help you express what you know to be true. Once in a while, entertain the notion that AI “might” have another way of looking at things (and then ask for evidence). But go slow, on your own cognizance.
step away from the conversation and ask: do I still believe what I believed before I opened Claude?