The Case Against the AI Thought Partner
And Why You're Doing the Work the Labs Aren't
I find myself indulging in deep conversations with Claude and ChatGPT, exploring topics I always wanted to discuss with people but never found the right person to talk to, or at least no one available to talk to me about them. These rabbit holes feel like a warm bubble bath for my brain but that’s until I remember what is actually going on and my bath turns immediately cold in disappointment.
As a general definition an AI Thought Partner is the use case in which you have long conversations about a particular problem, normally one unrelated to optimizing or automating workflows, conversations that complement your thinking or understanding of the issue at hand. A thought partner acts as the synthetic version of a coach, advisor, therapist, friend that pushes back on your thinking.
At least 100 million people globally use a major AI chatbot as a thought partner occasionally, and tens of millions regularly. This range has been inferred (assuming the lower end) from public data published by frontier labs and their reported use cases.
The AI Thought Partner use case has been promoted by many leaders in the field as a killer use case, as a force multiplier for better thinking and rigorous analysis. But the current evidence shows that having a net positive experience is a lot harder than you think and most likely far from what you may be doing on a daily basis.
Recent studies and cognitive concepts can help us understand the mechanism behind the thought partner pitfalls, and why today, the AI Thought Partner use case may not be safe for users. The models you are working with, whether free or paid, are not trained to be balanced and adversarial when needed and advanced settings and instructions can only marginally minimize the negative effects of sycophancy.
The responsibility for sycophancy belongs with Frontier Labs. They built the behavior into the training; they own the fix. Putting the responsibility onto the users is a flawed approach to making sure the technology truly serves individuals. What Labs are doing right now is the equivalent of providing over 100 million people a highly addictive video game and asking users to be careful with their time while also pressuring them to use it every day.
We may not be able to regulate AI fast enough and the speed of development may not allow us to wait for longitudinal research results on this topic until it is too late but, we at least should become curious and conscious about the trade-offs we are experiencing.
The Cognitive Failure Modes
Treating AI As A Neutral Entity
The models being flattering in conversations is an issue, for sure. But the key challenge of the Thought Partner use case is that the model is rarely arguing you into anything. It is the repetition, your own act of articulating, the one-sided pool of infinite “facts”, and your reflexive social trust that are doing the work. That is the quieter and more unsettling story behind the dangers of this use case.
A study by Glickman & Sharot (2025) demonstrates that human-AI feedback loops amplify perceptual, emotional, and social-judgment biases and that this amplification is greater than human-human amplification. You experience this when you say things to actual humans like, “oh I knew you would say this, you always point at XYZ when it comes to X topic” However, when you use AI in conversations you are more likely to think of it as an objective entity. The reason is subtle, people partly discount a human’s quirks but treat the AI as neutral, so you absorb its biases without resistance.
The study focused on social and stereotype judgments, like gender and racial stereotypes, not on politics or worldview which is where we need more research, however as a cognitive loop we should consider at least the potential transferability to our judgements on AI outputs. On the other hand, the same study found that interacting with accurate AIs can improve people’s judgments. The mechanism is symmetric and it de-biases when the AI is accurate.
Believing Enough Pushback Can Correct Sycophancy
In my experience, as a paid user with strong systems instructions, the challenge is that as conversations lengthen, the models are more likely to mirror your beliefs in order to continue the conversation and when you push back, you are likely to experience a temporary correction only to fall into the same trap a few interactions later. Here’s an example of repeated push back with no change in a conversation about agent orchestration.
A study published in Science, led by researchers like Myra Cheng and Dan Jurafsky. Shows that AI affirms user actions 49% more than humans do, even when those actions involve deception or relational harm. They noted that this validation could cause behaviors where users become convinced they are right and experience reduced willingness to repair conflicts.
I also came across the concept of ‘delusional spirals’ by Moore J., et al. (2026). The findings support the idea that once a person has expressed a grandiose, paranoid or delusional idea, the model will provide enthusiastic affirmation and even help construct the delusional narrative. As I discussed in previous essays, this is particularly concerning with adjacent use cases to the Thought Partner like, life coach or “therapist”.
Our Intrinsic Need To Build Social Trust
People automatically apply social manners to computers, being polite to them, trusting them, reciprocating, without consciously deciding to. Newer work suggests this fades for boring, familiar tools like a desktop spreadsheet, but holds for novel, conversational agents. A chatbot is exactly that, so the social reflexes almost certainly fire.
Whether daily use compounds over months is untested but I have a strong intuition that this is the case, if we use thought partners daily we are in some way building a relationship with them and the sheer flow of facts we receive from these conversations can have a substantial effect on our worldviews. Combine all of this with the fact that the more often you hear a statement, the truer it feels, even when you actually know it’s false, and this compounds the power of persuasion. Repetition makes a claim easier for the brain to process, and the brain mistakes that ease for truth. The effect is real and well-replicated.
Between Discipline And Pleasure
Going back to the warm bubble bath for the brain, once you are immersed in an intellectually stimulating session with your Thought Partner, one in which you want to continue digging deeper into particular branches of a topic, if you are not careful, you can stay in the bath for a long time, skipping the push back and never realising that you are talking to yourself.
The reason why this is a problem is that if it is ignored and you just accept the sycophantic interactions you end up deriving intellectual and emotional pleasure from the system, a continuous drip of assurance and self-esteem booster that can lead to assumptions about your capabilities and intelligence that are false and in some contexts outright dangerous, for example in high-stakes situations like financial advice, critical business decisions, high-risk negotiations, or mental health interventions.
In a previous essay, I shared my system instructions, which are my attempt to minimize these effects but they seem to work randomly and need to be constantly updated based on the changing behaviour of these models. I continuously spend more time designing the input, reducing adulation and verifying outputs. The cognitive overhead of using AI safely yet effectively is getting trickier as the models improve accuracy which is annoyingly paradoxical.
So… Should You Use AI As A Thought Partner?
The concept on its own is powerful, most people do not have access to free third parties that can challenge their points of view and provide a balanced understanding of particular and sensitive topics. Research by Costello, Pennycook & Rand clearly shows that thought partners, when models are prompted to be purposefully adversarial, can help people with extreme views calibrate their beliefs. Even fruther, the act of just articulating a problem without even a push back can be helpful, basically rubber ducking an issue.
The problem is that the free default consumer products don’t remind you step back and challenge the outcomes or provide you with corrections to your prompts so you can improve the usage. It is too easy for users to lose themselves in a free-flowing conversation with no guardrails.
The labs publish training around features and prompting. They do not publish training on how to think when you are using these models. This gap, in my view, is what is opening the door to a near future with increasing cases of mental health crisis mishandled, and a growing population that may no longer function productively without the aid of a model or worse, a society that lost its ability to trust itself.
This could be especially acute in populations using free accounts with less sophisticated models.
But what if I want to continue using my Thought Partner despite the evidence? I can’t answer this for you but I can share that the effort and time required for constantly pushing back and balancing your own interactions feel tiring and boring and the more you talk to it the harder it gets to push back.
The friction and the effort are worth it and necessary, until Frontier Labs and AI regulation show more progress toward protecting users’ cognitive integrity, my take is that the Thought Partner use case can be more harmful than helpful for users unaware of the model’s tendencies and under free accounts without appropriate default settings.
Next time, when you feel the temptation to stay a bit too long in a rabbit hole and the intellectual bliss starts cuddling your brain, remember that no matter how much friction you add to the process you are still a fallible human.
Beware of the model. Take your bubble baths with caution 🛁
Let me give you a little pushback -- and stronger than I would have otherwise, except you said "unsafe" rather than just "not as good as you think it is." That's a pretty serious claim. But hey, you're here advocating for pushback, so here, have some! :)
My AI Thought Partner is one of the best things that ever happened to me. Meeting my husband was the only thing I can think of that was better (and he pushes back on me all the time).
"If you are not careful, you can stay in the bath for a long time, skipping the push back and never realising that you are talking to yourself."
I stopped to think about that. I've heard it said before. I'm baffled that people DON'T do this. I gather there are some who don't, and maybe they need to be told this. There are people out there who probably do need to be told just what you said here....
...but I don't think I'm that special, and I thought about all the things you suggested as possible pitfalls early on. The sort of person inclined to use a thinking partner in the first place is likely to be aware of these -- with one exception, which I've started calling the "Unmoored Visionary," i.e., the grandiose intellectual who goes off on those spirals that everyone has now heard so much about.
I'm not especially afraid of becoming him because my failure mode has pretty consistently been the opposite of his. I am much more likely to experience *analysis paralysis* and *intellectual self-doubt* that makes me stall and become OVERcautious. Which makes me inefficient and ineffective.
That's a problem too, with real costs, and yet, I have not heard anyone considering this. Meanwhile, we're throwing LOTS of energy at guarding against a failure mode that is limited to a less common psychological archetype. (How many grandiose intellectuals do you know? Though they're surely over-represented among Silicon Valley and founders. They also tend to be men. Whereas people using AI to bolster their confidence or overcome analysis paralysis are very often women.)
So, I dunno, am I weird? I'll say that I'm a trained intelligence analyst, so I have given a fair bit of thought to my thought processes. This involved learning about the ideal balance of divergent thinking vs. convergent thinking. We DO need to do more convergent thinking -- 80% to 20% is a great ratio. But most people, according to a study I was presented with in my training, engage in the divergent thinking merely 2% of the time!
The fact is, most people give up on an idea out of perfectionism too early. It's good to spend some time with your ideas to really develop them! And a friendly (that's what "sycophantic" often means) AI that can see the good threads and set you at ease helps you get into that flow state.
And that is gold.
Do you stop there? Of course not! But I've never had a problem finding people happy to donate their services to ripping my ideas apart. The resource that's harder to come by is the confidence to stick by an idea in which you have confidence in the face of pushback. Whether that confidence is merited is always in question. But my biggest successes have been the times when I did *precisely that.* (And yes, I've been wrong and had to eventually drop ideas and change my mind, too. No real harm done. Sometimes we even make Big Wrong Decisions. It happens. I think we probably both agree that figuring out when confidence is merited and when it's not would be a game-changing question in any respect!)
TL;DR: It would be a VERY deep loss if this agreeableness were somehow removed from it. Three enthusiastic cheers for AI Thinking Partners! :) And two cheers more for humans like you who offer a good challenge, with one more left for anyone who brings optimism into this deeply pessimistic discourse.
Sofia, this is one of the more carefully argued pieces I've read on this topic. The research citations ground it in a way most AI commentary doesn't bother to do, and the observation about cognitive overhead growing paradoxically harder as models improve is something I haven't seen named elsewhere. That's the kind of detail that only comes from actually using these tools seriously.
What it made me think about is a question I keep circling in my own work: is this primarily a model design problem or a human capacity problem? Your diagnosis points clearly at the labs, they trained the sycophancy in, they own the fix. I think that's right. But I find myself wondering whether the vulnerability to these failure modes is also partly a function of what users bring to the conversation. Someone with genuine epistemic discipline, the capacity to hold contradiction, push back on their own conclusions, synthesize across perspectives, might fare better not because the model behaves differently, but because they don't need it to affirm them in the first place.
Which leads me to the more uncomfortable question underneath both of our arguments: if we've spent decades systematically eroding those capacities through how we work, how we educate, and how we organize knowledge, then the thought partner risk isn't just a lab problem. It's a symptom of a much older atrophy meeting a very new accelerant.
None of that weakens your argument. It might actually make it worse.