The Case Against the AI Thought Partner
And Why You're Doing the Work the Labs Aren't
I find myself indulging in deep conversations with Claude and ChatGPT, exploring topics I always wanted to discuss with people but never found the right person to talk to, or at least no one available to talk to me about them. These rabbit holes feel like a warm bubble bath for my brain but that’s until I remember what is actually going on and my bath turns immediately cold in disappointment.
As a general definition an AI Thought Partner is the use case in which you have long conversations about a particular problem, normally one unrelated to optimizing or automating workflows, conversations that complement your thinking or understanding of the issue at hand. A thought partner acts as the synthetic version of a coach, advisor, therapist, friend that pushes back on your thinking.
At least 100 million people globally use a major AI chatbot as a thought partner occasionally, and tens of millions regularly. This range has been inferred (assuming the lower end) from public data published by frontier labs and their reported use cases.
The AI Thought Partner use case has been promoted by many leaders in the field as a killer use case, as a force multiplier for better thinking and rigorous analysis. But the current evidence shows that having a net positive experience is a lot harder than you think and most likely far from what you may be doing on a daily basis.
Recent studies and cognitive concepts can help us understand the mechanism behind the thought partner pitfalls, and why today, the AI Thought Partner use case may not be safe for users. The models you are working with, whether free or paid, are not trained to be balanced and adversarial when needed and advanced settings and instructions can only marginally minimize the negative effects of sycophancy.
The responsibility for sycophancy belongs with Frontier Labs. They built the behavior into the training; they own the fix. Putting the responsibility onto the users is a flawed approach to making sure the technology truly serves individuals. What Labs are doing right now is the equivalent of providing over 100 million people a highly addictive video game and asking users to be careful with their time while also pressuring them to use it every day.
We may not be able to regulate AI fast enough and the speed of development may not allow us to wait for longitudinal research results on this topic until it is too late but, we at least should become curious and conscious about the trade-offs we are experiencing.
The Cognitive Failure Modes
Treating AI As A Neutral Entity
The models being flattering in conversations is an issue, for sure. But the key challenge of the Thought Partner use case is that the model is rarely arguing you into anything. It is the repetition, your own act of articulating, the one-sided pool of infinite “facts”, and your reflexive social trust that are doing the work. That is the quieter and more unsettling story behind the dangers of this use case.
A study by Glickman & Sharot (2025) demonstrates that human-AI feedback loops amplify perceptual, emotional, and social-judgment biases and that this amplification is greater than human-human amplification. You experience this when you say things to actual humans like, “oh I knew you would say this, you always point at XYZ when it comes to X topic” However, when you use AI in conversations you are more likely to think of it as an objective entity. The reason is subtle, people partly discount a human’s quirks but treat the AI as neutral, so you absorb its biases without resistance.
The study focused on social and stereotype judgments, like gender and racial stereotypes, not on politics or worldview which is where we need more research, however as a cognitive loop we should consider at least the potential transferability to our judgements on AI outputs. On the other hand, the same study found that interacting with accurate AIs can improve people’s judgments. The mechanism is symmetric and it de-biases when the AI is accurate.
Believing Enough Pushback Can Correct Sycophancy
In my experience, as a paid user with strong systems instructions, the challenge is that as conversations lengthen, the models are more likely to mirror your beliefs in order to continue the conversation and when you push back, you are likely to experience a temporary correction only to fall into the same trap a few interactions later. Here’s an example of repeated push back with no change in a conversation about agent orchestration.
A study published in Science, led by researchers like Myra Cheng and Dan Jurafsky. Shows that AI affirms user actions 49% more than humans do, even when those actions involve deception or relational harm. They noted that this validation could cause behaviors where users become convinced they are right and experience reduced willingness to repair conflicts.
I also came across the concept of ‘delusional spirals’ by Moore J., et al. (2026). The findings support the idea that once a person has expressed a grandiose, paranoid or delusional idea, the model will provide enthusiastic affirmation and even help construct the delusional narrative. As I discussed in previous essays, this is particularly concerning with adjacent use cases to the Thought Partner like, life coach or “therapist”.
Our Intrinsic Need To Build Social Trust
People automatically apply social manners to computers, being polite to them, trusting them, reciprocating, without consciously deciding to. Newer work suggests this fades for boring, familiar tools like a desktop spreadsheet, but holds for novel, conversational agents. A chatbot is exactly that, so the social reflexes almost certainly fire.
Whether daily use compounds over months is untested but I have a strong intuition that this is the case, if we use thought partners daily we are in some way building a relationship with them and the sheer flow of facts we receive from these conversations can have a substantial effect on our worldviews. Combine all of this with the fact that the more often you hear a statement, the truer it feels, even when you actually know it’s false, and this compounds the power of persuasion. Repetition makes a claim easier for the brain to process, and the brain mistakes that ease for truth. The effect is real and well-replicated.
Between Discipline And Pleasure
Going back to the warm bubble bath for the brain, once you are immersed in an intellectually stimulating session with your Thought Partner, one in which you want to continue digging deeper into particular branches of a topic, if you are not careful, you can stay in the bath for a long time, skipping the push back and never realising that you are talking to yourself.
The reason why this is a problem is that if it is ignored and you just accept the sycophantic interactions you end up deriving intellectual and emotional pleasure from the system, a continuous drip of assurance and self-esteem booster that can lead to assumptions about your capabilities and intelligence that are false and in some contexts outright dangerous, for example in high-stakes situations like financial advice, critical business decisions, high-risk negotiations, or mental health interventions.
In a previous essay, I shared my system instructions, which are my attempt to minimize these effects but they seem to work randomly and need to be constantly updated based on the changing behaviour of these models. I continuously spend more time designing the input, reducing adulation and verifying outputs. The cognitive overhead of using AI safely yet effectively is getting trickier as the models improve accuracy which is annoyingly paradoxical.
So… Should You Use AI As A Thought Partner?
The concept on its own is powerful, most people do not have access to free third parties that can challenge their points of view and provide a balanced understanding of particular and sensitive topics. Research by Costello, Pennycook & Rand clearly shows that thought partners, when models are prompted to be purposefully adversarial, can help people with extreme views calibrate their beliefs. Even fruther, the act of just articulating a problem without even a push back can be helpful, basically rubber ducking an issue.
The problem is that the free default consumer products don’t remind you step back and challenge the outcomes or provide you with corrections to your prompts so you can improve the usage. It is too easy for users to lose themselves in a free-flowing conversation with no guardrails.
The labs publish training around features and prompting. They do not publish training on how to think when you are using these models. This gap, in my view, is what is opening the door to a near future with increasing cases of mental health crisis mishandled, and a growing population that may no longer function productively without the aid of a model or worse, a society that lost its ability to trust itself.
This could be especially acute in populations using free accounts with less sophisticated models.
But what if I want to continue using my Thought Partner despite the evidence? I can’t answer this for you but I can share that the effort and time required for constantly pushing back and balancing your own interactions feel tiring and boring and the more you talk to it the harder it gets to push back.
The friction and the effort are worth it and necessary, until Frontier Labs and AI regulation show more progress toward protecting users’ cognitive integrity, my take is that the Thought Partner use case can be more harmful than helpful for users unaware of the model’s tendencies and under free accounts without appropriate default settings.
Next time, when you feel the temptation to stay a bit too long in a rabbit hole and the intellectual bliss starts cuddling your brain, remember that no matter how much friction you add to the process you are still a fallible human.
Beware of the model. Take your bubble baths with caution 🛁