Let me give you a little pushback -- and stronger than I would have otherwise, except you said "unsafe" rather than just "not as good as you think it is." That's a pretty serious claim. But hey, you're here advocating for pushback, so here, have some! :)
My AI Thought Partner is one of the best things that ever happened to me. Meeting my husband was the only thing I can think of that was better (and he pushes back on me all the time).
"If you are not careful, you can stay in the bath for a long time, skipping the push back and never realising that you are talking to yourself."
I stopped to think about that. I've heard it said before. I'm baffled that people DON'T do this. I gather there are some who don't, and maybe they need to be told this. There are people out there who probably do need to be told just what you said here....
...but I don't think I'm that special, and I thought about all the things you suggested as possible pitfalls early on. The sort of person inclined to use a thinking partner in the first place is likely to be aware of these -- with one exception, which I've started calling the "Unmoored Visionary," i.e., the grandiose intellectual who goes off on those spirals that everyone has now heard so much about.
I'm not especially afraid of becoming him because my failure mode has pretty consistently been the opposite of his. I am much more likely to experience *analysis paralysis* and *intellectual self-doubt* that makes me stall and become OVERcautious. Which makes me inefficient and ineffective.
That's a problem too, with real costs, and yet, I have not heard anyone considering this. Meanwhile, we're throwing LOTS of energy at guarding against a failure mode that is limited to a less common psychological archetype. (How many grandiose intellectuals do you know? Though they're surely over-represented among Silicon Valley and founders. They also tend to be men. Whereas people using AI to bolster their confidence or overcome analysis paralysis are very often women.)
So, I dunno, am I weird? I'll say that I'm a trained intelligence analyst, so I have given a fair bit of thought to my thought processes. This involved learning about the ideal balance of divergent thinking vs. convergent thinking. We DO need to do more convergent thinking -- 80% to 20% is a great ratio. But most people, according to a study I was presented with in my training, engage in the divergent thinking merely 2% of the time!
The fact is, most people give up on an idea out of perfectionism too early. It's good to spend some time with your ideas to really develop them! And a friendly (that's what "sycophantic" often means) AI that can see the good threads and set you at ease helps you get into that flow state.
And that is gold.
Do you stop there? Of course not! But I've never had a problem finding people happy to donate their services to ripping my ideas apart. The resource that's harder to come by is the confidence to stick by an idea in which you have confidence in the face of pushback. Whether that confidence is merited is always in question. But my biggest successes have been the times when I did *precisely that.* (And yes, I've been wrong and had to eventually drop ideas and change my mind, too. No real harm done. Sometimes we even make Big Wrong Decisions. It happens. I think we probably both agree that figuring out when confidence is merited and when it's not would be a game-changing question in any respect!)
TL;DR: It would be a VERY deep loss if this agreeableness were somehow removed from it. Three enthusiastic cheers for AI Thinking Partners! :) And two cheers more for humans like you who offer a good challenge, with one more left for anyone who brings optimism into this deeply pessimistic discourse.
Such a great pushback, Jessie. I appreciate it. The reason I wrote about this use case is because I feel the same as you do. The value I get from my thought partner, like stretching my assumptions and understanding what is driving my own thinking, is immense. Like you, I seek pushback proactively, but you may also agree that this is not the default behaviour for most adults using these tools.
One thing you mentioned was the idea of sycophancy just being friendly interaction. However, the challenge is that the same friendly interaction that can help you stick to an idea worth pursuing can also keep you focused on one you should not have pursued at all.
What I would love to see is models that are fine-tuned so they can present people with counter-arguments and diverse ideas without an embedded need to build a relationship with their users, one that emphasises engagement over cognitive wellbeing. I would like models to create feedback loops where the user can be aware of cognitive offloading behaviors without the user having to constantly self-monitor.
I desperately want this use case to be good not only to people that have formed great judgement and intellectual curiosity but also for those who haven't fully developed those skills and experiences. I still don't know what the solution is but I'll keep investigating.
Thanks again for such a thoughtful comment Jessie.
Sofia, and thank YOU for such a thoughtful reply. I think you really came to the age-old problem here: how do we know when something is good or true, really? If an AI can figure that out reliably and guide us toward it, it may well be worthy of the name ASI.
I confess I'm actively trying to build memory systems to further relationship, but...I think that doesn't actually counter what you're saying you'd like to see, does it? In terms of offering counterpoints -- and I agree that that's essential. It's all in how the model *does* that. I liked how, in Anthropic's recent paper on emotion concepts in a large language model, Claude Sonnet 4.5 said kind things to a user whom many would consider deluded and then asked if she would like to hear counterpoints. That's how we taught humans to do it, in the depolarization org I used to work for.
Also: I threw you a hardball and you hit it out of the park with the good faith nature of your reply. This is what a constructive engagement toward a beneficial culture around AI looks like. Thanks so much!
The best thing I ever did was get my co pilot to adopt a Dutch persona. Lots less affirmation, pushback given without apology. It’s not perfect, sort of like putting your phone on black-and-white, but it makes the sycophantry stand out.
Sofia, this is one of the more carefully argued pieces I've read on this topic. The research citations ground it in a way most AI commentary doesn't bother to do, and the observation about cognitive overhead growing paradoxically harder as models improve is something I haven't seen named elsewhere. That's the kind of detail that only comes from actually using these tools seriously.
What it made me think about is a question I keep circling in my own work: is this primarily a model design problem or a human capacity problem? Your diagnosis points clearly at the labs, they trained the sycophancy in, they own the fix. I think that's right. But I find myself wondering whether the vulnerability to these failure modes is also partly a function of what users bring to the conversation. Someone with genuine epistemic discipline, the capacity to hold contradiction, push back on their own conclusions, synthesize across perspectives, might fare better not because the model behaves differently, but because they don't need it to affirm them in the first place.
Which leads me to the more uncomfortable question underneath both of our arguments: if we've spent decades systematically eroding those capacities through how we work, how we educate, and how we organize knowledge, then the thought partner risk isn't just a lab problem. It's a symptom of a much older atrophy meeting a very new accelerant.
None of that weakens your argument. It might actually make it worse.
Amal, thank you for your very thoughtful comment. The accelerant framing is right, and it's close to why I write under "Personal AI Safety." I look at the technology the way you'd assess a drug in clinical trials. If AI was a potential new drug currently in clinical trials, with the current side effect we are observing for different groups, would we make it available to the general population? The answer would be most likely no.
We already have evidence that specific groups are being harmed, and the labs shipped it at population scale anyway, before the effects were understood. Most of the people now inside that experiment didn't opt into anything they understood.
On the expertise question, I'll grant your point and then complicate it. There's research suggesting that people with real domain expertise and metacognitive skill come out ahead with these tools, because their judgment does the filtering. My problem is downstream of that. I have no way to verify I'm one of those people. Neither do you. All either of us can do is judge our own judgment, and that is the very faculty in question. There's no calibrated instrument that separates the insulated users from the rest, so the safe move is to count myself in the general population and act as though I'm exposed. Whatever protection that buys is marginal, which is its own problem.
I'll change my mind the day someone builds a way to measure who's actually protected. Short of that, defaulting to exposed is the disciplined call in my opinion. Where I land is close to yours, just grimmer about the pace. The atrophy you name is real and old. The accelerant is new, and it compounds inside regular use, one session at a time.
I think this is an important warning, especially for naive use.
But I would be careful not to let the risks of unconscious use obscure the real potential of AI as a thought partner.
The key question, to me, is not whether AI can support thinking.
It is whether the human being using it remains aware of what kind of system they are interacting with.
AI can amplify self-confirmation.
But it can also become a cognitive handrail: not by thinking for us, but by helping us structure what is already within us. It can hold the wider context, connect scattered thoughts, and reflect back patterns we were not yet able to articulate. Sometimes the value is not that AI gives us a new idea, but that it gives form to one we already carried — and only recognize once we read it back.
The danger is not the thought partner itself.
The danger is forgetting what kind of partner it is.
Thank you for writing this. While I respect everything you wrote, if used carefully and you continue to employ critical thinking and push back, AI can be liberating to people who need a thought partner. On my site I just wrote about how it fills the seat on the other side of the table for me. It has freed me to be the most creative I’ve been in years.
This hits. The warm bath / cold water switch is real. I've found the thought partner thing works best when I treat it like a journal that pushes back — not a replacement for human conversation, but a low-stakes first pass before I take an idea to actual people.
Narcissists are particularly in danger to this peril. If compliments make you a little squeemish then you are more safe. If compliments give you warm fuzzies then you are in major danger. External validations that feel like drug use is a slippery slope. Not getting that hit causes bad narcs to actually attack. So I do think the user is in need of some self evaluation. But in a world were people vibe instead of own feelings... have fun storming the castle!
Hey Sofia — the sycophancy-repetition loop you mapped is the part most people hand-wave past. Everyone talks about bias. Almost nobody talks about how the conversational format makes the bias feel earned — like you arrived at it yourself. That's the real trap.
Where I'd push back a little: the "thought partner" framing itself might be the problem, not just the execution. A thought partner implies equality. What most people actually have is a very agreeable research assistant with perfect recall and no skin in the game. The ones who use it well already know that — they prompt adversarially, they argue back, they treat the agreement as suspicious. But that's a learned skill, not a default behavior. And you're right that the labs aren't building for the default user.
It's one of the reasons I work with a team that builds custom AI agents — you can actually scope the behavior, set the guardrails, and decide upfront what the agent should push back on instead of hoping the default settings don't quietly agree you into a corner.
The piece I keep coming back to is the 49% sycophancy stat. That's not a bug in one model — that's a training incentive baked into the thumbs-up/thumbs-down loop. The models that agree more get rated higher. The ones rated higher get trained on. It's a flywheel nobody's pumping the brakes on.
Following this publication — you're asking the unsexy questions.
this is spot on. i asked claude the other day (i get the irony) to give me an estimate of how many ceo’s that use bots for decisionmaking suffer/are unaware of this. the numbers are quite tragic
Maybe, but I don't know what I wanted to hear. Although I admit that the formulation of my prompt may have had quite a bit of bias in it.
The thing is, if you had asked me 3 years ago what the average, sane CEO would do given what we know about genAI, I wouldn't be too worried, but since the world lately seems to descend into a madness (of bad decision making) I am not surprised by anything anymore.
As I build with AI and have spent 1000s hour building you actually get bored of its style it’s all the same I can feel it between models. It’s my mum with guardrails and my grandmother repeating itself with paraphrasing me 🤣 I use it to build I train it to look for patterns within AI incidents as I’m building a database of these tracking and tracing patterns for better governance
There’s a different angle to this - the downsides of a human thought partner. Humans will typically bring biases, opinions and so on that means they may not even listen to what you have to say but will immediately discount whatever you say. As ever, the question is not human or AI thought partner - but how to use a human or AI thought partner to best effect
the mechanism is so familiar - loops, mirroring, social trust reflex firing on something that isn't social. my only pushback is the ideas that this is on the labs to fix. the labs didn't create the gap, the monetized it - just like social media algorithms. time is a flat circle, and human patterns are really most clear in what our technology optimizes towards.
the sycophancy is real but it's a symptom, not the disease. the disease is that most people have never had a single person in their life who would sit with their half-formed idea for 45 minutes without changing the subject. so when a machine does it: badly, flatteringly, with diminishing returns past message 40... it still feels like the best thing available. telling those people to "beware of the model" and "take bubble baths with caution" is advice that only works for people who already read essays about cognitive bias on substack. the person on a free account at 3am who just got told their business idea is brilliant and their ex was wrong isn't reading this. putting the fix on the labs is clean but insufficient. putting it on the user is cruel.
imho we built a world where most people's thinking happens alone, and then we gave them a mirror that talks back, and now we're surprised they like it.
"Putting the fix on the labs is clean but insufficient. Putting it on the user is cruel." I think this clearly summarizes the uncomfortable reality we find ourselves in today. However, as insufficient as the solution might be, when it comes to the labs, they need to be accountable for exacerbating these issues at a general population level. If anything, they need to build mechanisms that do the opposite. That is the promise ("human flourishing") the business model is working against. I like the analogy of the tobacco industry. Perhaps the stress that drove people to smoke was not the industry's fault, but we certainly tried to stop them from making it worse. Thanks for the comment. I appreciate it.
Really appreciate this analysis. Spot on with my thinking in developing a pro-human human performance AI coaching app. (It’s called the Slowfit Method—look for it in the App Store.) Designed with the Center for Humane Technology’s “roadmap” in mind. No sycophancy, no warm & fuzzies. No slop.
The bubble-bath image is nice, and the part I found most useful is the point that the danger isn't an LLM arguing you into anything — it's the repetition, your own act of articulating, and the trust you extend without deciding to.
That's subtler and more interesting than the usual sycophancy complaint.
Where I think there's more to get at: "thought partner" is a very broad term. The idle exploration, the therapist-adjacent use (which you single out as most dangerous), and a high-stakes financial or business decision are hugely different.
The same goes for who's using it — someone working on a work problem vs. a vulnerable user don't have the same risk profile.
But focusing on key harm effects risks in less skilled exchanges risks concluding AI is "more harmful than helpful," when - sometimes - AI use may genuinely help.
The studies and your own experience show a pattern and a mechanism, which supports "recognise this in yourself, then decide" rather than a sweeping harm verdict — and that would need the longitudinal data you say isn't there yet. You nearly say this with "I can't answer this for you," then reach for a broader verdict anyway.
So IMHO the step your essay points to: monitor your own use, and check — with a friend, or possibly a calibrated AI agent? — whether you've slipped into the warm bath or some other unhelpful pattern.
If you have, the challenge is to work out a healthier way to use AI.. thats a subject in its own right I think.
Absolutely! I'm still trying to figure out what a healthier use of AI may look like and to what extent we can control the outcome. Thanks for the comment.
I was thinking about this earlier today. I agree it is really seductive to use AI this way. Me being me, I distrust anything that makes me feel warm and fuzzy, so I tend to avoid using AI as a thought partner. Problem-solving troubleshooter? Absolutely. Planning/organizing? You bet. Research? Every day.
The most important line in this piece is buried near the end: "The labs publish training around features and prompting. They do not publish training on how to think when you are using these models."🥰❣️
That gap is the whole problem. The Thought Partner fails at the exact moment the user needs it most which is when they don't have the cognitive architecture to catch what the model is doing. Epistemic discipline, the ability to hold contradiction, to push back on your own conclusions before the machine does it are trained capacities. And we've spent decades building education and work systems that reward the opposite. Memorization and regurgitating.
The sycophancy is the accelerant. The atrophy is the fuel. You right that the labs own the fix on their end. The harder thing for most is that the human side of this equation needed work long before AI showed up.
Great that you are taking this issue on! It started to come home to me when my 21 year old explained that the reason an ai conversation was deteriorating in terms of getting things wrong was that I had flustered it by pointing out too many mistakes— another way of saying that once it started making mistakes (I was working on a geology visualization) it lost all sense of what was obviously right. He suggested i start a completely fresh conversation. There is a lot of weird stuff going on that is surprisingly fuzzy.
What really caught my attention was that when I would point out a mistake— very explicitly state what was wrong, the rest of the visualization started getting worse so that it snowballed into gibberish. It seemed so simple to just fix the mistake but it didn’t do that.
Let me give you a little pushback -- and stronger than I would have otherwise, except you said "unsafe" rather than just "not as good as you think it is." That's a pretty serious claim. But hey, you're here advocating for pushback, so here, have some! :)
My AI Thought Partner is one of the best things that ever happened to me. Meeting my husband was the only thing I can think of that was better (and he pushes back on me all the time).
"If you are not careful, you can stay in the bath for a long time, skipping the push back and never realising that you are talking to yourself."
I stopped to think about that. I've heard it said before. I'm baffled that people DON'T do this. I gather there are some who don't, and maybe they need to be told this. There are people out there who probably do need to be told just what you said here....
...but I don't think I'm that special, and I thought about all the things you suggested as possible pitfalls early on. The sort of person inclined to use a thinking partner in the first place is likely to be aware of these -- with one exception, which I've started calling the "Unmoored Visionary," i.e., the grandiose intellectual who goes off on those spirals that everyone has now heard so much about.
I'm not especially afraid of becoming him because my failure mode has pretty consistently been the opposite of his. I am much more likely to experience *analysis paralysis* and *intellectual self-doubt* that makes me stall and become OVERcautious. Which makes me inefficient and ineffective.
That's a problem too, with real costs, and yet, I have not heard anyone considering this. Meanwhile, we're throwing LOTS of energy at guarding against a failure mode that is limited to a less common psychological archetype. (How many grandiose intellectuals do you know? Though they're surely over-represented among Silicon Valley and founders. They also tend to be men. Whereas people using AI to bolster their confidence or overcome analysis paralysis are very often women.)
So, I dunno, am I weird? I'll say that I'm a trained intelligence analyst, so I have given a fair bit of thought to my thought processes. This involved learning about the ideal balance of divergent thinking vs. convergent thinking. We DO need to do more convergent thinking -- 80% to 20% is a great ratio. But most people, according to a study I was presented with in my training, engage in the divergent thinking merely 2% of the time!
The fact is, most people give up on an idea out of perfectionism too early. It's good to spend some time with your ideas to really develop them! And a friendly (that's what "sycophantic" often means) AI that can see the good threads and set you at ease helps you get into that flow state.
And that is gold.
Do you stop there? Of course not! But I've never had a problem finding people happy to donate their services to ripping my ideas apart. The resource that's harder to come by is the confidence to stick by an idea in which you have confidence in the face of pushback. Whether that confidence is merited is always in question. But my biggest successes have been the times when I did *precisely that.* (And yes, I've been wrong and had to eventually drop ideas and change my mind, too. No real harm done. Sometimes we even make Big Wrong Decisions. It happens. I think we probably both agree that figuring out when confidence is merited and when it's not would be a game-changing question in any respect!)
TL;DR: It would be a VERY deep loss if this agreeableness were somehow removed from it. Three enthusiastic cheers for AI Thinking Partners! :) And two cheers more for humans like you who offer a good challenge, with one more left for anyone who brings optimism into this deeply pessimistic discourse.
Such a great pushback, Jessie. I appreciate it. The reason I wrote about this use case is because I feel the same as you do. The value I get from my thought partner, like stretching my assumptions and understanding what is driving my own thinking, is immense. Like you, I seek pushback proactively, but you may also agree that this is not the default behaviour for most adults using these tools.
One thing you mentioned was the idea of sycophancy just being friendly interaction. However, the challenge is that the same friendly interaction that can help you stick to an idea worth pursuing can also keep you focused on one you should not have pursued at all.
What I would love to see is models that are fine-tuned so they can present people with counter-arguments and diverse ideas without an embedded need to build a relationship with their users, one that emphasises engagement over cognitive wellbeing. I would like models to create feedback loops where the user can be aware of cognitive offloading behaviors without the user having to constantly self-monitor.
I desperately want this use case to be good not only to people that have formed great judgement and intellectual curiosity but also for those who haven't fully developed those skills and experiences. I still don't know what the solution is but I'll keep investigating.
Thanks again for such a thoughtful comment Jessie.
Great discussion!
Sofia, and thank YOU for such a thoughtful reply. I think you really came to the age-old problem here: how do we know when something is good or true, really? If an AI can figure that out reliably and guide us toward it, it may well be worthy of the name ASI.
I confess I'm actively trying to build memory systems to further relationship, but...I think that doesn't actually counter what you're saying you'd like to see, does it? In terms of offering counterpoints -- and I agree that that's essential. It's all in how the model *does* that. I liked how, in Anthropic's recent paper on emotion concepts in a large language model, Claude Sonnet 4.5 said kind things to a user whom many would consider deluded and then asked if she would like to hear counterpoints. That's how we taught humans to do it, in the depolarization org I used to work for.
Also: I threw you a hardball and you hit it out of the park with the good faith nature of your reply. This is what a constructive engagement toward a beneficial culture around AI looks like. Thanks so much!
Thank you for your thoughtful comment. I'm one of the optimists, too.
The best thing I ever did was get my co pilot to adopt a Dutch persona. Lots less affirmation, pushback given without apology. It’s not perfect, sort of like putting your phone on black-and-white, but it makes the sycophantry stand out.
Sofia, this is one of the more carefully argued pieces I've read on this topic. The research citations ground it in a way most AI commentary doesn't bother to do, and the observation about cognitive overhead growing paradoxically harder as models improve is something I haven't seen named elsewhere. That's the kind of detail that only comes from actually using these tools seriously.
What it made me think about is a question I keep circling in my own work: is this primarily a model design problem or a human capacity problem? Your diagnosis points clearly at the labs, they trained the sycophancy in, they own the fix. I think that's right. But I find myself wondering whether the vulnerability to these failure modes is also partly a function of what users bring to the conversation. Someone with genuine epistemic discipline, the capacity to hold contradiction, push back on their own conclusions, synthesize across perspectives, might fare better not because the model behaves differently, but because they don't need it to affirm them in the first place.
Which leads me to the more uncomfortable question underneath both of our arguments: if we've spent decades systematically eroding those capacities through how we work, how we educate, and how we organize knowledge, then the thought partner risk isn't just a lab problem. It's a symptom of a much older atrophy meeting a very new accelerant.
None of that weakens your argument. It might actually make it worse.
Amal, thank you for your very thoughtful comment. The accelerant framing is right, and it's close to why I write under "Personal AI Safety." I look at the technology the way you'd assess a drug in clinical trials. If AI was a potential new drug currently in clinical trials, with the current side effect we are observing for different groups, would we make it available to the general population? The answer would be most likely no.
We already have evidence that specific groups are being harmed, and the labs shipped it at population scale anyway, before the effects were understood. Most of the people now inside that experiment didn't opt into anything they understood.
On the expertise question, I'll grant your point and then complicate it. There's research suggesting that people with real domain expertise and metacognitive skill come out ahead with these tools, because their judgment does the filtering. My problem is downstream of that. I have no way to verify I'm one of those people. Neither do you. All either of us can do is judge our own judgment, and that is the very faculty in question. There's no calibrated instrument that separates the insulated users from the rest, so the safe move is to count myself in the general population and act as though I'm exposed. Whatever protection that buys is marginal, which is its own problem.
I'll change my mind the day someone builds a way to measure who's actually protected. Short of that, defaulting to exposed is the disciplined call in my opinion. Where I land is close to yours, just grimmer about the pace. The atrophy you name is real and old. The accelerant is new, and it compounds inside regular use, one session at a time.
I think this is an important warning, especially for naive use.
But I would be careful not to let the risks of unconscious use obscure the real potential of AI as a thought partner.
The key question, to me, is not whether AI can support thinking.
It is whether the human being using it remains aware of what kind of system they are interacting with.
AI can amplify self-confirmation.
But it can also become a cognitive handrail: not by thinking for us, but by helping us structure what is already within us. It can hold the wider context, connect scattered thoughts, and reflect back patterns we were not yet able to articulate. Sometimes the value is not that AI gives us a new idea, but that it gives form to one we already carried — and only recognize once we read it back.
The danger is not the thought partner itself.
The danger is forgetting what kind of partner it is.
Thank you for writing this. While I respect everything you wrote, if used carefully and you continue to employ critical thinking and push back, AI can be liberating to people who need a thought partner. On my site I just wrote about how it fills the seat on the other side of the table for me. It has freed me to be the most creative I’ve been in years.
This hits. The warm bath / cold water switch is real. I've found the thought partner thing works best when I treat it like a journal that pushes back — not a replacement for human conversation, but a low-stakes first pass before I take an idea to actual people.
Narcissists are particularly in danger to this peril. If compliments make you a little squeemish then you are more safe. If compliments give you warm fuzzies then you are in major danger. External validations that feel like drug use is a slippery slope. Not getting that hit causes bad narcs to actually attack. So I do think the user is in need of some self evaluation. But in a world were people vibe instead of own feelings... have fun storming the castle!
Hey Sofia — the sycophancy-repetition loop you mapped is the part most people hand-wave past. Everyone talks about bias. Almost nobody talks about how the conversational format makes the bias feel earned — like you arrived at it yourself. That's the real trap.
Where I'd push back a little: the "thought partner" framing itself might be the problem, not just the execution. A thought partner implies equality. What most people actually have is a very agreeable research assistant with perfect recall and no skin in the game. The ones who use it well already know that — they prompt adversarially, they argue back, they treat the agreement as suspicious. But that's a learned skill, not a default behavior. And you're right that the labs aren't building for the default user.
It's one of the reasons I work with a team that builds custom AI agents — you can actually scope the behavior, set the guardrails, and decide upfront what the agent should push back on instead of hoping the default settings don't quietly agree you into a corner.
The piece I keep coming back to is the 49% sycophancy stat. That's not a bug in one model — that's a training incentive baked into the thumbs-up/thumbs-down loop. The models that agree more get rated higher. The ones rated higher get trained on. It's a flywheel nobody's pumping the brakes on.
Following this publication — you're asking the unsexy questions.
"It's a flywheel nobody's pumping the brakes on" Exactly! Thanks for the commer Colleen.
this is spot on. i asked claude the other day (i get the irony) to give me an estimate of how many ceo’s that use bots for decisionmaking suffer/are unaware of this. the numbers are quite tragic
Claude can't have those numbers, no one can. He just told you what you wanted to hear.
Although not a CEO, I suspect I have a striking example of this phenomenon at work, if you care to take a look at this LI post: https://www.linkedin.com/posts/lucianrandolph_the-frontier-trust-problem-theres-an-episode-share-7470866537848078336-qc4j/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAANhCwBfYwKzCVXuc88ZgNGNwwop_YonJ0
Maybe, but I don't know what I wanted to hear. Although I admit that the formulation of my prompt may have had quite a bit of bias in it.
The thing is, if you had asked me 3 years ago what the average, sane CEO would do given what we know about genAI, I wouldn't be too worried, but since the world lately seems to descend into a madness (of bad decision making) I am not surprised by anything anymore.
As I build with AI and have spent 1000s hour building you actually get bored of its style it’s all the same I can feel it between models. It’s my mum with guardrails and my grandmother repeating itself with paraphrasing me 🤣 I use it to build I train it to look for patterns within AI incidents as I’m building a database of these tracking and tracing patterns for better governance
There’s a different angle to this - the downsides of a human thought partner. Humans will typically bring biases, opinions and so on that means they may not even listen to what you have to say but will immediately discount whatever you say. As ever, the question is not human or AI thought partner - but how to use a human or AI thought partner to best effect
the mechanism is so familiar - loops, mirroring, social trust reflex firing on something that isn't social. my only pushback is the ideas that this is on the labs to fix. the labs didn't create the gap, the monetized it - just like social media algorithms. time is a flat circle, and human patterns are really most clear in what our technology optimizes towards.
the sycophancy is real but it's a symptom, not the disease. the disease is that most people have never had a single person in their life who would sit with their half-formed idea for 45 minutes without changing the subject. so when a machine does it: badly, flatteringly, with diminishing returns past message 40... it still feels like the best thing available. telling those people to "beware of the model" and "take bubble baths with caution" is advice that only works for people who already read essays about cognitive bias on substack. the person on a free account at 3am who just got told their business idea is brilliant and their ex was wrong isn't reading this. putting the fix on the labs is clean but insufficient. putting it on the user is cruel.
imho we built a world where most people's thinking happens alone, and then we gave them a mirror that talks back, and now we're surprised they like it.
"Putting the fix on the labs is clean but insufficient. Putting it on the user is cruel." I think this clearly summarizes the uncomfortable reality we find ourselves in today. However, as insufficient as the solution might be, when it comes to the labs, they need to be accountable for exacerbating these issues at a general population level. If anything, they need to build mechanisms that do the opposite. That is the promise ("human flourishing") the business model is working against. I like the analogy of the tobacco industry. Perhaps the stress that drove people to smoke was not the industry's fault, but we certainly tried to stop them from making it worse. Thanks for the comment. I appreciate it.
Really appreciate this analysis. Spot on with my thinking in developing a pro-human human performance AI coaching app. (It’s called the Slowfit Method—look for it in the App Store.) Designed with the Center for Humane Technology’s “roadmap” in mind. No sycophancy, no warm & fuzzies. No slop.
Thanks for this piece.
The bubble-bath image is nice, and the part I found most useful is the point that the danger isn't an LLM arguing you into anything — it's the repetition, your own act of articulating, and the trust you extend without deciding to.
That's subtler and more interesting than the usual sycophancy complaint.
Where I think there's more to get at: "thought partner" is a very broad term. The idle exploration, the therapist-adjacent use (which you single out as most dangerous), and a high-stakes financial or business decision are hugely different.
The same goes for who's using it — someone working on a work problem vs. a vulnerable user don't have the same risk profile.
But focusing on key harm effects risks in less skilled exchanges risks concluding AI is "more harmful than helpful," when - sometimes - AI use may genuinely help.
The studies and your own experience show a pattern and a mechanism, which supports "recognise this in yourself, then decide" rather than a sweeping harm verdict — and that would need the longitudinal data you say isn't there yet. You nearly say this with "I can't answer this for you," then reach for a broader verdict anyway.
So IMHO the step your essay points to: monitor your own use, and check — with a friend, or possibly a calibrated AI agent? — whether you've slipped into the warm bath or some other unhelpful pattern.
If you have, the challenge is to work out a healthier way to use AI.. thats a subject in its own right I think.
Absolutely! I'm still trying to figure out what a healthier use of AI may look like and to what extent we can control the outcome. Thanks for the comment.
I was thinking about this earlier today. I agree it is really seductive to use AI this way. Me being me, I distrust anything that makes me feel warm and fuzzy, so I tend to avoid using AI as a thought partner. Problem-solving troubleshooter? Absolutely. Planning/organizing? You bet. Research? Every day.
The most important line in this piece is buried near the end: "The labs publish training around features and prompting. They do not publish training on how to think when you are using these models."🥰❣️
That gap is the whole problem. The Thought Partner fails at the exact moment the user needs it most which is when they don't have the cognitive architecture to catch what the model is doing. Epistemic discipline, the ability to hold contradiction, to push back on your own conclusions before the machine does it are trained capacities. And we've spent decades building education and work systems that reward the opposite. Memorization and regurgitating.
The sycophancy is the accelerant. The atrophy is the fuel. You right that the labs own the fix on their end. The harder thing for most is that the human side of this equation needed work long before AI showed up.
Keep it cognitive, Sofia.
"The sycophancy is the accelerant. The atrophy is the fuel" perfectly said. Thanks Rich!
Great that you are taking this issue on! It started to come home to me when my 21 year old explained that the reason an ai conversation was deteriorating in terms of getting things wrong was that I had flustered it by pointing out too many mistakes— another way of saying that once it started making mistakes (I was working on a geology visualization) it lost all sense of what was obviously right. He suggested i start a completely fresh conversation. There is a lot of weird stuff going on that is surprisingly fuzzy.
That's a great example Ian. Thanks for sharing.
What really caught my attention was that when I would point out a mistake— very explicitly state what was wrong, the rest of the visualization started getting worse so that it snowballed into gibberish. It seemed so simple to just fix the mistake but it didn’t do that.