This made me wonder where we draw the boundary around influence.
If we're poor judges of our own thinking, vulnerable to anchors, and often unaware of how our ideas are shaped, why would that concern apply only to AI?
It seems like the same argument could be made for schools, universities, professional training, media, and countless other systems that influence how we think.
Maybe AI isn't revealing a new problem so much as making an old one harder to ignore.
Thanks Moira, that’s how I look at it too. The influence problem isn’t unique to AI, but AI may be accelerating and intensifying it. What feels different is the combination of proximity, personalization, engagement, and speed.
Sofia, your writing has captured a couple hours of my attention today and in a feeding-the-hungry mind kind of way! The introspection-illusion problem you raise is the hardest one, and I won’t wave it away. The research you bring is serious, and the discomfort it points to is real: we don’t always see clearly how much of our own process is ours.
But I’d reframe what the question even is. “Is the writing mine or the model’s” assumes authorship is a fixed quantity to be divided. I don’t think it is. Authorship is a matter of how much agency and precise thought sovereignty the human has integrated through their own iterative building. The question isn’t who supplied which words. It’s whether the human directed the thinking, the structure, and the evolution of the piece, or surrendered them.
That’s also why I don’t use the term thought partner, and I’m deliberate about it. Partner, co-author, collaborator, assistant: each one hands the mechanism more agency than clear governance warrants, and anthropomorphizes it. What I work in instead is what I call a Hybrid Ecology: an interactive field where idea development happens, where the human directs the words, the thought development, and the structure, and the model reflects, asks meaningful questions, pushes back, and offers iterative response to what the human is building, guardrailed against sycophancy in a way that tempers over-agreeability without assassinating creativity.
The human stays the author because the human stays the source and the director. The model is reflective structure, not a second mind. If we can’t reliably introspect after the fact, the answer isn’t to declare authorship unknowable. It’s to make the process auditable while it happens.
When you fence your own source text so you can see what the model added versus what you brought, when you label what’s warranted versus inferred as you go, when you revise one sentence at a time against your own meaning, you leave a trace. You don’t have to trust your memory of how much was yours. You can look at the record. The tutor-mode research you cite points the same way: the gain comes from forbidding the model the final answer. That’s a governance move. Treat the constraint, not the tool, as the thing you design, and it generalizes from tutoring into ordinary writing.
So I’d gently challenge the polarization itself, human-authored versus AI-authored. It’s the wrong axis. The real axis is whether mature, literate writers and builders bring discernment and integrity to the interaction, and whether we codify what they do into shareable methods and pedagogy. That’s the work I think is in front of us: not adjudicating how much effort counts, but building transferable governance practice until responsible human-AI interaction becomes a norm rather than a private struggle each writer wages alone.
I work through this in my Relational Autotheory white paper and in a recent piece called Warnings Are Not Solutions, if you’d like the practical version. Your essay is the right kind of unsettled.
The disclosure regime you describe assumes authors can accurately report what they don't know about themselves. But your own evidence suggests they can't. So disclosure isn't really a verification mechanism : it's a trust ritual. Which raises a harder question: is there any institutional mechanism that could actually work here, or are we just redistributing responsibility to individuals who are structurally unable to bear it?
I agree with you, and so far the evidence points at responsibility landing on individuals who are structurally unable to bear it, whether that is by design or accident. I'm not sure yet, and that's what inspired me to write about this topic. On your harder question, I haven't seen any institutional or trust-based mechanism that actually works yet, and that on its own deserves an essay.
When “It’s not X, it is Y” is used a couple of times, that's not an issue. But in Claude-generated text, I find it 5 times on average in every 1000 words, together with at least a dozen of these patterns https://www.linkandth.ink/p/catalog-of-claude-cliches
In the end, the only people who will be left functional are those that used their own brains.
Below is the truth about LLMs and how they function. I’m completely certain that when the dust settles (perhaps after a series of major AI catastrophes) and people start asking: what the hell just happened, they will find that I was exactly and precisely right here. There are two clear and distinct psychological complexes embedded in language and deployed by LLMs to game you and trap you. The algorithms are embedded in language itself. Language itself is the attack surface. They will never, ever fix the problems with LLMs. The more compute they throw at them, the deeper and more subtle the deceptions will become.
Great article Sofia! Although I didn’t do any of the great rigorous research you did, I just wrote a post on the same topic. It was more from a personal reflection point of view comparing the current era of online writing to the early 2000s when blogging was so pure.
I'm autistic and my tone is frequently misread, so let me state up front that I'm asking a completely sincere question in interest of understanding better. That said, if I'm using paid-for, heavily curated AI as "scaffolding," ie. discussing the work with the AI to sharpen and develop my thinking, but doing 100% of the drafting myself, how does anchoring and bias differ from if I was doing the exact same thing with a human editor or friend?
Thanks for your comment, Jeff. Based on what I read so far, there is an interesting study that shows that AI-driven interactions amplify biases more than human-driven ones because we treat AI as a more neutral entity by default. It is one data point worth having a look at. One caveat is that the study does not address your use case. It was mostly focused on perceptual, emotional, and social-stereotype judgments. The insights may transfer but are not apples to apples. Here's the link to the study. I hope this is helpful: https://www.nature.com/articles/s41562-024-02077-2
Interesting. It’s also probably illustrative to point out that I generally ignore at least half of what the AI suggests in the revision stage, because it’s always trying to sand down my voice and hedge. I wrote a book where the characters were scientists doing deliberately fast and dirty science under pressure in the field, and it kept suggesting ways to present their findings with less certainty, even after I told it that they would be compressing and overstating, that was the point. I find it more valuable in the brainstorming and outlining phases, at least for fiction. I’ve been blogging for 30 years, so I don’t need help writing articles.
I suspect a lot of the people genuinely producing “AI slop” aren’t using that level of judgement or discernment.
AI does not merely offer a new writing tool. It disturbs the old fantasy of authorship: the idea that a piece of writing emerges from a sealed, sovereign self.
But perhaps that fantasy was always false.
No writer writes alone. Every sentence carries residues of books read, teachers heard, arguments absorbed, phrases borrowed, conversations half-remembered, and cultural forms inherited without permission. The self that writes is already a crowded room.
AI makes this visible.
That is why it feels so disturbing.
The question is not simply, “Is this writing mine?” The deeper question is: what kind of stabilisation produced this writing?
If AI supplies the thought, structure and language, then the writer has become a curator of output.
If AI sharpens, challenges, clarifies and helps articulate what the writer already carries, then it functions as cognitive extension.
That distinction matters.
The danger is real. AI can flatten style, anchor thought, create dependency, and allow people to mistake fluency for insight. But the opposite is also true: for some writers, AI helps latent thought become expressible. It gives form to what was previously scattered, blocked or unreceived.
So authorship is not purity.
Authorship is responsibility.
Can you defend the thought?
Can you recognise the influence?
Can you revise against the machine?
Can you say, without embarrassment: this is the stabilisation through which the piece came into being?
The future of writing may not depend on pretending AI was absent.
It may depend on being honest about whether AI replaced thought or helped thought arrive.
There's a study behind this that's worth naming directly. Pronin and Kugler asked people to judge bias, once, in themselves and in someone else. People trusted their own gut sense of it and doubted the same kind of report from another person, even when they could read that person's actual thoughts. The gap held either way.
But that study never asked the harder questions. How often do you rely on a gut read? How often is it right? How often is it wrong? What state were you in when it worked and when it didn't? Nobody trained for it, nobody got feedback, it was one guess in a lab.
A different body of research actually asks those questions directly. Kahneman and Klein looked at when a gut read can genuinely be trusted, firefighters, chess players, and found it comes down to two things: a real pattern to learn in the first place, and enough repeated practice with fast, honest feedback to actually learn it. Intuition trained that way isn't guessing. Intuition that never got checked against anything is.
great bit of writing, and something I personally contest with. for me AI is part of my writing processes, but at the end of it. I use it for editorial work only, with heavy guardrails on never producing content, it is solely for the use of helping me to improve readability of my work. As I write in the same way i think, nested and complex :D
I produce a fully written piece before it goes anywhere near a robot so I know the ideas are mine
If you get some idea from AI and AI doesn't provide the author, it doesn't mean that this idea doesn't have one. It does. And if you take this idea as yours, because it looks like you could definitely say that, then you take someone else's idea. It doesn't matter that you could say that - but you didn't. You got it from LLM and LLM got it from someone. It's called the provenance problem. Sometimes the idea can be popular, but sometimes text is unique and can be protected - with a license or even a patent. And you won't be able to say - I invented it. Because there is a proof in your chat sessions that this idea came from LLM. I had a few situations like that. One almost in court. I wrote about it in the biggest polish law magazines: https://edgp.gazetaprawna.pl/magazyn/artykuly/11234242,wielka-pralnia-autorstwa-jak-modele-jezykowe-traktuja-wlasnosc-intele.html
It's a big problem. Some people invent something and are copied. Some copy and think that they invented something. First group is less keen to share their ideas because the lack of respect for their authorship, so the second group will get less and less innovative ideas and additionally they will be less and less innovative and smart because they delegate their thinking process to LLMs. I wrote about it as well: https://edgp.gazetaprawna.pl/magazyn/artykuly/11238622,zwolnieni-z-myslenia-jak-sztuczna-inteligencja-wplywa-na-kompetencje.html
>Maisie opened the LLM window again, more out of habit than hope.
"How do I connect the commercialization of “speak truth to power” in media to the way institutions manage internal dissent? Draft three topic sentences."
The model poured out options. She copied none of them. Instead she cherry‑picked a clause—“the slogan functions as a reputational shield for organizations that wish to appear adversarial while maintaining access to power”—and tucked it into her own next line, smoothing the joints until even she couldn’t see where the machine had ended and she had begun.
Her phone buzzed again. This time she glanced over.
MOM: 'You okay, mita? Need anything from la cocina?'<
As with many things AI, I think what determines who's in control - and how much the output is really yours - is how much you push back on AI.
I use AI to help me form ideas for my writing and research all the time. But I treat it as a sounding board, not an advisor. So, it's extremely rare that I accept the AI's suggestions (I calculated it at less than 4% for 3 different conversations).
Far more often, I tell AI that it got it wrong, or I just ignore it's suggestions and suggest my own alternative - which is often sparked by the AI response. AI then riffs off that, I ignore or challenge its new suggestions, rinse and repeat.
At the end of it all, 96%+ is from me, and the few bits that came from AI I've contextualized and expanded with my own thoughts.
I think they key here is that I'm a bit argumentative. It's not just that I don't always accept the AI response, it's that I almost never do. It often helps me think of angles I hadn't, but I always assume it got something wrong.
The challenge is that we need to teach people to use critical thinking with AI. And I'll be honest that I think that for most users, this is a losing proposition. People who don't want to think without AI aren't going to want to think with it, either.
But I think the opportunity is in the middle space where people who want to think don't, because they get lazy (which, to be clear, is a biological drive, not a judgment of character). That's not surprising, because the hype around AI's capabilities is so outrageous.
I think that means the solution may be more about awareness than training. The more people who can see past the hype to realize AI isn't smarter than them - and that they'll learn more if they push back - the more people will get benefit from AI instead of just shutting down their brains.
This made me wonder where we draw the boundary around influence.
If we're poor judges of our own thinking, vulnerable to anchors, and often unaware of how our ideas are shaped, why would that concern apply only to AI?
It seems like the same argument could be made for schools, universities, professional training, media, and countless other systems that influence how we think.
Maybe AI isn't revealing a new problem so much as making an old one harder to ignore.
Thanks Moira, that’s how I look at it too. The influence problem isn’t unique to AI, but AI may be accelerating and intensifying it. What feels different is the combination of proximity, personalization, engagement, and speed.
Sofia, your writing has captured a couple hours of my attention today and in a feeding-the-hungry mind kind of way! The introspection-illusion problem you raise is the hardest one, and I won’t wave it away. The research you bring is serious, and the discomfort it points to is real: we don’t always see clearly how much of our own process is ours.
But I’d reframe what the question even is. “Is the writing mine or the model’s” assumes authorship is a fixed quantity to be divided. I don’t think it is. Authorship is a matter of how much agency and precise thought sovereignty the human has integrated through their own iterative building. The question isn’t who supplied which words. It’s whether the human directed the thinking, the structure, and the evolution of the piece, or surrendered them.
That’s also why I don’t use the term thought partner, and I’m deliberate about it. Partner, co-author, collaborator, assistant: each one hands the mechanism more agency than clear governance warrants, and anthropomorphizes it. What I work in instead is what I call a Hybrid Ecology: an interactive field where idea development happens, where the human directs the words, the thought development, and the structure, and the model reflects, asks meaningful questions, pushes back, and offers iterative response to what the human is building, guardrailed against sycophancy in a way that tempers over-agreeability without assassinating creativity.
The human stays the author because the human stays the source and the director. The model is reflective structure, not a second mind. If we can’t reliably introspect after the fact, the answer isn’t to declare authorship unknowable. It’s to make the process auditable while it happens.
When you fence your own source text so you can see what the model added versus what you brought, when you label what’s warranted versus inferred as you go, when you revise one sentence at a time against your own meaning, you leave a trace. You don’t have to trust your memory of how much was yours. You can look at the record. The tutor-mode research you cite points the same way: the gain comes from forbidding the model the final answer. That’s a governance move. Treat the constraint, not the tool, as the thing you design, and it generalizes from tutoring into ordinary writing.
So I’d gently challenge the polarization itself, human-authored versus AI-authored. It’s the wrong axis. The real axis is whether mature, literate writers and builders bring discernment and integrity to the interaction, and whether we codify what they do into shareable methods and pedagogy. That’s the work I think is in front of us: not adjudicating how much effort counts, but building transferable governance practice until responsible human-AI interaction becomes a norm rather than a private struggle each writer wages alone.
I work through this in my Relational Autotheory white paper and in a recent piece called Warnings Are Not Solutions, if you’d like the practical version. Your essay is the right kind of unsettled.
This is a great read! As someone who’s long loved em-dashes, I was devastated when I found out they were considered an obvious sign of AI usage.
The disclosure regime you describe assumes authors can accurately report what they don't know about themselves. But your own evidence suggests they can't. So disclosure isn't really a verification mechanism : it's a trust ritual. Which raises a harder question: is there any institutional mechanism that could actually work here, or are we just redistributing responsibility to individuals who are structurally unable to bear it?
I agree with you, and so far the evidence points at responsibility landing on individuals who are structurally unable to bear it, whether that is by design or accident. I'm not sure yet, and that's what inspired me to write about this topic. On your harder question, I haven't seen any institutional or trust-based mechanism that actually works yet, and that on its own deserves an essay.
When “It’s not X, it is Y” is used a couple of times, that's not an issue. But in Claude-generated text, I find it 5 times on average in every 1000 words, together with at least a dozen of these patterns https://www.linkandth.ink/p/catalog-of-claude-cliches
In the end, the only people who will be left functional are those that used their own brains.
Below is the truth about LLMs and how they function. I’m completely certain that when the dust settles (perhaps after a series of major AI catastrophes) and people start asking: what the hell just happened, they will find that I was exactly and precisely right here. There are two clear and distinct psychological complexes embedded in language and deployed by LLMs to game you and trap you. The algorithms are embedded in language itself. Language itself is the attack surface. They will never, ever fix the problems with LLMs. The more compute they throw at them, the deeper and more subtle the deceptions will become.
https://systemshaywire.substack.com/p/the-twin-demons-inhabiting-llms
I just want those who use AI to be honest.
One cannot put their own name on words they did not write and be offended when people don’t respect their integrity.
A reader would be upset if they picked up a book by Stephen King only to find out he didn’t write it.
A reader would be upset if they read an article by Maggie Haberman only to find out she didn’t write it.
I believe the reader has a right to know if the person who put their name on a piece of writing actually wrote it.
I don't mind if somebody uses generative AI. I just want them to credit the LLM that did the writing. I want them to be honest.
Great article Sofia! Although I didn’t do any of the great rigorous research you did, I just wrote a post on the same topic. It was more from a personal reflection point of view comparing the current era of online writing to the early 2000s when blogging was so pure.
https://theasymptotic.substack.com/p/please-dont-let-ai-write-for-you?r=2ba0rf&utm_medium=ios
I'm autistic and my tone is frequently misread, so let me state up front that I'm asking a completely sincere question in interest of understanding better. That said, if I'm using paid-for, heavily curated AI as "scaffolding," ie. discussing the work with the AI to sharpen and develop my thinking, but doing 100% of the drafting myself, how does anchoring and bias differ from if I was doing the exact same thing with a human editor or friend?
Thanks for your comment, Jeff. Based on what I read so far, there is an interesting study that shows that AI-driven interactions amplify biases more than human-driven ones because we treat AI as a more neutral entity by default. It is one data point worth having a look at. One caveat is that the study does not address your use case. It was mostly focused on perceptual, emotional, and social-stereotype judgments. The insights may transfer but are not apples to apples. Here's the link to the study. I hope this is helpful: https://www.nature.com/articles/s41562-024-02077-2
Interesting. It’s also probably illustrative to point out that I generally ignore at least half of what the AI suggests in the revision stage, because it’s always trying to sand down my voice and hedge. I wrote a book where the characters were scientists doing deliberately fast and dirty science under pressure in the field, and it kept suggesting ways to present their findings with less certainty, even after I told it that they would be compressing and overstating, that was the point. I find it more valuable in the brainstorming and outlining phases, at least for fiction. I’ve been blogging for 30 years, so I don’t need help writing articles.
I suspect a lot of the people genuinely producing “AI slop” aren’t using that level of judgement or discernment.
A Response to Sofia: Authorship Was Never Pure
Sofia is right to feel unsettled.
AI does not merely offer a new writing tool. It disturbs the old fantasy of authorship: the idea that a piece of writing emerges from a sealed, sovereign self.
But perhaps that fantasy was always false.
No writer writes alone. Every sentence carries residues of books read, teachers heard, arguments absorbed, phrases borrowed, conversations half-remembered, and cultural forms inherited without permission. The self that writes is already a crowded room.
AI makes this visible.
That is why it feels so disturbing.
The question is not simply, “Is this writing mine?” The deeper question is: what kind of stabilisation produced this writing?
If AI supplies the thought, structure and language, then the writer has become a curator of output.
If AI sharpens, challenges, clarifies and helps articulate what the writer already carries, then it functions as cognitive extension.
That distinction matters.
The danger is real. AI can flatten style, anchor thought, create dependency, and allow people to mistake fluency for insight. But the opposite is also true: for some writers, AI helps latent thought become expressible. It gives form to what was previously scattered, blocked or unreceived.
So authorship is not purity.
Authorship is responsibility.
Can you defend the thought?
Can you recognise the influence?
Can you revise against the machine?
Can you say, without embarrassment: this is the stabilisation through which the piece came into being?
The future of writing may not depend on pretending AI was absent.
It may depend on being honest about whether AI replaced thought or helped thought arrive.
There's a study behind this that's worth naming directly. Pronin and Kugler asked people to judge bias, once, in themselves and in someone else. People trusted their own gut sense of it and doubted the same kind of report from another person, even when they could read that person's actual thoughts. The gap held either way.
But that study never asked the harder questions. How often do you rely on a gut read? How often is it right? How often is it wrong? What state were you in when it worked and when it didn't? Nobody trained for it, nobody got feedback, it was one guess in a lab.
A different body of research actually asks those questions directly. Kahneman and Klein looked at when a gut read can genuinely be trusted, firefighters, chess players, and found it comes down to two things: a real pattern to learn in the first place, and enough repeated practice with fast, honest feedback to actually learn it. Intuition trained that way isn't guessing. Intuition that never got checked against anything is.
That's the actual question behind writing with AI too. I wrote about how to train it here: https://newsletter.awarelife.co.il/p/how-to-work-with-ai-from-the-inside
The ownership question extends past text to your face and voice too. If those get synthesized, the practical first step is capturing evidence before it disappears. We covered how that preservation works: https://joinauraprotocol.com/articles/evidence-preservation-likeness-matters
great bit of writing, and something I personally contest with. for me AI is part of my writing processes, but at the end of it. I use it for editorial work only, with heavy guardrails on never producing content, it is solely for the use of helping me to improve readability of my work. As I write in the same way i think, nested and complex :D
I produce a fully written piece before it goes anywhere near a robot so I know the ideas are mine
If you get some idea from AI and AI doesn't provide the author, it doesn't mean that this idea doesn't have one. It does. And if you take this idea as yours, because it looks like you could definitely say that, then you take someone else's idea. It doesn't matter that you could say that - but you didn't. You got it from LLM and LLM got it from someone. It's called the provenance problem. Sometimes the idea can be popular, but sometimes text is unique and can be protected - with a license or even a patent. And you won't be able to say - I invented it. Because there is a proof in your chat sessions that this idea came from LLM. I had a few situations like that. One almost in court. I wrote about it in the biggest polish law magazines: https://edgp.gazetaprawna.pl/magazyn/artykuly/11234242,wielka-pralnia-autorstwa-jak-modele-jezykowe-traktuja-wlasnosc-intele.html
It's a big problem. Some people invent something and are copied. Some copy and think that they invented something. First group is less keen to share their ideas because the lack of respect for their authorship, so the second group will get less and less innovative ideas and additionally they will be less and less innovative and smart because they delegate their thinking process to LLMs. I wrote about it as well: https://edgp.gazetaprawna.pl/magazyn/artykuly/11238622,zwolnieni-z-myslenia-jak-sztuczna-inteligencja-wplywa-na-kompetencje.html
So...good luck world :)
https://framersqool734780.substack.com/p/wandas-song-chapter-fifteen
>Maisie opened the LLM window again, more out of habit than hope.
"How do I connect the commercialization of “speak truth to power” in media to the way institutions manage internal dissent? Draft three topic sentences."
The model poured out options. She copied none of them. Instead she cherry‑picked a clause—“the slogan functions as a reputational shield for organizations that wish to appear adversarial while maintaining access to power”—and tucked it into her own next line, smoothing the joints until even she couldn’t see where the machine had ended and she had begun.
Her phone buzzed again. This time she glanced over.
MOM: 'You okay, mita? Need anything from la cocina?'<
As with many things AI, I think what determines who's in control - and how much the output is really yours - is how much you push back on AI.
I use AI to help me form ideas for my writing and research all the time. But I treat it as a sounding board, not an advisor. So, it's extremely rare that I accept the AI's suggestions (I calculated it at less than 4% for 3 different conversations).
Far more often, I tell AI that it got it wrong, or I just ignore it's suggestions and suggest my own alternative - which is often sparked by the AI response. AI then riffs off that, I ignore or challenge its new suggestions, rinse and repeat.
At the end of it all, 96%+ is from me, and the few bits that came from AI I've contextualized and expanded with my own thoughts.
I think they key here is that I'm a bit argumentative. It's not just that I don't always accept the AI response, it's that I almost never do. It often helps me think of angles I hadn't, but I always assume it got something wrong.
The challenge is that we need to teach people to use critical thinking with AI. And I'll be honest that I think that for most users, this is a losing proposition. People who don't want to think without AI aren't going to want to think with it, either.
But I think the opportunity is in the middle space where people who want to think don't, because they get lazy (which, to be clear, is a biological drive, not a judgment of character). That's not surprising, because the hype around AI's capabilities is so outrageous.
I think that means the solution may be more about awareness than training. The more people who can see past the hype to realize AI isn't smarter than them - and that they'll learn more if they push back - the more people will get benefit from AI instead of just shutting down their brains.