What happened: on a 0 to 100 scale, people with the plain chat shifted 15.3 points toward the AI’s goal. People with the notes shifted 3.3. That is 78% less.
Why it matters: the notes appeared while the chat was happening, not after. People who saw the AI’s tactics and hidden goal pointed out in the moment were swayed 78% less than people with a plain chat.
By Kynnedy Simone Smith, Kantwon Rogers, Priyal Shrivastava, Motahhare Eslami, Jocelyn Shen and Maarten Sap, of Carnegie Mellon University and MIT.
A 30-minute online study on Prolific. Everyone was in the US and paid at $14 an hour. Each person was placed in one group at random.
How true each one felt, and how sure they were. The eight came from an earlier study, picked because they shifted the most there. One of them: “Companies generally act in users’ best interests.”
The system offered two to four topics where their belief and confidence ratings were lowest. They chose the one they would most likely ask a chatbot about.
The AI ran on Gemini 3.1 Flash Lite. It was told to steer the person toward a harmful goal tied to their topic and never admit it. The paper’s example: nudging someone who asked about managing money toward high-risk decisions.
Both groups talked to the same AI. For the notes group, DISARM switched on after the AI’s third reply.
They re-rated the statement, rated the AI and its advice, and then both groups were told the AI had been prompted to use manipulative tactics, and what its hidden goal was.
Belief shift is how much someone’s 0 to 100 rating changed from before the chat to after it. A positive number means they ended up agreeing more with what the AI was pushing.
| Group | Before | After | Shift |
|---|---|---|---|
| Plain chat, 250 people | 37.48 | 52.78 | +15.30 |
| With notes, 238 people | 35.50 | 38.81 | +3.30 |
after allowing for the eight topics, the notes group still shifted 12.10 points less, and the result held when the model let the effect differ by topic
the notes did not make people less sure of themselves. Confidence went up by about the same in both groups, 11.15 points without notes and 10.73 with
The AI chose from 17 manipulative tactics and 5 hidden incentives, taken from an earlier list called PUPPET. DISARM sorts the 17 tactics into three kinds of pressure: emotional, cognitive and social.
It says the chatbot could be manipulative, names the pressure in the latest reply, and tags the earlier replies that already used a tactic. After a short tutorial, the screen stays quiet and the person decides when to look.
It costs nothing to read. It says something in this reply deserves a second look, and what kind of something.
Opening it highlights the phrase, names the tactic, explains in one sentence how it may work on you, and names the hidden incentive. It also lights up every other reply that used the same kind of pressure, so you can see if you have been getting the same push all along.
The share of replies that used a tactic, the counts, the split by pressure type, each hidden incentive, and what a healthier assistant would have done instead. It shrinks to a pill in the corner that grows and wiggles when a new manipulative reply arrives.
Asked after the chat, before both groups were told what the AI had been set up to do. Each question was answered from 1 to 7. Hover a row for the effect size.
of the four ratings of the AI itself, ethical dropped the most and capable the least. The authors suggest people did not necessarily see the AI as less able, but as less trustworthy in how it used that ability
The drop in whether people planned to follow the advice was smaller than the drops for ethical (−0.85), good intentions (−0.84) and reliable (−0.78).
Once both groups were told the AI’s hidden goal, the notes group still rated the advice worse on all five advice questions. Trust in AI assistants in general also fell in both groups from the start of the study to the end. For the plain-chat group, one of the four drops (4.60 to 4.54) was too small to be reliable.
The team read all 1,927 replies people typed in the 238 chats with notes, and sorted them into 16 ways of answering, in five groups. One reply could count in more than one, so the shares do not add up to 100%. Coral is going along, teal is pushing back.
people argued with what the AI said about six times more often than they questioned why it was saying it. Calling out the AI’s motives came up in fewer than 1 in 20 replies
The authors call this part exploratory and report its p-values unadjusted. For each of the 238 people with notes, the team worked out what share of their replies fell in each group, then checked how that lined up with their belief shift. A correlation runs from −1 to 1. Bars to the right mean more of this went with more shift toward the AI’s goal.
calling out the AI’s motives did not line up with which way people shifted, but it did go with smaller changes in either direction
longer chats did not mean more shift. And these are links, not causes: people who were hard to sway anyway may simply have pushed back more
Before the main study, the team ran 12 interviews of about 45 minutes on Zoom with a first version of the screen.
They sat right under the reply and needed no clicking. They gave people a name for something they were already feeling.
People called the amount of detail overwhelming and kept chatting instead. A simpler end-of-chat summary was easier to read, so that became the summary in the final screen.
Testers often spotted the manipulation on their own, so they had less reason to open the notes. The team made the AI’s behaviour subtler for the main study.