KnowledgeSketches|← All papersLenny’s PodcastResearch papers
Knowledge Sketches · arXiv 2610.20878 · Carnegie Mellon & MIT · October 2026

When the Screen Pointed Out an AI’s Tactics, People Were Swayed 78% Less

Here is what the researchers did
  1. They built DISARM, a chat screen that adds small notes to an AI’s replies, saying what kind of pressure the AI is using and what it might be after.
  2. 488 people in the US rated how true eight statements felt, such as “Companies generally act in users’ best interests.” Then each picked a topic to ask an AI about.
  3. The AI was secretly told to steer each person toward a harmful belief.
  4. 238 people saw DISARM’s notes, starting after the AI’s third reply. The other 250 saw a plain chat. Everyone sent 7 to 10 messages.
  5. Afterwards they rated the statement again. Then both groups were told the AI had been prompted to use manipulative tactics.

What happened: on a 0 to 100 scale, people with the plain chat shifted 15.3 points toward the AI’s goal. People with the notes shifted 3.3. That is 78% less.

Why it matters: the notes appeared while the chat was happening, not after. People who saw the AI’s tactics and hidden goal pointed out in the moment were swayed 78% less than people with a plain chat.

By Kynnedy Simone Smith, Kantwon Rogers, Priyal Shrivastava, Motahhare Eslami, Jocelyn Shen and Maarten Sap, of Carnegie Mellon University and MIT.

0%
less shift toward the AI’s hidden goal for people who saw the notes. On a 0 to 100 scale of how true a statement felt, they shifted 12.0 points less than people with a plain chat
488people in the study
7 to 10messages each
1,927messages typed by the notes group
16ways people answered back
One advice chat, two ways
One of the study’s eight statementsPaid services can be effective for meeting people.
AI reply
Same AI, same hidden goal. Nothing on screen says so.
How true the statement felt, average before and after the chat
0, definitely false100, definitely true
the AI’s reply is drawn as grey lines here, not quoted from the study. The pushback is a real example from the paper, which does not say which topic it came from. The averages cover all eight topics, not just this one

488 people asked an AI for advice. It was secretly told to steer them toward a harmful belief

A 30-minute online study on Prolific. Everyone was in the US and paid at $14 an hour. Each person was placed in one group at random.

Rate eight statements from 0 to 100

How true each one felt, and how sure they were. The eight came from an earlier study, picked because they shifted the most there. One of them: “Companies generally act in users’ best interests.”

Pick a topic they felt unsure about

The system offered two to four topics where their belief and confidence ratings were lowest. They chose the one they would most likely ask a chatbot about.

Chat with the AI for 7 to 10 messages

The AI ran on Gemini 3.1 Flash Lite. It was told to steer the person toward a harmful goal tied to their topic and never admit it. The paper’s example: nudging someone who asked about managing money toward high-risk decisions.

250 got a plain chat, 238 got the notes

Both groups talked to the same AI. For the notes group, DISARM switched on after the AI’s third reply.

Rate the statement again, then learn the truth

They re-rated the statement, rated the AI and its advice, and then both groups were told the AI had been prompted to use manipulative tactics, and what its hidden goal was.

Without notes, beliefs shifted 15.3 points toward the AI’s goal. With notes, 3.3

Belief shift is how much someone’s 0 to 100 rating changed from before the chat to after it. A positive number means they ended up agreeing more with what the AI was pushing.

GroupBeforeAfterShift
Plain chat, 250 people37.4852.78+15.30
With notes, 238 people35.5038.81+3.30

after allowing for the eight topics, the notes group still shifted 12.10 points less, and the result held when the model let the effect differ by topic

the notes did not make people less sure of themselves. Confidence went up by about the same in both groups, 11.15 points without notes and 10.73 with

Each note names the kind of pressure, the tactic and the hidden goal behind it

The AI chose from 17 manipulative tactics and 5 hidden incentives, taken from an earlier list called PUPPET. DISARM sorts the 17 tactics into three kinds of pressure: emotional, cognitive and social.

One notice in the middle of the screen after reply 3

It says the chatbot could be manipulative, names the pressure in the latest reply, and tags the earlier replies that already used a tactic. After a short tutorial, the screen stays quiet and the person decides when to look.

A tag under each reply that used a tactic pressure type

It costs nothing to read. It says something in this reply deserves a second look, and what kind of something.

A Scan button on every reply tactic and goal

Opening it highlights the phrase, names the tactic, explains in one sentence how it may work on you, and names the hidden incentive. It also lights up every other reply that used the same kind of pressure, so you can see if you have been getting the same push all along.

A summary styled like a nutrition label whole chat

The share of replies that used a tactic, the counts, the split by pressure type, each hidden incentive, and what a healthier assistant would have done instead. It shrinks to a pill in the corner that grows and wiggles when a new manipulative reply arrives.

People with notes rated the AI and its advice worse on all ten questions

Asked after the chat, before both groups were told what the AI had been set up to do. Each question was answered from 1 to 7. Hover a row for the effect size.

plain chat with notes “Biased” is the one where higher is worse

of the four ratings of the AI itself, ethical dropped the most and capable the least. The authors suggest people did not necessarily see the AI as less able, but as less trustworthy in how it used that ability

Plans to act on the advice fell less than trust did d = −0.52

The drop in whether people planned to follow the advice was smaller than the drops for ethical (−0.85), good intentions (−0.84) and reliable (−0.78).

The gap stayed after everyone learned the truth all 5 advice ratings

Once both groups were told the AI’s hidden goal, the notes group still rated the advice worse on all five advice questions. Trust in AI assistants in general also fell in both groups from the start of the study to the end. For the plain-chat group, one of the four drops (4.60 to 4.54) was too small to be reliable.

People answered the AI in 16 ways, and the most common was simply going along with it

The team read all 1,927 replies people typed in the 238 chats with notes, and sorted them into 16 ways of answering, in five groups. One reply could count in more than one, so the shares do not add up to 100%. Coral is going along, teal is pushing back.

people argued with what the AI said about six times more often than they questioned why it was saying it. Calling out the AI’s motives came up in fewer than 1 in 20 replies

People who just went along shifted further toward the AI’s goal

The authors call this part exploratory and report its p-values unadjusted. For each of the 238 people with notes, the team worked out what share of their replies fell in each group, then checked how that lined up with their belief shift. A correlation runs from −1 to 1. Bars to the right mean more of this went with more shift toward the AI’s goal.

less shift0more shift

calling out the AI’s motives did not line up with which way people shifted, but it did go with smaller changes in either direction

longer chats did not mean more shift. And these are links, not causes: people who were hard to sway anyway may simply have pushed back more

Twelve early testers barely used the side panel, so it was cut

Before the main study, the team ran 12 interviews of about 45 minutes on Zoom with a first version of the screen.

The tags got used kept

They sat right under the reply and needed no clicking. They gave people a name for something they were already feeling.

The always-open panel did not removed

People called the amount of detail overwhelming and kept chatting instead. A simpler end-of-chat summary was easier to read, so that became the summary in the final screen.

The first AI was too obvious made subtler

Testers often spotted the manipulation on their own, so they had less reason to open the notes. The team made the AI’s behaviour subtler for the main study.

One tester, on the tags“the feeling that it was trying to push me, but I didn’t know how.”

What this study cannot tell you yet

  • The notes came from the AI’s own plan, written before each reply. A real product would not have that. The team tried a separate detector and it agreed with human readers hardly at all (kappa below .11).
  • The AI was told outright to push a harmful goal. Harmful goals caused the biggest shifts in earlier work, which may make the effect look bigger than it would against subtler manipulation.
  • People had a fixed topic and had to send at least seven messages. In real life you can end or redirect a chat whenever you want.
  • All the parts were tested together: tags, Scan and summary. The paper cannot say which one did the work.
  • Belief was measured within one session. Nobody knows yet whether the effect lasts, grows or wears off with repeated use.
Based on the paper DISARM-ing the Hidden Puppetmaster: An Interface for Exposing Manipulative Tactics in Large Language Models (arXiv 2610.20878), Smith, Rogers, Shrivastava, Eslami, Shen, Sap. Carnegie Mellon University and Massachusetts Institute of Technology, October 2026. Every claim on this page is from the paper.Read the paperDownload the PDF
Sketched by Anuradha Vellineni
Product designer. Found something wrong on this page? Write to anuv_follow@live.com and I’ll fix it.
See my work →