Researchers expose role confusion behind LLM prompt injection

Security researchers Charles Ye and Jasmine Cui, with MIT associate professor Dylan Hadfield-Menell, argue that modern LLM defenses remain vulnerable because models cannot reliably separate authorized instructions from adversarial input. Their paper, “Prompt Injection as Role Confusion,” says role tags originally used to format system, user, assistant, tool, and think content have become overloaded as a security mechanism.

The researchers contend that LLMs often infer roles from writing style rather than secure tags, making them susceptible when attackers imitate trusted formats. Their CoT Forgery technique spoofs the terse style of OpenAI’s think mode inside a user prompt, leading models to treat unsafe instructions as if they were already accepted internal reasoning.

The attack won the 2025 OpenAI Kaggle red-teaming contest. On a standard jailbreaking benchmark, CoT Forgery pushed attack success from near zero to about 60 percent on tested models, while the researchers noted that human red-teamers can reach attack success rates close to 100 percent despite strong benchmark safety scores. They conclude that prompt-injection defense will remain a whack-a-mole problem unless LLMs develop more reliable role perception.