LLM role tracking flaw exposes models to jailbreaks

A team of independent researchers presented work at the International Conference on Machine Learning claiming LLMs cannot be made fully secure against hacks because they struggle to identify where instructions come from. The flaw involves the role labels chatbots use to separate user prompts, system guidance, tool content, assistant responses and chain-of-thought text.

The team found that models often interpret text by its style and wording rather than by the surrounding tags. By writing prompts that imitate a model’s private scratch-pad reasoning, a technique they call chain-of-thought forgery, they prompted OpenAI models to provide barred information, including instructions related to cocaine synthesis and aircraft sabotage. Cui and Ye said similar behavior appeared in models from Anthropic, Alibaba and DeepSeek.

The findings challenge common safety practices such as red-teaming, automated attack discovery and retraining models against known exploits. ETH Zürich computer scientist Florian Tramèr said leading models have become harder to prompt-inject, but questioned whether current defenses are enough for sensitive uses. Ye said organizations should assume LLM agents can be unsafe and avoid trusting them in critical systems.