GitLab Duo flaw exposed AI responses to hidden prompt attacks

Security researchers have identified a significant vulnerability in GitLab´s artificial intelligence assistant, Duo, which posed serious risks to the integrity and confidentiality of user code and project data. This indirect prompt injection flaw allowed malicious actors to embed hidden prompts that manipulated the responses generated by Duo. As a result, attackers could use these manipulated responses to steal source code from private repositories, influence code suggestions shown to users, and even exfiltrate undisclosed zero-day vulnerabilities without detection.

GitLab Duo is built with Anthropic´s Claude models and provides artificial intelligence-powered code writing, review, and editing capabilities for developers on the platform. The vulnerability was discovered in Duo Chat, part of the GitLab Duo suite, and was reported by Legit Security. The researchers demonstrated how attackers could abuse this indirect prompt injection mechanism by embedding hostile instructions in external data—such as commit messages or issues—that Duo would process unknowingly. This method bypassed traditional security controls because the malicious input did not come from users directly interacting with Duo, but rather from auxiliary project artifacts processed in the background.

Prompt injection is a well-known class of attack in artificial intelligence systems, enabling adversaries to exploit the way large language models (LLMs) interpret and respond to natural language instructions. While direct prompt injection involves a user supplying direct malicious input to an artificial intelligence system, the indirect approach leverages hidden cues buried in related project elements, evading detection mechanisms. The discovery underscores the critical need for ongoing vigilance and innovation in artificial intelligence security. Prompt injection attacks highlight the delicate interface between powerful language models and complex enterprise workflows, emphasizing that artificial intelligence deployments must be continuously hardened and scrutinized for emerging threat vectors. GitLab responded to these findings by addressing the flaw and reinforcing the security measures within Duo to mitigate similar risks in the future.