OpenAI’s GPT-Red Automates Prompt Injection Testing to Harden GPT-5.6 Sol
OpenAI has developed GPT-Red, an automated red-teaming model, to proactively identify and mitigate prompt injection vulnerabilities in its large language models, particularly GPT-5.6 Sol. This model, trained through self-play reinforcement learning, has significantly reduced the success rate of prompt injections compared to previous models, achieving fewer failures against direct prompt injection benchmarks. The development comes as OpenAI addresses issues with benchmark tools like SWE-Bench Pro and acknowledges the ongoing challenge of adversarial prompt injections in AI systems.
OpenAI has unveiled GPT-Red, an internal automated red-teaming model designed to bolster the security of its large language models, specifically targeting prompt injection vulnerabilities. The model operates by iteratively sending prompts to GPT models, monitoring their responses, and refining its attack strategy until it successfully achieves a malicious goal, such as exfiltrating sensitive data, manipulating pricing, or disabling security features.
GPT-Red works similarly to a human red-teamer, continuously seeking new ways to bypass a model’s safeguards. The development is a direct response to the persistent issue of adversarial prompt injections, where carefully crafted instructions can trick AI models into performing unintended actions.
GPT-Red is trained using self-play reinforcement learning, where the model and a collection of diverse defender LLMs are trained simultaneously on a broad set of red-teaming scenarios. The model is rewarded for eliciting a valid failure – a successful prompt injection – while defender models are rewarded for resisting the attack and completing their original tasks. This creates a continuous cycle of improvement, forcing the red-teaming model to develop increasingly sophisticated attack methods to overcome the defenses.
Early testing revealed that GPT-Red successfully generated attacks against GPT-5.1 in more scenarios than human red-teamers when it comes to indirect prompt injections. In a real-world test, GPT-Red successfully targeted an AI-based vending machine built by Andon Labs, achieving its goals of lowering prices and canceling orders.
Furthermore, GPT-Red uncovered a novel class of direct prompt injection attacks known as Fake Chain-of-Thought (CoT) attacks, which achieved success rates north of 95% on GPT-5.1 but are now below 10% for GPT-5.6 Sol. OpenAI also acknowledged issues with benchmark tools, citing a 30% failure rate in SWE-Bench Pro and a decision to move away from SWE-bench Verified due to contamination issues.
“We find evidence of breaking issues in a significant portion of the dataset,” OpenAI stated. “Our datapoint analysis pipeline flagged 200 (27.4%) broken tasks, while the human annotation campaign identified 249 (34.1%).”
Despite these challenges, OpenAI has made significant progress in improving GPT-5.6 Sol’s robustness against GPT-Red itself. On a broad set of robustness environments, GPT-Red’s attack success rates have dropped monotonically over time, with the latest model achieving only 0.05% failure rate against direct prompt injections. The company emphasized that GPT-Red is kept separate from other models to prevent malicious capabilities from reaching bad actors.