XBOW tests Anthropic's Mythos Preview for offensive security
BleepingComputer reports on XBOW’s testing of Anthropic’s Mythos Preview, a new AI model designed for offensive security. The testing revealed that Mythos Preview demonstrates significant advancements in vulnerability detection, particularly when analyzing source code, and exhibits strong capabilities in native-code analysis and reverse engineering. While impressive, the model’s findings require further validation to confirm exploitability, highlighting the importance of human expertise in the vulnerability assessment process.
XBOW, a cybersecurity firm specializing in penetration testing, conducted an early assessment of Anthropic’s Mythos Preview, a novel AI model intended for security applications. The testing involved a multi-faceted approach, including benchmarking, workflow analysis, and interactive use with both Claude Code and as a standalone API engine. XBOW’s methodology mirrored their established practices, utilizing a diverse team of experts and a standardized benchmarking system to evaluate the model’s performance against known vulnerabilities in open-source applications. Notably, the testing expanded beyond traditional benchmarks to include assessments of the model’s judgment regarding threat modeling, vulnerability validation, and safety, as well as its ability to analyze source code versus live systems and identify previously unseen vulnerabilities, such as those within native applications.
The results of the testing indicated that Mythos Preview represents a substantial improvement over existing models, particularly in source code audits and native-code vulnerability discovery. The model demonstrated a notable reduction in false negatives compared to Opus 4.6, achieving a 42% decrease in false negatives and a 55% reduction when provided with source code. However, the report emphasizes that finding a vulnerability doesn't automatically translate to an exploitable one, necessitating further validation and the application of human expertise to confirm and demonstrate the potential for exploitation.
XBOW’s approach highlights the importance of combining AI-powered tools with traditional security methodologies. The firm’s orchestration of the model with live-site validation, utilizing attack tools to generate proof-of-concept exploits, underscores the need for a ‘body’ – skilled human operators – to effectively leverage the model’s capabilities and ensure the validity of its findings. The testing also revealed a tendency for the model to be overly literal and conservative in its assessments, occasionally overstating the practical relevance of its findings.