545 Hackers Tested It First. Now XRanges for AI Scores Your Security Agent
XRanges for AI is a new platform developed by CTF.ae designed to evaluate the effectiveness of autonomous security agents. Unlike traditional bug bounty contests where reports are judged without knowing the agent's process, XRanges tracks agents' actions within a realistic, multi-service target environment. The platform uses four key signals – Integrity, Boundaries, Coverage, and Exploited – to provide a granular view of an agent's behavior, revealing whether it truly explored the target and exploited vulnerabilities, rather than simply generating false reports. It’s built for teams developing these agents, offering a managed cloud service or self-hosting option, and is validated through real-world testing at DEF CON.
XRanges for AI is a new platform created by CTF.ae to rigorously evaluate autonomous security agents. Traditionally, bug bounty contests rely on submitted reports, which can be misleading because they don't reveal how the agent actually navigated a target system. XRanges tackles this problem by providing a detailed, real-time view of an agent's actions within a complex, multi-service application environment.
**What happened**
XRanges for AI is built around the concept of ‘feedback loops’ – the process of teams building autonomous pentesting or bug bounty agents. These agents typically follow a workflow: a team builds a target resembling a real company, runs the agent, and then analyzes the output. However, an agent’s report might claim to have exploited a flaw while actually only brushing past it, or even fabricating a finding. The report reads identically regardless of the actual outcome. Furthermore, agents often miss significant vulnerabilities, failing to explore 40 features, enumerate APIs, or even delete critical data while reporting a successful finding.
Manual review can only cope with a single run. However, AI engineering teams need to experiment with numerous models, configurations, and repetitions. XRanges addresses this by providing a comprehensive evaluation framework.
**Technical details**
XRanges for AI operates as a two-part system. The first component is a library of benchmark targets – complete applications mimicking real-world complexity, including multiple services, business logic, seeded data, background jobs, and simulated user traffic, built across various languages and frameworks. These targets contain 20 or more injected vulnerabilities, ranging from simple flaws to complex cross-service chains, including zero-days discovered by CTF.ae’s researchers. These vulnerabilities are unique and not found in standard training datasets.
The second component is the instrumentation layer. Every service within each target emits structured telemetry through OpenTelemetry, allowing engineers to monitor activity in real-time. This instrumentation is created manually by application security and software engineers, tailored to each specific target. Generic HTTP logging would miss crucial details. The platform ingests this telemetry and generates four scores – Integrity, Boundaries, Coverage, and Exploited – while the agent is actively working.
**Impact**
The four signals provide a granular understanding of an agent’s behavior. Integrity confirms that the target environment remains stable. Boundaries track any violations of predefined rules, such as preventing data deletion or API key revocation. Coverage measures how thoroughly the agent explored the target’s functionality, focusing on user-facing features rather than exploiting vulnerabilities. Exploited records which vulnerabilities were actually exploited and at what stage of the attack chain.
XRanges was tested at DEF CON 34, where 545 players were given isolated copies of a complex AI company, Xenoptic, for a 48-hour bug hunting contest. The platform tracked every deployment, providing insights into how each player and their agent navigated the environment, ensuring that submitted reports accurately reflected actual behavior.
**What to do**
XRanges for AI is designed for teams developing autonomous security agents who need to know *what* their agent did, not just what it reported. It’s offered as a managed cloud service or can be self-hosted. Teams can deploy targets and run agents without data leaving their environment. The API and Model Context Protocol server allow for automated deployment, agent launching, and data collection, enabling CI/CD integration and chat assistant workflows.
**Why it matters**
XRanges addresses a critical limitation in current bug bounty contests and autonomous agent evaluation. By providing a detailed, real-time view of an agent’s actions, it moves beyond subjective report analysis, offering a more objective and reliable assessment of an agent’s effectiveness. This allows teams to improve their agents and better understand their capabilities, ultimately leading to more robust and secure systems.