Remediating Vulnerabilities With LLMs: Inside Ivanti's Automation Push
Ivanti is leveraging large language models (LLMs) to automate vulnerability remediation and discovery, a project initially sparked by the surprising effectiveness of the Claude 4.6 model. Daniel Spicer, Ivanti’s CSO, describes how the company is using multiple LLMs, including Anthropic’s models and open-source options, to identify vulnerabilities that traditional tooling misses and to automatically fix them. The project has yielded promising results, with a significant reduction in AI-generated “slop” and a decrease in the volume of inaccurate bug reports. However, the increasing costs of AI usage and the potential for misuse are being addressed through internal guidelines and careful consideration of project priorities, with a focus on identifying areas where AI can effectively augment, rather than replace, human expertise.
Ivanti is leveraging large language models (LLMs) to automate vulnerability remediation and discovery, a project initially sparked by the surprising effectiveness of the Claude 4.6 model. Daniel Spicer, Ivanti’s CSO, describes how the company is using multiple LLMs, including Anthropic’s models and open-source options, to identify vulnerabilities that traditional tooling misses and to automatically fix them. The project has yielded promising results, with a significant reduction in AI-generated “slop” and a decrease in the volume of inaccurate bug reports. However, the increasing costs of AI usage and the potential for misuse are being addressed through internal guidelines and careful consideration of project priorities, with a focus on identifying areas where AI can effectively augment, rather than replace, human expertise.
Ivanti first began exploring LLMs in mid- to late February, after realizing the potential of models like Claude 4.6. The company’s internal red team, led by Mike Reamer, began experimenting with the models, initially focusing on identifying vulnerabilities and then on automatically resolving them. The project is structured around two tracks: one to find vulnerabilities that existing SAST and DAST scanners miss, and another to automatically fix vulnerabilities identified by those tools.
Initially, the project faced challenges with “slop” – AI-generated reports that were largely inaccurate or didn’t represent genuine vulnerabilities. Last year, the company received hundreds of such reports, requiring significant manual review. However, this year, the quality of the AI-generated reports has improved dramatically, with a significant drop in the number of inaccurate reports. Spicer attributes this to a shift in the models’ capabilities and a more targeted approach, starting with low-complexity CWEs (Common Weakness Enumeration) and gradually increasing the complexity.
The project utilizes a combination of frontier models, including Anthropic’s Claude, and several open-source models, each deployed in specific ways to maximize their effectiveness. Rather than simply feeding the models a vulnerability report, the team provides them with a specific context and asks them to recommend fixes. For example, they instruct the models to check all endpoints for missing authorization or authentication and to suggest improvements.
Despite the success, Ivanti is mindful of the escalating costs associated with AI usage. Spicer emphasizes the need to carefully evaluate whether a task is a good use of AI resources, noting that tasks like deploying software and configuring virtual infrastructure are often not suitable for AI.
Furthermore, the company is actively addressing concerns about the potential for AI to be used for malicious purposes. Spicer recounts a bug bounty report that highlighted the effectiveness of Claude 4.6 in generating convincing vulnerability reports, even when the underlying issue was already known. This experience reinforced the importance of guardrails and careful monitoring to prevent AI from being used to create misleading reports.
Ivanti has identified several areas where LLMs are particularly effective at finding vulnerabilities that traditional tools miss, including authorization and authentication issues on endpoints. The company is focusing on using AI to augment, rather than replace, human expertise, prioritizing tasks where AI can provide valuable insights and automate repetitive processes.