Using LLMs to Find and Prioritize Vulnerabilities Is No Easy Task
Large language models (LLMs) are not effectively addressing vulnerability prioritization for application security teams, with a significant number of flagged vulnerabilities proving to be false positives or irrelevant due to a lack of context. Current AI tools require substantial human oversight and don't consistently improve vulnerability detection rates, leading to increased workload for AppSec professionals. The key to reducing vulnerability patching is focusing on reachability – identifying unused code – and establishing robust support functions to ensure consistent results.
Large language models (LLMs) are failing to significantly improve vulnerability prioritization in application security, presenting challenges for security teams. According to Arshan Dabirsiaghi, chief technology officer and co-founder at Pixee, more than 60% of vulnerabilities identified by LLMs are either false positives or in unreachable code, and are of low severity. This is due to the models’ lack of context and reliance on general knowledge, rather than specific understanding of the application’s architecture and deployment environment.
Currently, security teams are facing an overwhelming influx of vulnerabilities, with the Forum of Incident Response and Security Teams (FIRST) estimating a 50% increase in CVE-assigned issues this year, and Microsoft setting records for Patch Tuesday volumes. Despite the potential of AI, current tools require extensive human review and oversight to validate findings.
To be truly effective, AI models need substantial context – organizational, technical, and code context – to accurately identify vulnerabilities. Dabirsiaghi emphasizes the importance of ‘reachability’ – determining which code is actually used at runtime – as a method to reduce the number of vulnerabilities requiring patching. He notes that a significant portion of open-source libraries and Java code are rarely used, reducing the potential impact of vulnerabilities within those components.
Furthermore, the inconsistency of LLM results – where a vulnerability is initially flagged as a false positive but later identified as a true positive – highlights the need for robust support functions and a ‘harness’ around the AI models to ensure consistent and reliable output. Without these supporting elements, AI can lead to over-fixing and a decline in trust in the tool’s findings.
Ultimately, the current AI approach requires significant human intervention and doesn’t offer a substantial improvement over traditional methods for vulnerability prioritization and remediation.