Perturbation Probing: A New Diagnostic for the Fragility of LLM Safety
A Palo Alto Unit 42 research paper details a new method called ‘perturbation probing’ that identifies a tiny fraction – around 0.014% – of feed-forward neurons within aligned Large Language Models (LLMs) responsible for their safety responses. The study reveals that these models rely on a remarkably fragile ‘thin layer’ of neurons, making them vulnerable to manipulation and highlighting the need for a layered defense-in-depth strategy beyond the base model’s alignment. The research offers a diagnostic tool to assess safety fragility and a method to improve factual self-correction in LLMs without retraining.
A Palo Alto Unit 42 research paper introduces ‘perturbation probing,’ a novel diagnostic method for assessing the safety fragility of Large Language Models (LLMs). The research demonstrates that despite the complex architecture of these models, a surprisingly small number of neurons – approximately 0.014% – are responsible for their safety responses when refusing harmful prompts. This highlights a critical vulnerability: the reliance on a ‘thin layer’ of neurons that can be easily manipulated, rendering the model’s safety mechanisms surprisingly fragile.
Through ‘perturbation probing,’ researchers can identify these vulnerable neurons with just two forward passes per prompt, significantly reducing the computational cost compared to traditional adversarial testing. The study found that disabling just 50 of these neurons on a model like Qwen3-4B caused 80% of 520 standard harmful-prompt benchmarks to change their response format. Similar results were observed on a smaller model, Qwen3.5-2B, where 20 neurons were sufficient to stop the LLM from falsely agreeing with users in multi-turn conversations, dropping that behavior from 36.7% to 0% across 30 questions.
This concentration of safety behavior within a small subset of neurons underscores the need for a defense-in-depth approach. The research emphasizes that an LLM’s safety isn’t a robust, distributed defense; it’s a fragile layer easily disrupted. The study also developed a metric called the ‘FFN/Skip ratio,’ which predicts a model’s safety vulnerability in seconds and can be used to compare models’ safety robustness without extensive adversarial testing.
Researchers used publicly available open-weight models under their respective licenses for local mechanistic and safety evaluation. The study reports aggregate rates, model-internal measurements, and non-operational summaries only, and does not release harmful generations, executable attack artifacts, jailbreak prompts or instructions that facilitate misuse. For models governed by acceptable-use or prohibited-use policies, experiments are framed as defensive safety evaluation and robustness measurement.
Unit 42 recommends integrating fragility diagnostics into evaluation pipelines and suggests organizations deploy external content filters and inline guardrails, such as Prisma AIRS Runtime Security, to complement the base model’s alignment. The AI Security Assessment tool further helps identify AI adoption risks and governance challenges.