Perturbation Probing: A New Diagnostic for the Fragility of LLM Safety

A Palo Alto Unit 42 research paper details a new method called ‘perturbation probing’ that identifies a tiny fraction – around 0.014% – of feed-forward neurons within aligned Large Language Models (LLMs) responsible for their safety responses. The study reveals that these models rely on a remarkably fragile ‘thin layer’ of neurons, making them vulnerable to manipulation and highlighting the need for a layered defense-in-depth strategy beyond the base model’s alignment. The research offers a diagnostic tool to assess safety fragility and a method to improve factual self-correction in LLMs without retraining.