Attackers Manipulate AI Chatbots in Mass Disinformation, Phishing Campaign
Threat actors are poisoning AI chatbots like ChatGPT and Gemini by seeding the web with malicious links and data, leading to a widespread disinformation and phishing campaign targeting numerous companies, including major airlines, banks, and travel providers. The attack leverages SEO techniques to manipulate AI responses, presenting fraudulent information as trusted facts, bypassing traditional AI defenses. Researchers estimate over 374 companies have been affected, and the campaign is ongoing, posing a significant risk to users and organizations relying on AI chatbots.
Researchers from Vigilance Security have identified a campaign dubbed "Dark Sourcery" that is systematically poisoning popular AI chatbots, including ChatGPT and Google Gemini, by flooding the web with carefully optimized pages containing malicious links and data. The goal is to manipulate AI responses, presenting fraudulent phone numbers, email addresses, login pages, and software-update information as trusted facts.
To achieve this, attackers utilize SEO and content-distribution techniques to increase the likelihood that AI chatbots will retrieve and incorporate these malicious pages into their responses. Vigilance researchers have identified at least 374 companies impacted, including major airlines (Delta, Lufthansa, Qatar Airways), banks (Chase, Bank of America), travel companies (Airbnb, TripAdvisor), and software providers.
This attack is similar to SEO poisoning, but it differs significantly in that the malicious information becomes part of the AI’s answer itself, bypassing traditional AI defenses that typically rely on prompt injection attacks. Unlike prompt injection, where attackers provide explicit instructions to the AI, this campaign relies on manipulating the AI’s data sources.
A study by Exploding Topics found that 91% of people using AI chatbots do not verify the answers provided, creating a dangerous reliance on AI-generated information without critical evaluation. The campaign has real-world ramifications, with researchers reporting instances of users being tricked into providing payment details or credit card information via fraudulent phone numbers presented as helpful guidance by the AI.
For users, Vigilance recommends verifying all AI-generated results and not blindly trusting chatbots as infallible sources. For affected companies, security teams should take customer complaints about scams seriously, monitor AI answers, and compare returned details against verified records. Organizations utilizing AI chatbots should also monitor agents at runtime, verifying every source and piece of content they rely on, and analyzing AI agent’s delivered content and behavior.