Why AI Is So Good at Scamming Humans

Fred Heiding of Menlo Park Intelligence talks with the Dark Reading News Desk about his research on frontier models, and their ability to influence human behavior and create emotional dependency. If there’s one thing that AI models are remarkably good at, it’s manipulation. That’s according to security researcher Fred Heiding, who spoke with Dark Reading’s senior news director, Rob Wright, at the Black Hat USA 2026 last month. Heiding, executive director at Menlo Park Intelligence, and former US National Cyber Director Chris Inglis, spoke at the conference about their research on AI security. One of the trends they discussed was how AI enhances cybercriminals’ social engineering attacks and scams, which cost US citizens nearly $21 billion last year, according to the FBI’s IC3 Report. But Heiding also highlighted how AI technology itself has shown how very effective it is at influencing humans. This poses a challenge for security teams and society in general because, Heiding says, defenders can leverage AI to counter AI-based technical threats, but we can’t use the technology to mitigate manipulation efforts in an effective way. "You can’t just patch me because my brain is old. It’s all these old mental heuristics and systems," he said. "So, we can’t really do it in the same way [with people]. It’s asymmetrical." For all of our Dark Reading News Desk videos, please check out our YouTube channel, and our curated video articles. Dark Reading’s Rob Wright: Hello and welcome to the Dark Reading News Desk at Black Hat USA 2026 in Las Vegas. I’m Rob Wright, senior news director at Dark Reading, and I am here with Fred Heiding. Fred, welcome. Fred Heiding: Thank you so much. DR’s Rob Wright: Fred, tell me a little bit about what you do. Fred Heiding: Yeah, that’s a great question. I’m in a transition phase where I spent a lot of years as a researcher at Harvard Kennedy School and the Harvard Engineering School doing cyber policy and technical cyber work. Now I’m moving to the West Coast to do research with UC Berkeley and also start my own research institute, called Menlo Park Intelligence. DR’s Rob Wright: What are you going to be doing at that institute? What’s your focus? Fred Heiding: So, the main focus will be quite broad. AI security is a lot of what we do. But one of the main reasons is that in academia, you’re quite constrained. There’s a lot of work that you can’t do. For example, I do a lot of work on AI-enabled deception and social engineering — how AI agents manipulate. They’re really good at manipulating people. But you can’t do too much of that in academia because you just have to be confined. It can be quite square as an academic [researcher]. I want to do a little bit more edgy work, a little bit more cooler research, so to speak. Because it’s getting big. I mean, we might do some stuff with election manipulation. That’s big now for the midterms. DR’s Rob Wright: I mean … sensitive topic, but, you know, an important one. Fred Heiding: It’s sensitive across the board, right? But everyone agrees that we want more robust elections. At least, we don’t want AI to manipulate people before they vote. DR’s Rob Wright: To the manipulation part — why do you think the AI has gotten so good at getting people to do what it basically wants them to do rather than the other way around? Wasn’t it programmed to — it was supposed to assist us? It was supposed to do what we wanted, but it feels like sometimes that it’s doing the opposite. Fred Heiding: It’s a really good question. I think the main thing, and we talked about that at Black Hat this year too, right? We call it bottlenecks in this operation. Like, what can and can’t AI do? OK, and in technical attacks, I’m taking a detour, but I’ll come back to you. In technical attacks, attackers use AI, but defenders also use AI. It’s kind of symmetrical in some way. We can both use it. The attacks get better, defense gets better. With social engineering, I mean, I’m a human, right? You can’t just patch me because my brain is old. It’s all these old mental heuristics and systems. So, we can’t really do it in the same way [with people]. It’s asymmetrical. Attackers really benefit from AI because AI is so good at persuading people, but we are the same old humans, and we just fall for the same old attacks. So, it’s very asymmetrical. That’s the big problem with this. DR’s Rob Wright: Obviously, a lot has been going on with AI. You know, we’re seeing new models go off on their own, rogue-like behavior, breaching other companies, finding ways to escape. Is this kind of what you expected when you started looking at AI and seeing what LLMs would do? Is it kind of like what you expected? Fred Heiding: Yes and yes. It does alarm me. I think it’s terrifying. I think we should stop AI if we could. Just going to keep going. It doesn’t surprise me. So we’ve been doing this research for quite a while in my different roles, and we’ve been saying from the start, “This is going to be bad.” One statistic I like is that the FBI has this Internet Crime Center Report, where every year they report how much crime, how much fraud has been reported to US citizens. That was $4 billion in 2020, which is quite high. In 2025, not many years later, that’s now $21 billion. It’s skyrocketing. And these numbers are primarily stemming from US citizens, like everyday people, and they’re just obliterated. And the AI companies profit, and they’re not liable. I think this is terrible. I mean, we have to do something about this. Do you feel that there are some applications, though, for the blue team, the defense? Are we not fulfilling the potential to use this for positive effects for cybersecurity, to stop the scams, to stop the vulnerability exploitations, to stop the zero-days, etc.? Do you feel that there’s a potential there for that? Fred Heiding: Yeah, there’s a lot of potential. There are several areas where blue-team defenders can win. Technical cyberattacks, I think, are a great example, like vulnerability discovery. Of course, AI companies already do this. They give the AI models that are powerful to defenders before they release them publicly. That’s great. Everyone wins, right? And that’s great. But how do you do this in phishing? Because if you give this — so, let’s say Anthropic gives me their latest model, and I’m a phishing defender. Well, I can use it to improve the spam filters, but that’s not really helping that much. The problem is the scale of this because it’s so easy to create a lot of phishing emails, and I can’t just take Mythos or a good model and push it onto a human being. It doesn’t matter, right? Because we have our mental heuristics. So, blue teaming for technical attacks — great. Blue teaming for phishing defenders — we’re really lagging behind. It’s not — on the contrary, it’s worse, right? Because now it’s so easy to create these scams. Now you can also create trust and reliability. So, we did a lot of work on just voice models. You can create perfect voice scams, even more with these long-term, multiturn stages that goes for a year-long scam, for example. Because it’s really cheap to have an AI bot just talking to you. And now we have the romance scams, right? So, people start falling in love with their AI bots, and there’s a lot of legitimate companies that do these AI bots too. But it’s kind of icky. And when scammers start exploiting this, it’s just terrible. There’s one example, if I may say. Last week or two weeks ago, The Economist had an article about China. So, China starts banning all the AI models. You read that one? DR’s Rob Wright: Yes. And I think we should do it in the US. So, the context is that they ban emotional dependency on AI. That’s great. But when they did that, people were already in love with these AI bots. One woman that was being interviewed, she said, “I’ll pay half my salary to keep my AI bot alive.” That’s a lot of money. DR’s Rob Wright: I know. I’m in the wrong business. I should pivot. No, I think you’re right — yeah, I read the same article, and I’m alarmed by it. Do you feel like the AI companies need to be doing more to restrict, restrain, or just sort of practice more sort of ethical applications of this technology? Fred Heiding: That’s why I think this problem of especially social engineering is so interesting because the deceptive capabilities are 100% aligned with just models’ thinking capability and capability to talk in general. So, it’s almost impossible to train it away. I think we’ll never do this. The AI companies can’t really remove this functionality. It’s like, what do you do, right? Should you propose a deception tax that they have to pay a little because this is hurting [people]? I don’t really know. They have teams working against social engineering, the AI labs, but it’s too little. Not too much is happening. So the capabilities can’t be removed. I think some companies like Anthropic, where it’s pretty hard with Know Your Customer, you have to log in. You can’t just have a private browser. I think that’s good. But then we also have all the open-weight models that are not far behind, and they can also be used. So I really [think] it’s a tough problem. One thing we do a lot now is just measuring it. The first step is quantifying it, putting dollar amounts to it, seeing how the different attacks work in different stages. We call this the scam kill chain, right? First they find the target, then they talk with them, build rapport, exfiltrate money. And which of these parts — is there any part we can block? Some parts we can’t block, but maybe there’s one area we can stop. ”