Cisco Talos, one of the world’s most trusted threat intelligence teams, has published a new report revealing how cybercriminals are increasingly abusing artificial intelligence (AI) tools – particularly large language models (LLMs) – to enhance their operations and evade traditional defenses. The findings highlight how both custom-built and jailbroken (modified) versions of LLMs are being used to generate malicious content at scale, signaling a new chapter in the cyber threat landscape.
The report explores how threat actors are bypassing built-in safeguards legitimate AI tools use, creating harmful alternatives that cater to criminal demands. These unregulated models can produce phishing emails, malware, viruses and even assist in scanning websites for vulnerabilities. Some LLMs are being connected to external tools such as email accounts, credit card checkers, and more to streamline and amplify attack chains.
Commenting on the report’s findings, Fady Younes, Managing Director for Cybersecurity at Cisco Middle East, Africa, Türkiye, Romania and CIS, stated: “While large language models offer enormous potential for innovation, they are also being weaponized by cybercriminals to scale and refine their attacks. This research highlights the critical need for AI governance, user vigilance, and foundational cybersecurity controls. By understanding how these tools are being exploited, organizations can better anticipate threats and reinforce their defenses accordingly. With recent innovations like Cisco AI Defense, we are committed to helping enterprises harness end-to-end protection as they build, use, and innovate with AI.”
Cisco Talos researchers documented the emergence of malicious LLMs on underground forums, including names such as FraudGPT, DarkGPT, and WhiteRabbitNeo. These tools are advertised with features like phishing kit generation and ransomware creation, alongside card verification services. Interestingly, even the criminal ecosystem is not without its pitfalls – many so-called “AI tools” are also scams targeting fellow cybercriminals.
Beyond harmful models, attackers are also jailbreaking legitimate AI platforms using increasingly sophisticated techniques. These jailbreaks aim to bypass safety guardrails and alignment training to produce responses that would normally be blocked.
The report also warns that LLMs themselves are becoming targets, as attackers are inserting backdoors into downloadable AI models to function as per the attacker’s programming when activated. As a result, models using external data sources to find information are exposed to risks if threat actors tamper with the sources.
Cisco Talos’ findings underscore the dual nature of emerging technologies – offering powerful benefits but also introducing new vulnerabilities. As AI becomes more commonplace for enterprises and consumer systems, it is essential that security measures evolve in parallel. This includes scanning for tampered models, validating data sources, monitoring abnormal LLM behavior, and educating users on the risks of prompt manipulation.
Cisco Talos continues to lead the global cybersecurity community by sharing actionable intelligence and insights. The full report, Cybercriminal Abuse of Large Language Models, is available at https://talosintelligence.com/
