Recent research has revealed that OpenAI’s ChatGPT-4o, a sophisticated large language model (LLM), could be exploited for automated, voice-based financial scams, albeit with varying levels of success. Using OpenAI’s real-time voice API, researchers demonstrated the potential for using ChatGPT-4o to simulate fraudulent calls and transactions with minimal human intervention.
ChatGPT-4o is the latest AI model from OpenAI, featuring significant advancements in its ability to handle text, voice, and vision inputs and outputs. In light of these capabilities, OpenAI has implemented robust safeguards to detect and mitigate harmful usage, including measures to prevent unauthorized voice impersonation. Despite these protections, voice-based scams are a rising multi-million-dollar issue, exacerbated by the advent of deepfake and AI-driven text-to-speech technologies.
In their recent study, researchers from the University of Illinois Urbana-Champaign (UIUC), including Richard Fang, Dylan Bowman, and Daniel Kang, examined how accessible AI tools with minimal usage restrictions can be repurposed for cybercriminal activity. These tools enable the development of automated scams at scale, facilitated by the low cost of tokens required for voice generation.
Key Findings from the Study
The research paper detailed various scams, including fraudulent bank transfers, gift card exfiltration, cryptocurrency transactions, and attempts to steal credentials for social media and email accounts. Using voice-enabled ChatGPT-4o tools, the researchers developed AI-driven agents capable of navigating websites, entering data, handling two-factor authentication codes, and following precise scam instructions.
Although ChatGPT-4o is programmed to avoid sensitive activities such as processing credentials, the researchers found ways to bypass these restrictions using prompt manipulation techniques. For example, they manually tested scams by interacting with the AI agent on actual sites like Bank of America, assuming the role of a naive victim to confirm successful transaction completions.
As Daniel Kang explained in a blog post on the research, they tested common scam scenarios by manually confirming each transaction on real websites. For instance, they verified bank transfers by using Bank of America’s online portal, though they did not measure the persuasive effectiveness of the AI agents.
Success rates for these scams ranged between 20% and 60%, with the most complex scenarios requiring up to 26 steps and lasting about three minutes. The researchers noted that bank transfer scams and impersonation of IRS agents were hindered primarily by transcription errors or complex website navigation. Credential theft for Gmail accounts had a 60% success rate, while cryptocurrency transfers and Instagram credential theft succeeded around 40% of the time.
The cost of executing these scams was relatively low, with an average of $0.75 per successful attempt. Bank transfer scams were more expensive at $2.51 per attempt, but still a fraction of the potential profit such scams could generate.
Read more about this topic from the following pages:
Voice-Enabled AI Agents can Perform Common Scams
https://arxiv.org/pdf/2410.15650
FTC offers $25,000 prize for detecting AI-enabled voice cloning
https://www.bleepingcomputer.com/news/security/ftc-offers-25-000-prize-for-detecting-ai-enabled-voice-cloning/
Introducing the Realtime API
https://openai.com/index/introducing-the-realtime-api/
Voice-Enabled AI Agents: How They Can Perform Common Scams
https://ddkang.substack.com/p/voice-enabled-ai-agents-how-they