Mimecast Threat Intelligence: How ChatGPT Upended Email

UPDATE: This blog was originally published on September 30, 2024 as a follow up to Mimecast's Global Threat Intelligence Report 2024 H1. We have since updated the data reflected herein as we have continued to closely monitor the use of generative AI tools like ChatGPT by cybercriminals in their efforts to craft more believable phishing emails and other malicious content. Be assured that we will continue to monitor this attack method and report updated findings again when relevant.

Across most cybersecurity media, references to generative AI are not only increasing exponentially, but publications are citing its use by malicious actors which has the strong potential to negatively impact any organization targeted.

When interviewing Mimecast threat researchers for our Global Threat Intelligence Report 2024 H1, questions were asked about the pervasiveness of AI in phishing emails – but no metrics could be quantified. This left unanswered questions, including how prevalent is this, and can it be measured? Our data science team took on the challenge to help, by building a detection engine to determine if a message is human- or AI-generated based on a mixture of current and historical emails, and synthetic AI-generated emails. 

The research indicated a point in time when we started observing an increasing trend in AI-generated emails correlating with the release of ChatGPT. We also observed malicious AI-generated BEC, fraud, and phishing emails. The net effect of this was a need for understanding by analysts/security teams and end users of the indicators of AI-generated content which could help them spot these attacks. 

Telltale Signs of AI-Generated Emails 

ChatGPT made AI-assisted email writing accessible to everyone, even malicious actors but this is not the only set of tools available to them. In a previous blog post we outlined some of the generative AI tools they use. Previously, such tools were mainly for businesses. Now, anyone can use AI to write well-crafted emails suited to various situations. As AI-generated content becomes more prevalent, the ability to discern between human-written and machine-generated text has become increasingly difficult. One of the most notable characteristics of AI language models is the use of complex words and sentence structures, which can reveal their involvement in writing. Researchers at Cornell University found AI language models favor certain words in scientific writing. ”Analyzing 14 million papers from 2010-2024, they noticed a sharp increase in specific ’style words’ after late 2022, when AI tools became widely available. For example, ’delves’ appeared 25 times more often in 2024 than before. Other AI-favored words include ’showcasing,’ ’underscores,’ and ’crucial.’”