How Effective Are Users at Reporting Real Phishing Messages?

If you’re measuring the reporting rate of phishing simulations, that’s great. Even better if a substantial portion of the users in your organization are using a phishing reporting add-in when simulated phishing message comes their way. 

But how does this behavior translate to real-world phishing reporting—when that truly malicious or suspicious message enters a user’s inbox, and the user does a double-take on its legitimacy? 

To determine how effective users are at reporting real phishing message and compile the metrics for this blog—which is the third in a series—we analyzed data from the millions of people using the Proofpoint PhishAlarm phishing button to report messages. 

Figure 1. Proofpoint Nexus Threat Graph showing the breadth of our threat intelligence which powers our detection stack

We also wanted to understand what kinds of messages users report. So, we used the Proofpoint Nexus Threat Graph and Proofpoint Targeted Attack Protection (TAP) detection stack to assign a disposition automatically. This capability is something Proofpoint Security Awareness Training customers get included.

What are the types of reported messages?

Let’s start with the classification types of reported messages, to put everything in context:

• Malicious: These emails contain malware, phish, impostor or other threats.
• Suspicious: These emails are likely to be malicious, so quarantining them is recommended; however, they should be reviewed to ensure no legitimate email is lost.
• Spam: These emails are a nuisance and could contain malicious content.
• Bulk: These emails are the low-priority or promotional type; they don’t pose a threat
• Low risk: Analysis of these emails finds no indication of malicious content. 
• Unlikely a threat: These emails don’t present any malicious content upon dynamic sandboxing and analysis by the detection stack; therefore, they aren’t likely a threat.

Reporting accuracy of users: better than you might think

We used two distinct methodologies to examine the types of messages users report. That’s because some organizations want their users to report only malicious messages; they don’t want them to use PhishAlarm to report spam, or they prefer they use another tool to do so. And other customers want their users to report malicious messages and spam to their abuse mailbox, as either of those email types could potentially do harm. 

It’s worth noting that customers using PhishAlarm may or may not be using Proofpoint Email Security and Protection, which can impact the likelihood of malicious/suspicious messages reaching users. 

In applying our first methodology (including malicious or suspicious messages only), we were surprised to learn how good users are at spotting malicious messages:

First methodology: malicious or suspicious-only messages included

Figure 2. Y axis: Number of customers, X axis: Percentage of users at an organization reporting malicious or suspicious messages. Most organizations tend to fall into ~30% of their users reporting malicious or suspicious messages accurately, but top performers can reach a number well above 50% accuracy.

Average reporting rate of malicious or suspicious messages shown in above chart by percentiles:

After analyzing millions of reported messages at thousands of organizations worldwide, we were delighted to see the progression of user reporting accuracy. Our findings underscore that users can be a powerful line of defense in the fight against phishing attacks.

For example, in the top-performing quartile, almost two-fifths of reported emails were malicious or suspicious, on average. Many organizations were able to achieve accuracies above 60%. And if you include spam messages, the average reporting rate was even higher.

Second methodology: malicious, suspicious or spam messages included

Figure 3. PhishAlarm reporting chart showing report rate of malicious, suspicious or spam messages

Average reporting rate of malicious, suspicious, or spam messages:

In applying our second methodology, we were pleasantly surprised—once again—to see that users are accurately submitting malicious, suspicious or spam messages with the PhishAlarm add-in. In this examination, we found that users are even better at reporting bad messages to internal teams, averaging about one-third of messages.

Also, the new Email Warning Tags with Report Suspicious feature from Proofpoint mean we think user reporting could improve. With the presence of real-time, embedded HTML notifications on all devices and an easier reporting mechanism with fewer clicks or taps, this should enable users to improve reporting frequency and accuracy. These nudges can be customized for different types of potentially malicious messages to get users’ attention and allow them to report phish more easily.

Figure 4. Email Warning Tags with Report Suspicious, now available to Proofpoint Email Protection customers, can help boost user reporting rates and accuracy

Takeaways: Empower your people—and augment with automation

Right before the COVID-19 pandemic, I was at an event in Chicago talking to a customer who told me his team was testing our PhishAlarm reporting button. He also said they were hesitant to commit fully to using the add-in until they had Proofpoint Closed-Loop Email Analysis and Response (CLEAR)—our abuse mailbox automation solution—in place. The customer explained that his team was worried they’d be overburdened by user-reported false-positive messages. 

On average, about two-thirds of the email messages that users submit for investigation aren’t dangerous, so there’s definitely room for improvement in user reporting accuracy. So how do you handle the newly found behavior of users reporting more messages to your abuse mailbox that might not all be dangerous? Take users’ behavioral instincts and combine them with powerful automation for ideal results. 

Learn more about that process in the next installment of our blog series, which is coming soon! Subscribe below to get a digest of new blog posts from Proofpoint like this one.