Key findings:
- The Emerging Threats team made significant updates to the Emerging Threats ruleset to provide more comprehensive information for customers and the community.
- Updates include populating since-created metadata tags within legacy rules, as well as adding MITRE ATT&CK tags in rule metadata.
- Emerging Threats metadata provides additional context to an alert where you initially only have a rule message to make decisions. Without metadata, information security personnel are left with only the rule message from which to act.
- Updating metadata is an investment that strengthens our research and defenses, providing more actionable information and intelligence.
Overview
To fully defend against the rapidly changing threat landscape – from malware to credential phishing to espionage – effective detection is not just about creating rules. It is about optimizing the rules for smarter performance. For organizations leveraging the Emerging Threats ruleset, metadata plays a vital role, delivering invaluable context to security operations analysts, threat researchers, and data scientists that enhances detection beyond mere alerts.
This post takes a closer look at Emerging Threats metadata, addresses the rationale behind specific metadata tags and values, offers practical guidance on how to make the most of this information, and sheds light on the comprehensive approach to implementing recent large-scale metadata updates.
How does Proofpoint utilize Emerging Threats rules?
The Emerging Threats team and the world-class Threat Research organization at Proofpoint work to ensure that our customers and the information security community are protected against threat actors and their techniques within the threat landscape by analyzing malicious network traffic and crafting impactful detection rules for alerting. The Threat Research team including threat hunters, intelligence analysts, reverse engineers, and detection engineers all use the Emerging Threats ruleset in several ways. For example, the rules are baked into the internal Proofpoint sandbox and pipeline to help identify malware families observed in email traffic; analysts collaborate with the ET team to develop new rules based on newly identified activity; and the team uses the ET intelligence portal to surface detections and help identify indicators of compromise while conducting investigations.
Emerging Threats has both ET Open rules that are free for the community, and the paid ET Pro ruleset that contains additional rules based on internal Proofpoint intelligence, threat hunting, and detection. ET Pro is a timely and accurate rule set for detecting and blocking advanced threats using an organization’s existing network security appliances, such as next generation firewalls (NGFW) and network intrusion detection/prevention systems (IDS/IPS). Updated daily and available in Suricata and Snort formats, ET Pro covers more than 40 different categories of malware command and control, credential phishing, DDoS, botnets, network anomalies, exploits, vulnerabilities, SCADA exploit kit activity, and much more.
Updates to the Emerging Threats ruleset
Emerging Threats has produced rules since 2010, with nearly half a million revisions made to the over 100,000 rules In the ruleset, which is updated daily. Since its initial use in 2010, the team has continuously updated and enhanced the ruleset structure and made several metadata advancements, including improving severity and confidence scores, and adding MITRE ATT&CK tags.
Once a new metadata tag is introduced, we are presented with the issue that older rules need to be updated with values for the new metadata tag. Enriching metadata across the ruleset is a large-scale undertaking, and is designed to provide more actionable insight for organizations using the Emerging Threats ruleset. In the latest iteration of updates, the Emerging Threat team focused on enhancing three metadata tags: “signature_severity”, “confidence”, and MITRE ATT&CK coverage to improve the utility and reliability of the rules.
Emerging Threats defines the “signature_severity” values as follows:
- Informational: This is a signature meant to detect activity which may not be malicious in and of itself, but useful to record to add context to other events or alerts. It is often associated with other malicious activity or undesirable behavior.
- Minor: Minor signatures are often associated with reconnaissance, scanning, and other profiling activities. It may not directly indicate something malicious but often precedes malicious activity.
- Major: Major signatures indicate an active attempt at compromise of a service or end system.
- Critical: Critical means that an end system is likely to be compromised based on the activity detected in these signatures. This is the highest severity level. Confidence should primarily be defined by the potential for false positives. A rule with "High" confidence indicates a minimal likelihood of generating false positive alerts. “Low” confidence rules may generate false positives but are valuable not only for observing difficult-to-detect malicious activity (“threat hunting”) but also for adding context into other surround alerts. Additionally, “confidence” is not indicative of malicious intent and does not carry any condemnation weight by itself.
Prior to the latest metadata updates, the “confidence” metadata tag was present in roughly 30% of the ruleset. This tag was created in 2022. Through intensive and consistent updates, the Emerging Threats team increased this to over 70% across more than 100,000 rules. For rules created in 2023 and 2024, “confidence” now has 100% coverage. Similarly, the “signature_severity” tag, which already had relatively strong coverage, has now reached 100% for all rules created between 2010 and 2024. These updates ensure consistent metadata for newer rules, allowing for improved prioritization and response.
The evaluation process
Emerging Threats evaluates “confidence” tag values in the following ways. When evaluating confidence, it is essential to avoid conflating it with factors that contribute to other metadata tags, such as “signature_severity”, and vice versa. For instance, if one determines the “signature_severity” of a rule as "Major" but downgrades it to "Minor" due to doubts about the detection logic, this introduces confidence-based reasoning into a severity-based decision, thereby compromising the integrity of both classifications.
Progress on MITRE ATT&CK mapping coverage has also been substantial. MITRE ATT&CK is a knowledge base for classifying cyberattacks and intrusions that provide resources for security teams to identify and detect malicious activity. By adding MITRE ATT&CK tags to applicable rules in the ruleset, people can reference the technique in the ATT&CK database to see related activity and defensive recommendations. Since commencement of this significant metadata update, half of all rules have had ATT&CK coverage added, which is a huge upgrade and provides users with additional metadata to cross-reference when investigating alerts in their environment. This mapping is invaluable for aligning detections with broader security frameworks, but there are limitations. Emerging Threats will not tag every rule with an ATT&CK technique or tactic. This is an important note, because not all rules align with specific ATT&CK techniques, and Emerging Threats remains committed to maintaining accuracy rather than assigning mappings that do not genuinely apply. If a tag is inaccurately applied, it could result in misunderstandings, or at worst, incorrect defensive actions taken by a user.
Once Emerging Threats achieves 100% coverage for the “confidence” metadata tag and the “signature_severity” tag across the entire ruleset, the focus will shift toward further enhancing ATT&CK mapping coverage. While the initiative is a significant undertaking, it is one that we firmly believe will deliver long-term value for organizations to have as much information as possible in their alerts. Providing comprehensive and accurate metadata is not just about enabling new possibilities—it is also a responsibility to our users, ensuring they have the best possible tools to defend their networks.
Improving metadata
Enriching the ruleset with updated metadata encompasses a variety of processes which may require manual changes at the individual rule level. The Emerging Threats team made the initial rounds of updates by creating regular expression filters in a Python script that interacts with the API of our internal rule submission platform. While these filters are simple, they managed to accurately update the confidence of approximately 40,000 rules across the entire ruleset (note: rules from the “DELETED” category are not considered or updated).
ET(?:PRO)?\s(HUNTING|ATTACK_RESPONSE).*?Obfuscated
Consider the regular expression above. Rules in the “HUNTING” and “ATTACK_RESPONSE” categories are not inherently malicious on their own. However, when combined with “Obfuscated”, their severity is elevated slightly. This distinction enables precise filtering of relevant rules, allowing them to be assigned a “signature_severity” tag value of “Minor”. Without the “Obfuscated” context, the “signature_severity” would instead be set to “Informational” but would again be re-adjusted to account for additional context that the given rule may provide.
Another regular expression filter that targets specific exploit classes and assigns, in this case, a “signature_severity” of “Major” in this instance.
(?:(?:Buffer|Stack)\sOverflow|Co(?:de|mmand)\s(?:Injection|Execution))
There are around 50 regular expression-based filters in total that funnel updates into large segments of the ruleset. Here are some additional examples of how we filtered various rule chunks into specific “signature_severity” values:
High -> DNS over HTTPS
High (for 2018-2024) -> Malicious SNI/TLS signatures (IOC-based)
High (for 2018-2024) -> Outbound Malicious DNS Requests (in the MALWARE category)
Medium (for 2010-2017) -> Malicious SNI/TLS signatures
Medium (for 2010-2017) -> Outbound Malicious DNS Requests
Low -> Signatures detecting memory corruption and other heap-based exploits
The same concept and process was applied to “confidence”, albeit with much stricter filters which drastically limited how many rules were filtered out in the initial first pass. However, this does allow us to address a pain point in the ruleset which has been raised
to us by several customers: the usage of words such as “Possible” and “Suspected”, which the internal Emerging Threats rule writing style guide now safeguards against, preventing their inclusion in rule messages going forward. A filter was created to capture these rules and assign a ceiling “confidence” of “Medium”. Emerging Threats will address the presence of these remaining keywords in rule messages and removing them in favor of the “confidence” tag in a forthcoming metadata update iteration.
Additionally, we estimate around 60,000 rules have been reviewed manually and their “confidence” and “signature_severity” updated.
MITRE ATT&CK and “evidence of intent”
MITRE ATT&CK popularity is rapidly increasing across the threat research landscape and customers requested that Emerging Threats utilize ATT&CK to categorize network rules. In response, the Emerging Threats team implemented ATT&CK tagging support to the rulesets in Q2 2022. While many older rules that predate the addition of ATT&CK support do not have tagging, there are ongoing efforts to update the older rules as soon as possible. While the MITRE ATT&CK framework contains a vast amount of tactics and techniques, the majority are typically seen as applicable to host-only tactics, techniques, and procedures (TTPs). Using an “evidence of intent” approach, these ATT&CK tactics and techniques can be comfortably applied to network rules. This approach allows our rule writers and analysts to analyze network traffic, identify behaviors that would fit a host-only tactic and technique, and apply that ATT&CK tag to the network rule under evidence of intent. To demonstrate the “evidence of intent” approach, consider the batch script below, referenced in the DFIR Report during a network compromise in 2021.
Batch script identified in a 2021 report from the DFIR Report.
Several ATT&CK applicable behaviors are demonstrated in this batch script and while those behaviors are not explicitly active in the network traffic, their presence in a downloaded script is evidence of intent on the host.
netsh advfirewall firewall add rule name="Terminal Server" dir=in action=allow protocol=TCP localport=3389
During the compromise investigated by DFIR Report, threat actors utilized this batch script to achieve several goals, such as adding rules to the host firewall to allow inbound RDP connections. The ATT&CK tactic “Defense Evasion” (TA0005) and technique “Impair Defense” (T1562) matches this behavior. With the evidence of intent approach, we can now apply this ATT&CK tactic and technique to the following Suricata rule.
alert http any any -> [$HOME_NET,$HTTP_SERVERS] any (msg:"ET INFO Suspicious Batch Script - Allow Inbound RDP Rule Set in Windows Firewall"; flow:established,to_client; http.stat_code; content:"200"; http.response_body; content:"|0a|netsh"; nocase; content:"advfirewall|20|firewall|20|add|20|rule"; nocase; content:"|20|dir=in"; nocase; content:"|20|action=allow"; nocase; content:"|20|protocol=TCP"; nocase; content:"|20|localport=3389"; nocase; fast_pattern; pcre:"/^netsh(?:.*?(?:dir|action|protocol)=.*?){3}.*?localport=3389/mi"; classtype:misc-activity; sid:1; rev:1; metadata:mitre_tactic_id TA0005, mitre_tactic_name Defense_Evasion, mitre_technique_id T1562, mitre_technique_name Impair_Defenses;)
While the evidence of intent concept allows the mapping of several host-only MITRE ATT&CK tactics and techniques, there are several that cannot be applied to network rules, such as the “Exfiltration Over Physical Medium” (T1052) technique, part of the “Exfiltration” (TA0010) tactic which requires physical access to a system.
As ATT&CK framework continues to evolve, Emerging Threats will be mirroring new tactic and technique combinations into our rulesets where applicable.
Metadata Applications Emerging Threats metadata provides additional context to an alert where you initially only have a rule message to make decisions. Metadata assists with providing the missing context for that alert and can help steer decisions based on what values have been set. For example: suppose you have a sensor deployed on a network perimeter that monitors a web server cluster. By utilizing our additional metadata, one initial filter option available to begin curating a ruleset is to remove all rules that do not detect malicious web server traffic from various high severity categories such as “WEB_SPECIFIC_APPS”, “EXPLOIT”, and “MALWARE” by isolating our rules of interest with regular expressions. You can query the “attack_target” tag and apply the filter we just discussed.
Here is a regular expression organizations may use to isolate rules of interest for this scenario:
msg:"ET(?:PRO)?\s(?:WEB_SPECIFIC_APPS|EXPLOIT|MALWARE)[^"]+";.*?metadata:.*? attack_target\s(?:client_and_server|server)
Using the query above will extract rules from the “WEB_SPECIFIC_APPS”, “EXPLOIT”, and “MALWARE” categories where the attack_target involves servers, giving us 4,972 rules at the time of writing. If we include “client” attack targets, we have 35,181 rules at the time of writing, meaning we have removed 30,209 rules and isolated 4,972 relevant rules for our curated web server cluster ruleset. Some of the rule types you will extract here include:
- Several vulnerability types (with CVE tagging) such as remote code execution, SQL injections, command injections, server-side request forgery attempts etc.
- Malware families that have been known to or have been observed targeting servers.
- Network requests from compromised libraries involved in supply-chain attacks.
Including categories such as “INFO” and “HUNTING” (and all of their associated rules) in any curated ruleset is highly recommended or else you greatly risk losing coverage on potentially serious incidents. For example, within the HUNTING category we have the following rule that could be categorized as “EXPLOIT” but is more of an experimental signature not tied to any specific vulnerable application or CVE.
ET HUNTING Generic POST with Common Control/Escape Character in Filename Parameter - Possible Command Injection Attempt
By including these additional categories alongside your curated rules, users can add additional detection points for potentially malicious activity targeting your web server cluster.
Emerging Threats have also heard several success stories regarding our metadata being utilized by data scientists to develop a more granular level of rule clustering and categorization. Ignacio Arnaldo, Director of Data Science at Corelight recently presented an Open Information Security Foundation (OISF) webinar on applying data science to Emerging Threats rules and metadata, you can watch that webinar here.
For additional information on Emerging Threats metadata, please refer to this Emerging Threats Discourse post.
Conclusion
The Emerging Threats team continues to strive for complete metadata coverage where possible and will continue to adapt and innovate on new metadata tags when we identify benefits to users of our rulesets. Additionally, Emerging Threats will continue to take steps internally to provide analysts and rule writers additional guidance on the incorporation of metadata.
Emerging Threats will continue to collaborate with the community and partners to provide comprehensive detections, as well as publicly available training such as the recent OISF webinar on how to use the ET ruleset for anomaly detection.
To keep up with the latest Emerging Threat updates from Proofpoint, follow the team on Discourse, X and Mastodon. For requests, feedback and questions, contact the team via Discourse.
