CardinalOps’ third annual report analyzes real-world data from production SIEMs covering nearly 4,000 detection rules across diverse industry verticals
If you’re looking to benchmark your SIEM with respect to its MITRE ATT&CK coverage and rule health, you may be interested in reading our 3rd Annual Report on the State of SIEM Detection Risk.
The report analyzes real-world data collected from production SIEMs – including Splunk, Microsoft Sentinel, IBM QRadar, and Sumo Logic – covering more than 4,000 detection rules, nearly one million log sources, and hundreds of unique log source types.
The data spans diverse industry verticals including banking and financial services, insurance, manufacturing, energy, media & telecommunications, professional & legal services, and MSSP/MDRs.
Why the SIEM Matters
According to industry analysts, the SIEM continues to be the “operating system of the SOC” and is not going away anytime soon.
In fact, according to the SANS 2023 SOC Survey, SIEMs and EDR are the top two technologies considered critical to having an effective SOC.
Assessing and Strengthening SIEM Effectiveness
However, most organizations face the challenge of how to continuously assess and strengthen the effectiveness of their existing SIEMs, using standard frameworks like MITRE ATT&CK to measure their readiness to detect the highest-priority threats.
This is a major challenge because organizations have to grapple with constant change in adversary techniques plus constantly expanding attack surfaces, combined with the difficulty of hiring and retaining skilled detection engineers.
These challenges are clearly illustrated in data from this year’s SIEM Detection Risk report. Using MITRE ATT&CK as the baseline, CardinalOps found that, on average:
- Actual detection coverage remains far below what most organizations expect: Enterprise SIEMs only have detections for 24% of all MITRE ATT&CK techniques. That means they’re missing detections for around three-quarters of all techniques that adversaries use to deploy ransomware, steal sensitive data, and execute other cyberattacks.
- SIEMs don’t need more data: SIEMs are already ingesting sufficient data to potentially cover 94% of all MITRE ATT&CK techniques. But many enterprises are still relying on manual and error-prone processes for developing new detections, making it difficult to reduce their backlogs and act quickly to plug detection gaps. A more effective strategy would be to scale SIEM detection engineering processes to develop more detections faster, via automation.
- Broken rules are also common: 12% of SIEM rules are broken and will never fire due to data quality issues such as misconfigured data sources and missing fields – resulting in increased risk of breach due to undetected attacks.
- Organizations are implementing “detection-in-depth”– but monitoring of containers lags behind: Enterprise SIEMs are following best practices and collecting data from multiple security layers such as Windows endpoints (96%), network (96%), IAM (96%), Linux/Mac (87%), cloud (83%), and email (78%).
But monitoring of containers lags far behind other layers at only 32%, despite Red Hat data showing that 68% of organizations are running containers. This low number could be because it’s challenging for detection engineers to write high-fidelity detections to uncover anomalous behavior in these highly-dynamic environments.
Conclusions
Based on our conversations with scores of enterprise organizations, we’ve found that most organizations don’t have good visibility into their MITRE ATT&CK coverage and are struggling to get the most effectiveness from their existing SIEMs.
Moreover, they typically can’t predict how their coverage will improve over time, and they don’t have a systematic way to prioritize the development of new detections based on what MITRE calls a “threat-informed defense.”
Why is this important? Because common sense dictates that preventing breaches starts with having the right detections in your SIEM – according to the adversary techniques most relevant to your organization – and ensuring they’re actually working as intended.
Our customers also tell us that leveraging automation and detection posture management are critical capabilities for making this happen in a predictable and programmatic manner (rather than an ad hoc approach).
To help organizations address their detection challenges, the 2023 CardinalOps report also includes a series of best practices – based on recommendations from Dr. Anton Chuvakin – to help SOC teams measure and continuously improve the robustness of their detection posture over time (with or without automation).
You can view an infographic with a summary of results below, and download the full report here.
