AI Buzzwords vs. AI Breakthroughs
Analysts read reports every day that hint at adversary behavior but stop short of mapping to MITRE ATT&CK. Valuable context is lost, and blind spots grow. At the same time, vendors promise “AI-powered” features, but too often, what’s under the hood is little more than regex rules, keyword matches, or indexing tricks presented as innovation.
Instead of making the job easier, these claims create more noise, false positives, and wasted time. A true breakthrough in AI for cyber threat intelligence doesn’t just check the “AI box.” It gives analysts reliable, high-fidelity results they can trust in real workflows, surfaces context where it’s missing, closes blind spots, and reclaims time for the work that matters.
As cyber threats evolve in real time, AI must do more than assist analysis. It must help teams identify and act on relevant signals before impact.
Why Generic AI Falls Short
Across the industry, advanced CTI teams are experimenting with small, tightly controlled LLM workflows that summarize reports, search breach data, or build agents for teams to create repeatable outputs. These smaller-scale experiments can be effective when paired with a human in the loop who fine-tunes the tools, validates the outputs, and takes responsibility for the results. But most teams can’t be experts in both AI and cyber threat intelligence, and that’s where generic, one-size-fits-all features fall short.
The problem is that generic LLMs aren’t built for CTI. They guess too much, inherit bias from broad training data, and often leave teams blind to new techniques because there’s no clarity on which version of MITRE ATT&CK they reflect. Independent research backs this up: the CyberSOCEval benchmark of LLMs on Threat Intelligence reasoning found leading LLMs scored only 43–53% accuracy on CTI reasoning tasks and just 15–28% on malware analysis, concluding that “current LLMs are far from saturating our evaluations,” and that there is a “significant hill to climb for AI developers to improve AI cyber defense capabilities.”
Current LLMs aren’t “analysts in a box.” That’s why CTI teams should ask harder questions:
- What problem is this tailored to solve?
- Which models are being used?
- How is accuracy measured and monitored?
- Where does the training data come from?
- How often are models updated?
- What standards guide their design?
In practice, AI must deliver intelligence that is transparent, continuously improving, and grounded in real-world threat activity.
The Dataminr Difference: A Purpose-Built Approach to AI for CTI
MITRE ATT&CK has been central to CTI for years, but most tools only scratch the surface. They tag explicit mentions or keywords when they’re written into a report and leave the harder and more valuable context undiscovered, creating blind spots for analysts.
Analysts need a way to surface ATT&CK techniques even when they’re only implied. Dataminr for Cyber Defense applies AI to go beyond keywords and reveal context that would otherwise slip through, surfacing techniques that authors never name outright. When a report implies Credential Access: Brute Force without saying it, analysts still get the signal they need to evaluate exposure and response.
At the same time, explicit references are never missed. By combining precise extraction with AI-driven classification, analysts gain a more complete view of adversary behavior and the context needed to act.
AI That Works Where Analysts Work
Adversaries never stand still, and static AI models fall behind quickly. Generic models lag because they’re trained on broad datasets, biased toward noisy outputs, and often tied to outdated frameworks. Keeping pace requires an approach that delivers both coverage and precision in real time.
Three advances help power this approach:
- Context-aware classification. Analyzing the meaning of a report—not just its keywords—surfaces techniques that might never be named outright. For analysts, this reduces blind spots and improves the completeness of intelligence.
- High-fidelity training data. By incorporating curated and synthetic training data, models can recognize emerging adversary behaviors earlier and with greater accuracy.
- Quality-controlled outputs. Potential matches go through multiple layers of validation, with only the strongest signals surfaced. This reduces false positives and ensures intelligence teams can act with confidence.
These models are continuously monitored and refined to align with evolving threat activity and responsible AI practices—ensuring intelligence keeps pace with adversaries.
MITRE ATT&CK AI Classification in OSINT Reports
The scale of the difference is clear. In a 90-day period, automated analysis of OSINT reporting shows:
- 9 out of 10 reports classified: AI-driven classification applied to 90% of reports compared to 37% using explicit ATT&CK mentions alone
- 7.7× more context surfaced: Nearly 29,000 TTPs identified compared to fewer than 4,000 from explicit mentions
This isn’t just a statistical improvement. It reduces manual effort and enables detections and control gap analysis based on a more complete understanding of adversary behavior.
Why It Matters for Analysts
For CTI teams, the benefits are immediate and practical. Analysts spend less time hand-tagging reports and more time acting on the intelligence inside them. Implicit behaviors that would have slipped through are now surfaced with confidence, improving visibility into adversary tactics.
This intelligence can be applied across workflows—from enrichment to triage to reporting—supporting faster, more informed decisions.
Most importantly, the system is built for trust. Transparent design, continuous monitoring, and human oversight ensure that outputs remain reliable as adversary behavior evolves.
What was once slow, manual, and partial becomes faster, clearer, and more complete.
Use AI That Delivers in Real Workflows
Every day, analysts are asked to do more with less time and greater pressure. What makes AI valuable isn’t a polished demo—it’s whether it holds up in real workflows.
Purpose-built AI enables analysts to surface missing context, close gaps in coverage, and act on intelligence with confidence. When adversaries are evolving in real time, the measure of AI isn’t how it looks—it’s how effectively it helps teams move from insight to action.
Ready to see Dataminr for Cyber Defense in action? Contact us for a demo.
