Eliminate Risk Blindspots with Multi-modal Fusion AI

In the event of a plane crash, the first report might be a tweet noting a loud bang, followed by a different user sharing a video of smoke in the distance, and a third post a selfie-video from an eyewitness with the location of the fire in the caption.

In the cyber world, a threat actor might chat on a Telegram channel about targeting a new organization while, separately on X, they begin sharing links to recently disclosed CVEs and exploit code. Then within minutes, employee credentials are dumped on a paste site while fake SSO login pages are spun up impersonating the targeted organization. 

In both examples, every disconnected detail is a critical piece of a larger story. Both physical and cyber security teams often rely on AI-powered tools to gain visibility into potential concerns and parse key information. Unfortunately, many AI-powered risk intelligence or threat intelligence tools analyze only one type of data input. This approach misses the critical data scattered across various sources and modalities, and therefore presents an incomplete or inaccurate portrayal of the event, threat, or risk. 

For today’s organizations, every second counts when it comes to incident and threat response. The faster an organization can get the most comprehensive version of the story, the more effective their response is, and the more resilient their organization becomes.

What is Multi-Modal Fusion AI?

As illustrated, a single-modality approach to event detection can be flawed. Relying on image-only, video-only, sound-only, or text-only inputs can be effective for certain circumstances, but comes with limitations. 

In the plane crash example, single-modality AI might capture the first tweet, missing the critical and timely context that the next two posts provided. In the cyber scenario, a security operations center (SOC) might not even be aware of the early signals until after the threat actor had duped employees into giving away their SSO credentials and gained access to critical systems. 

This is why multi-modal fusion AI—which enables cross-modality reasoning to correlate disparate, disconnected signals into single, comprehensive events—is so critical for rapid detection and contextualization of incidents for security teams, emergency services, and more to take rapid, decisive action.

Multimodal AI refers to models that can understand and process multiple types of data—for example, text, images, video, audio, and network telemetry—to derive insights or make predictions. Each modality contributes unique context: text for intent, images for visual cues, telemetry for behavioral signals, etc.

Multimodal Fusion AI, by contrast, goes a step further. It doesn’t just process each modality independently—it fuses them into a unified analytical model, allowing cross-modal reasoning. In cybersecurity, that means the AI can correlate a tweet about a new CVE, a malware sample shared on a forum, and suspicious traffic patterns in real time to recognize they’re all part of the same emerging threat.

Dataminr’s AI model uses deep learning to fuse multi-modal signals to capture the full, real-time context of events that other systems will miss or alert to much later. This multi-modal fusion technique outperforms single mode or basic multi-modal techniques and methods focused on one-modal-only inputs.

Evolving Alongside Public Data

As the amount and types of public data increase, Dataminr has integrated those sources into the platform, enabling even more responsive, more accurate alerts.

Consider the earlier plane crash example, which relied on social media posts from on-the-ground observers as a key piece of the story. However, plane crashes—or other incidents—can happen in remote areas with no one around. These events require emergency response, but with no eyewitness reporting events, they require monitoring of a broader range of sources.

Data from sensors—in this case the ADS-B Signals of an airplane transponder—would record a sudden drop in altitude of the plane before the transponder suddenly goes offline. Data captured from the International Civil Aviation Organization (ICAO), including flight plans (i.e., origination, route, planned destination, etc.), might detect a deviation from the expected route and abrupt loss of altitude. 

The plane crash example can be applied to any critical event happening in both the physical or cyber domains (or both) around the world. Once the critical incident has been detected, Dataminr uses Intel Agents and ReGenAI to automatically create real-time and continuously updated summaries for impacted customers as the situation evolves. 

Dataminr’s Multi-Modal Fusion AI Tells the Whole Story

Incident data is often fragmented across a range of data types with varying degrees of accuracy and detail. Each piece of data is useful, but much like a puzzle, can only tell the full story once they have been put together. 

To better equip customers, Dataminr processes billions of public data inputs per day from both the physical and cyber worlds, including text, images, video, sound, and public IoT sensors from hundreds of thousands of distinct data sources. As the data is ingested, AI models leverage natural language understanding, computer vision, sound/audio detection, and anomaly detection in machine-generated data streams to fuse these discrete data modalities. Dataminr’s AI platform processes and cross-correlates these diverse data types to take the multi-modal approach beyond what was previously possible. 

The platform also delivers faster, more accurate real-time intelligence, giving security and emergency response teams critical hours or days of notice—much earlier than other sources—about risk and events that are most actionable to their organization. Organizations are able to respond more quickly, reduce the impact of the incident, and stay better-informed as the situation evolves.