Multivariate Big Data Analysis for Intrusion Detection: 5 steps from the haystack to the needle

by   José Camacho, et al.

The research literature on cybersecurity incident detection & response is very rich in automatic detection methodologies, in particular those based on the anomaly detection paradigm. However, very little attention has been devoted to the diagnosis ability of the methods, aimed to provide useful information on the causes of a given detected anomaly. This information is of utmost importance for the security team to reduce the time from detection to response. In this paper, we present Multivariate Big Data Analysis (MBDA), a complete intrusion detection approach based on 5 steps to effectively handle massive amounts of disparate data sources. The approach has been designed to deal with the main characteristics of Big Data, that is, the high volume, velocity and variety. The core of the approach is the Multivariate Statistical Network Monitoring (MSNM) technique proposed in a recent paper. Unlike in state of the art machine learning methodologies applied to the intrusion detection problem, when an anomaly is identified in MBDA the output of the system includes the detail of the logs of raw information associated to this anomaly, so that the security team can use this information to elucidate its root causes. MBDA is based in two open software packages available in Github: the MEDA Toolbox and the FCParser. We illustrate our approach with two case studies. The first one demonstrates the application of MBDA to semistructured sources of information, using the data from the VAST 2012 mini challenge 2. This complete case study is supplied in a virtual machine available for download. In the second case study we show the Big Data capabilities of the approach in data collected from a real network with labeled attacks.


page 1

page 2

page 3

page 4


A Case Study on Using Deep Learning for Network Intrusion Detection

Deep Learning has been very successful in many application domains. Howe...

Extending Isolation Forest for Anomaly Detection in Big Data via K-Means

Industrial Information Technology (IT) infrastructures are often vulnera...

ZYELL-NCTU NetTraffic-1.0: A Large-Scale Dataset for Real-World Network Anomaly Detection

Network security has been an active research topic for long. One critica...

Networkmetrics unraveled: MBDA in Action

We propose networkmetrics, a new data-driven approach for monitoring, tr...

Analyzing Root Causes of Intrusion Detection False-Negatives: Methodology and Case Study

Intrusion Detection Systems (IDSs) are a necessary cyber defense mechani...

Anomaly Detection for Industrial Big Data

As the Industrial Internet of Things (IIoT) grows, systems are increasin...

Importance of Small Probability Events in Big Data: Information Measures, Applications, and Challenges

In many applications (e.g., anomaly detection and security systems) of s...

Please sign up or login with your details

Forgot password? Click here to reset