Hunter in the Dark: Discover Anomalous Network Activity Using Deep Ensemble Network

by   Shiyi Yang, et al.

Machine learning (ML)-based network intrusion detection system (NIDS) plays a critical role in discovering unknown and novel threats in a large-scale cyberspace. It has been widely adopted as a mainstream hunting method in many organizations, such as financial institutes, manufacturing companies and government agencies. However, there are two challenging issues in the existing designs: 1) achieving excellent performance of threat detection is often at the cost of a large number of false positives, leading to the problem of alert fatigue and 2) the interpretability of detection results is low, making it difficult for the security analyst to obtain the insight of threats and take prompt actions against the attacks. To tackle the above issues, in this paper we propose a defense mechanism, DarkHunter, that includes three parts: stream processor, detection engine and incident analyzer. The stream processor converts raw network packet streams into data records of a set of statistical features that can be effectively used for learning; The detection engine leverages an efficient ensemble neural network (EnsembleNet) to identify anomalous network traffic; The incident analyzer applies a correlation analysis to filter out the mis-predictions from EnsembleNet, traces each detected threat from its statistical representation back to its source traffic flow to enhance its intelligibility and prioritizes the threats to be responded to minimize security risks. Our evaluations, based on the UNSW-NB15 testbed, show that DarkHunter significantly outperforms state-of-the-art ML-based NIDS designs by achieving higher accuracy, higher detection rate, higher precision, higher F1 score while keeping lower false alarm rate.


page 10

page 11


Pelican: A Deep Residual Network for Network Intrusion Detection

One challenge for building a secure network communication environment is...

DualNet: Locate Then Detect Effective Payload with Deep Attention Network

Network intrusion detection (NID) is an essential defense strategy that ...

NetSentry: A Deep Learning Approach to Detecting Incipient Large-scale Network Attacks

Machine Learning (ML) techniques are increasingly adopted to tackle ever...

That Escalated Quickly: An ML Framework for Alert Prioritization

In place of in-house solutions, organizations are increasingly moving to...

Supervised Feature Selection Techniques in Network Intrusion Detection: a Critical Review

Machine Learning (ML) techniques are becoming an invaluable support for ...

AnyThreat: An Opportunistic Knowledge Discovery Approach to Insider Threat Detection

Insider threat detection is getting an increased concern from academia, ...

Holmes: An Efficient and Lightweight Semantic Based Anomalous Email Detector

Email threat is a serious issue for enterprise security, which consists ...

Please sign up or login with your details

Forgot password? Click here to reset