Locality-Sensitive Sketching for Resilient Network Flow Monitoring

by   Yongquan Fu, et al.

Network monitoring is vital in modern clouds and data center networks for traffic engineering, network diagnosis, network intrusion detection, which need diverse traffic statistics ranging from flow size distributions to heavy hitters. To cope with increasing network rates and massive traffic volumes, sketch based approximate measurement has been extensively studied to trade the accuracy for memory and computation cost, which unfortunately, is sensitive to hash collisions. In addition, deploying the sketch involves fine-grained performance control and instrumentation. This paper presents a locality-sensitive sketch (LSS) to be resilient to hash collisions. LSS proactively minimizes the estimation error due to hash collisions with an autoencoder based optimization model, and reduces the estimation variance by keeping similar network flows to the same bucket array. To illustrate the feasibility of the sketch, we develop a disaggregated monitoring application that supports non-intrusive sketching deployment and native network-wide analysis. Testbed shows that the framework adapts to line rates and provides accurate query results. Real-world trace-driven simulations show that LSS remains stable performance under wide ranges of parameters and dramatically outperforms state-of-the-art sketching structures, with over 10^3 to 10^5 times reduction in relative errors for per-flow queries as the ratio of the number of buckets to the number of network flows reduces from 10% to 0.1%.


A Fast and Compact Invertible Sketch for Network-Wide Heavy Flow Detection

Fast detection of heavy flows (e.g., heavy hitters and heavy changers) i...

Sketch for traffic measurement: design, optimization, application and implementation

Network measurement probes the underlying network to support upper-level...

HashFlow For Better Flow Record Collection

Collecting flow records is a common practice of network operators and re...

Scaling Up Anomaly Detection Using In-DRAM Working Set of Active Flows Table

In the zettabyte era, per-flow measurement becomes more challenging owin...

Network Activities Recognition and Analysis Based on Supervised Machine Learning Classification Methods Using J48 and Naïve Bayes Algorithm

Network activities recognition has always been a significant component o...

ALBUS: a Probabilistic Monitoring Algorithm to Counter Burst-Flood Attacks

Modern DDoS defense systems rely on probabilistic monitoring algorithms ...

GB-KMV: An Augmented KMV Sketch for Approximate Containment Similarity Search

In this paper, we study the problem of approximate containment similarit...

Please sign up or login with your details

Forgot password? Click here to reset