A Fast and Efficient Change-point Detection Framework for Modern Data
share
Change-point analysis is thriving in this big data era to address problems arising in many fields where massive data sequences are collected to study complicated phenomena over time. It plays an important role in processing these data by segmenting a long sequence into homogeneous parts for follow-up studies. The task requires the method to be able to process large datasets quickly and to deal with various types of changes for high-dimensional and non-Euclidean data. We propose a novel approach making use of approximate k-nearest neighbor information of the observations, and derive an analytic formula to control the type I error. The time complexity of the method is O(dnlog n) for an n-length sequence of d-dimensional data. Moreover, we incorporate a useful pattern of data in high dimension that the proposed method could detect various types of changes in the sequence.
READ FULL TEXT