A Hybrid Iterative Learning Framework for Anomaly Detection

Detection of anomalies is an important data analysis technique that can alert users about underlying issues in a system. Most anomaly detection algorithms today rely only on pure statistical approaches, which, although rigorous, cannot to take into account many of contextual nuances of the anomaly. This results in generation of many false alerts, wasting valuable time in the subsequent root cause analysis that follows the detected anomaly events. In this paper, we propose a hybrid framework that takes into account human expert feedback, and augments statistical anomaly detection with machine learning to learn the contextual aspects of significant anomalies for which alerts should be generated. Our framework is domain agnostic and independent of the underlying statistical anomaly detection technique or the machine learning algorithm, and therefore easily adaptable to many more real world applications such as detecting network performance degradation or abnormal events observed in video streams. We evaluate our framework with data from a mobile network operator and show that it can better detect anomalies with significance resulting in much smaller number of false positives