Data Anomaly Detection - What, Why and How?

Data anomaly detection identifies unusual patterns that deviate from expected behavior, known as outliers. Applications span intrusion detection, system health monitoring, fraud detection, and fault identification.

Anomalies fall into three types: Point anomalies occur when individual instances deviate significantly from the rest. Contextual anomalies are abnormalities specific to particular conditions, common in time-series data. Collective anomalies involve groups of instances that together indicate abnormal behavior.

Anomaly detection differs from related concepts. Novelty detection specifically identifies unobserved patterns in new data not seen during training. Noise removal eliminates unwanted signals from meaningful data.

Machine learning approaches for anomaly detection fall into three categories based on label availability. Supervised methods require accurately labeled training data for both normal and anomalous instances but face challenges obtaining representative anomaly examples. Semi-supervised techniques assume only normal class labels in training data, making them widely applicable. Unsupervised methods, requiring no training labels, are most commonly used and assume entire datasets contain primarily normal behavior.

Clustering algorithms are suitable for anomaly detection. DBSCAN, a density-based approach, identifies high-density regions as clusters while marking sparse points as anomalies. Gaussian mixture models use probabilistic approaches with expectation-maximization algorithms to recover underlying distributions.

K-Means clustering proves unsuitable because it forces all points into clusters, potentially incorporating anomalies that distort cluster parameters. This can prevent detecting genuine anomalies in test data.