An Adaptive Stream-Native Anomaly Detection Framework Using Hybrid Unsupervised Learning

Abstract: Real-time anomaly detection in high-velocity data streams is critical for modern distributed systems, financial transactions, and IoT environments. Traditional batch-based anomaly detection techniques fail to adapt to evolving data distributions and concept drift. This paper proposes an adaptive stream-native anomaly detection framework using hybrid unsupervised learning techniques combining Isolation Forest and Autoencoder models. The system is designed over Apache Kafka-based streaming architecture to process continuous data with minimal latency. Experimental evaluation on real-world streaming datasets demonstrates improved detection accuracy and reduced false positive rates compared to standalone models. The proposed framework enables scalable, adaptive, and efficient anomaly detection in dynamic environments.

Keywords: Stream Processing, Anomaly Detection, Hybrid Unsupervised Learning, Isolation Forest, Autoencoder, Apache Kafka