Voice fraud has recently become a critical cybersecurity concern, where fraudsters impersonate trusted individuals over phone calls to deceive victims and obtain confidential information. Conventional fraud detection systems, which rely on predetermined rules or post-call detection, are increasingly becoming ineffective against evolving fraud patterns and do not ensure user safety in real-time. This study proposes the design and implementation of a real-time voice fraud detection system that analyses real-time conversations to identify potential fraudulent activity. The proposed system integrates OpenAI’s Whisper model for speech-to-text transcription with a fine-tuned Phi-3-mini Large Language Model (LLM) for contextual fraud detection. The system architecture uses the Browser MediaStream API for audio capture and WebSocket communication to transmit streaming data to a FastAPI-based backend for real-time processing. The fine-tuned LLM was trained on a dataset comprising 447 labeled conversational samples and subsequently evaluated on 307 unseen test samples, achieving an overall accuracy of 95%. Experimental results demonstrate a precision of 97% and a recall of 98% for normal conversation, and a precision of 75% and a recall of 67% for fraudulent conversation. These findings underscore the effectiveness of leveraging context-aware language modelling and real-time audio transcription in detecting fraudulent speech patterns, thereby surpassing the limitations of traditional rule-based systems. The study contributes to the body of knowledge by demonstrating the feasibility of combining speech processing and large language models to enable proactive, real-time detection of voice fraud during phone calls.

Audio Stream Processing Deep Learning Large Language Models (LLMs) Real-Time Fraud Detection Speech-to-Text Processing