Gesture Controlled Virtual Mouse with Voice Commands

Abstract: Contactless human–computer interaction has become increasingly important for accessibility, hygiene, and natural user experience. This project presents a practical Gesture Controlled Virtual Mouse with Voice Commands framework that allows users to control the computer cursor and perform system actions using only a standard webcam and microphone. The proposed system leverages Google’s Media Pipe for reliable hand landmark detection and OpenCV for image preprocessing and frame handling, while Speech Recognition (with local microphone input) maps spoken commands to system-level actions. The design emphasizes low computing requirements, real-time responsiveness, and user accessibility. Extensive experimental evaluation under multiple lighting and background scenarios demonstrate an average gesture recognition accuracy of ~96%, voice command recognition accuracy of ~93%, an average latency of ~45 ms, and stable operating frame rates (25– 30 fps) on commodity hardware. The system is lightweight, platform flexible, and suitable for applications in healthcare, education, and assistive technologies.

Keywords: Gesture Recognition, Speech Recognition, MediaPipe, OpenCV, PyAutoGUI, Human–Computer Interaction, Virtual Mouse.