Identity-verified Gesture Recognition based on Time-of-Flight Sensors

Smart indoor environments increasingly rely on natural, touch-free interfaces to enable seamless Human-Computer Interaction (HCI). Gesture recognition has emerged as an intuitive approach for triggering actions in such spaces. However, conventional RGB-based systems raise two fundamental concerns: privacy, since color cameras capture personally identifiable visual information, and access control, since gesture commands can be issued by any person in the scene regardless of their authorization. This thesis proposes a privacy-conscious, identity-verified gesture recognition system for indoor smart environments. A single Time-of-Flight (ToF) depth camera mounted in an overhead position provides continuous depth sensing without capturing color or texture information. An Inertial Measurement Unit (IMU) worn on the wrist of the authorized person serves as a passive identity token. The system fuses depth-based person tracking with IMU motion signals to determine which tracked agent corresponds to the authorized user, enabling gesture commands to be accepted exclusively from that individual without requiring prior user enrollment or storing any kind of personal information of users. The processing pipeline encompasses depth-based foreground segmentation, morphological filtering, blob detection, Kalman filter tracking, confidence-based IMU-vision fusion for authorization, IMU-driven gesture segmentation, and Dynamic Time Warping (DTW) template matching for gesture classification. The system supports both single-agent and multi-agent scenarios. The system is evaluated on a dataset of 57 recorded scenarios across 11 participants. Offline gesture recognition achieves 87.0% overall accuracy with a micro F1-score of 0.906 and near-perfect precision of 0.997, confirming reliable rejection of out-of-distribution movements. In single-agent online scenarios, authorization achieves an F1-score of 0.951 (accuracy: 93.0%) with 99.7% of authorized time attributed to the correct agent. In multiagent scenarios, authorization achieves an F1-score of 0.723 (accuracy: 73.3%) and close to 80% of authorized time attributed to the correct agent, with the reduction attributed primarily to transient false authorizations caused by simultaneous motion of multiple tracked agents. Across all scenarios, 88.7% of total authorized time is correctly attributed to the ground-truth agent. Online gesture precision is 1.000 across all scenarios, with recognition latency averaging 2.97 s end-to-end. The results demonstrate the feasibility of privacy-preserving, identity-aware gesture interaction using a single ToF camera and a wearable IMU, without relying on RGB imagery or prior biometric registration.