Freebie: A Multi-Camera System for Real-Time Markerless Hand Tracking in Non-Human Primates

Objective. Neural decoders for dexterous finger movements are limited by the accuracy with which ground truth movement is measured. In non-human primates (NHPs), hand kinematics are typically recorded with instrumentation that constrains movement dimensionality. We evaluated whether a markerless camera system could replace conventional methods for tracking movements in real time. Approach. We developed Freebie, a real-time markerless system for tracking NHP finger movements. Keypoints detected online were used in place of manipulandum bend sensor signals. Two NHPs performed tasks by flexing fingers to move a virtual hand into target zones. The virtual hand was directly controlled using bend sensors or Freebie-derived kinematics, or indirectly controlled through a Kalman filter (KF) decoding kinematics from forearm electromyographic (EMG) activity. We quantified Freebie's accuracy relative to manual labels, agreement with bend sensor measurements, and latencies. We assessed online performance under Freebie control and compared EMG KFs trained using Freebie-derived labels with those trained using bend-sensor-derived labels. Main Results. We observed a median end-to-end latency of 15.8 ms at a 30 FPS camera acquisition rate, and 16.2 ms at 60 FPS. Freebie labels closely tracked bend sensor positions (median |r|=0.96). Median keypoint errors were 1.7 to 6.5 pixels, below the 20-pixel width of the smallest tracked finger. Direct Freebie control did not differ from bend sensor control on five of six online metrics; path efficiency was higher with Freebie. EMG KFs trained on Freebie-derived labels performed similarly to those trained on bend sensor labels online. When evaluated offline, models trained with Freebie had lower error. Significance. These findings indicate that bend sensors are not required to obtain finger labels for training a decoder: markerless keypoints alone provided effective labels. This is an important first step towards real-time tracking of more complex movements, beyond what could be easily measured with a manipulandum.