Decoding multi-limb movements from low temporal resolution calcium imaging using deep learning

Two-photon imaging has been a critical tool for dissecting brain circuits and understanding brain function. However, relating slow two-photon calcium imaging data to fast behaviors has been challenging due to relatively low imaging sampling rates, thus limiting potential applications to neural prostheses. Here, we show that a recurrent encoder-decoder network with an output length longer than the input length can accurately decode limb trajectories of a running mouse from two-photon calcium imaging data. The encoder-decoder model could accurately decode information about all four limbs (contralateral and ipsilateral front and hind limbs) from calcium imaging data recorded in a single cortical hemisphere. Furthermore, neurons that were important for decoding were found to be well-tuned to both ipsilateral and contralateral limb movements, showing that artificial neural networks can be used to understand the function of the brain by identifying sub-networks of neurons that correlate with behaviors of interest.