Publications

Real-Time Closed-Loop Feedback System For Mouse Mesoscale Cortical Signal And Movement Control: CloPy

Published in bioRxiv, 2024

We present the implementation and efficacy of an open-source closed-loop neurofeedback (CLNF) and closed-loop movement feedback (CLMF) system. In CLNF, we measure mm-scale cortical mesoscale activity with GCaMP6s and provide graded auditory feedback (within ∼50 ms) based on changes in dorsal-cortical activation within regions of interest (ROI) and with a specified rule. Single or dual ROIs (ROI1, ROI2) on the dorsal cortical map were selected as targets. Both motor and sensory regions supported closed-loop training in male and female mice. Mice modulated activity in rule-specific target cortical ROIs to get increasing rewards over days (RM ANOVA p=2.83e-5) and adapted to changes in ROI rules (RM ANOVA p=8.3e-10, Table 4 for different rule changes). In CLMF, feedback was based on tracking a specified body movement, and rewards were generated when the behavior reached a threshold. For movement training, the group that received graded auditory feedback performed significantly better (RM-ANOVA p=9.6e-7) than a control group (RM-ANOVA p=0.49) within four training days. Additionally, mice can learn a change in task rule from left forelimb to right forelimb within a day, after a brief performance drop on day 5. Offline analysis of neural data and behavioral tracking revealed changes in the overall distribution of ΔF/F0 values in CLNF and body-part speed values in CLMF experiments. Increased CLMF performance was accompanied by a decrease in task latency and cortical ΔF/F0 amplitude during the task, indicating lower cortical activation as the task gets more familiar.

Recommended citation: PK Gupta, TH Murphy https://www.biorxiv.org/content/10.1101/2024.11.02.619716v1.full

PyMouseTracks: flexible computer vision and RFID-based system for multiple mouse tracking and behavioral assessment

Published in eNeuro, 2023

PyMouseTracks (PMT) is a scalable and customizable computer vision and radio frequency identification (RFID)-based system for multiple rodent tracking and behavior assessment that can be set up within minutes in any user-defined arena at minimal cost. PMT is composed of the online Raspberry Pi (RPi)-based video and RFID acquisition with subsequent offline analysis tools. The system is capable of tracking up to six mice in experiments ranging from minutes to days. PMT maintained a minimum of 88% detections tracked with an overall accuracy >85% when compared with manual validation of videos containing one to four mice in a modified home-cage. As expected, chronic recording in home-cage revealed diurnal activity patterns. In open-field, it was observed that novel noncagemate mouse pairs exhibit more similarity in travel trajectory patterns than cagemate pairs over a 10-min period. Therefore, shared features within travel trajectories between animals may be a measure of sociability that has not been previously reported. Moreover, PMT can interface with open-source packages such as DeepLabCut and Traja for pose estimation and travel trajectory analysis, respectively. In combination with Traja, PMT resolved motor deficits exhibited in stroke animals. Overall, we present an affordable, open-sourced, and customizable/scalable mouse behavior recording and analysis system.

Recommended citation: T Fong, H Hu, P Gupta, B Jury, TH Murphy https://www.eneuro.org/content/10/5/ENEURO.0127-22.2023

Using Computational Analysis of Behavior To Discover Developmental Change In Memory-Guided Attention Mechanisms In Childhood

Published in PsyArXiv, 2021

We tested 4-9.5-year-old children on a naturalistic memory-guided attention visual search task. We measured fixation distribution during a search using wearable eye tracking, and simultaneously recorded depth video data for each participant and used computer vision algorithms to track them during navigation. We manipulated object placement and trial order such that nearby objects would be encountered during initial search for reference objects. We used a computational model of top-down guidance for reference object visual features and examined the use of this top-down attention for reference objects during subsequent nearby object search. The data suggest that the value of physical navigation during initial spatial exploration for subsequent memory-guided attention, specifically in early childhood, is in its association with stronger visual representations of goal reference objects during spatial exploration. By middle childhood, visual search times were not impacted by memory engagement.

Recommended citation: Dima Amso, Lakshmi Govindarajan, Pankaj Gupta, Diego Placido, Heidi Baumgartner, Andrew Lynn, Kelley Gunther, Tarun Sharma, Vijay Veerabadran, Kalpit Thakkar, Seung Chan Kim, Thomas Serre https://psyarxiv.com/gq4rt/

A three-dimensional virtual mouse generates synthetic training data for behavioral analysis

Published in Nature Methods, 2021

We developed a three-dimensional (3D) synthetic animated mouse based on computed tomography scans that is actuated using animation and semirandom, joint-constrained movements to generate synthetic behavioral data with ground-truth label locations. Image-domain translation produced realistic synthetic videos used to train two-dimensional (2D) and 3D pose estimation models with accuracy similar to typical manual training datasets. The outputs from the 3D model-based pose estimation yielded better definition of behavioral clusters than 2D videos and may facilitate automated ethological classification.

Recommended citation: Luis A. Bolaños, Dongsheng Xiao, Nancy L. Ford, Jeff M. LeDue, Pankaj K. Gupta, Carlos Doebeli, Hao Hu, Helge Rhodin & Timothy H. Murphy https://www.nature.com/articles/s41592-021-01103-9

Real-time neural feedback of mesoscale cortical GCAMP6 signals for training mice

Published in CoSyne 2021, 2021

Mice can learn to control specific neuronal ensembles using sensory (eg. auditory) cues (Clancy et al. 2014) or even artificial optogenetic stimulation (Prsa et al. 2017). In the present work, we measure mesoscale cortical activity with GCaMP6s and provide graded auditory feedback (within ~100 ms after GCaMP fluorescence) based on changes in dorsal-cortical activation within specified regions of interest (ROI)s with a specified rule. We define a compact, low-cost optical brain-machine-interface (BMI) capable of image acquisition, processing, and conducting closed-loop auditory feedback and rewards, using a Raspberry Pi (Fig. 1). The changes in fluorescence activity (ΔF/F) are calculated based on a running baseline (eg. 5 sec.). Two ROIs (R1, R2) on the dorsal cortical map were selected as targets. We started with a rule of ‘R1-R2’ (ΔF/F of R1 minus ΔF/F of R2) where the activity of R1 relative to R2 was mapped to frequency of the audio feedback (Fig. 1D) and if it were to cross a set threshold, a water drop reward is generated. To investigate learning in this context, water-deprived tetO-GCaMP6s mice (N=8) were trained for 30-minutes every day on the system for several days, with a task to increase audio frequency leading to reward. We found that mice could modulate activity in the rule-specific target ROIs to get an increasing number of rewards over days (Figure 2C). Analysis of the reward-triggered ΔF/F over time indicated that mice progressively learned to activate the cortical ROI to a greater extent (Figure 2B, 2A). In conclusion, we developed an open-source system (to-be released) for closed-loop feedback that can be added to experimental scenarios for brain activity training and could be possibly effective in inducing neuroplasticity.

Recommended citation: Pankaj K Gupta, Timothy H Murphy https://www.cosyne.org/s/Cosyne2021_program_book.pdf

Neuromatch Academy- a 3-week, online summer school in computational neuroscience

Published in Journal of Open Source Education, 2021

Neuromatch Academy (https://neuromatch.io/academy) was designed as an online summer school to cover the basics of computational neuroscience in three weeks. The materials cover dominant and emerging computational neuroscience tools, how they complement one another, and specifically focus on how they can help us to better understand how the brain functions. An original component of the materials is its focus on modeling choices, i.e. how do we choose the right approach, how do we build models, and how can we evaluate models to determine if they provide real (meaningful) insight. This meta-modeling component of the instructional materials asks what questions can be answered by different techniques, and how to apply them meaningfully to get insight about brain function.

Recommended citation: 't Hart, Bernard M., et al. https://jose.theoj.org/papers/101bd5b60c63dc778dfcb9da787820b1

The value of choice in 3- to 7-year-olds’ use of working memory gating strategies in a naturalistic task

Published in Developmental Science, 2020

Rule-guided behavior depends on the ability to strategically update and act on content held in working memory. Proactive and reactive control strategies were contrasted across two experiments using an adapted input/output gating paradigm (Neuron, 81, 2014 and 930). Behavioral accuracies of 3-, 5-, and 7-year-olds were higher when a contextual cue appeared at the beginning of the task (input gating) rather than at the end (output gating). This finding supports prior work in older children, suggesting that children are better when input gating but rely on the more effortful output gating strategy for goal-oriented action selection (Cognition, 155, 2016 and 8). A manipulation was added to investigate whether children's use of working memory strategies becomes more flexible when task goals are specified internally rather than externally provided by the experimenter. A shift toward more proactive control was observed when children chose the task goal among two alternatives. Scan path analyses of saccadic eye movement indicated that giving children agency and choice over the task goal resulted in less use of a reactive strategy than when the goal was determined by the experimenter.

Recommended citation: Livia Freier,Pankaj Gupta,David Badre,Dima Amso https://onlinelibrary.wiley.com/doi/full/10.1111/desc.13017

Real-time selective markerless tracking of forepaws of head fixed mice using deep neural networks

Published in eNeuro, 2020

Here, we describe a system capable of tracking specific mouse paw movements at high frame rates (70.17 Hz) with a high level of accuracy (mean = 0.95, SD < 0.01). Short-latency markerless tracking of specific body parts opens up the possibility of manipulating motor feedback. We present a software and hardware scheme built on DeepLabCut—a robust movement-tracking deep neural network framework—which enables real-time estimation of paw and digit movements of mice. Using this approach, we demonstrate movement-generated feedback by triggering a USB-GPIO (general-purpose input/output)-controlled LED when the movement of one paw, but not the other, selectively exceeds a preset threshold. The mean time delay between paw movement initiation and LED flash was 44.41 ms (SD = 36.39 ms), a latency sufficient for applying behaviorally triggered feedback. We adapt DeepLabCut for real-time tracking as an open-source package we term DeepCut2RealTime. The ability of the package to rapidly assess animal behavior was demonstrated by reinforcing specific movements within water-restricted, head-fixed mice. This system could inform future work on a behaviorally triggered “closed loop” brain–machine interface that could reinforce behaviors or deliver feedback to brain regions based on prespecified body movements.

Recommended citation: Brandon J. Forys, Dongsheng Xiao, Pankaj Gupta and Timothy H. Murphy https://www.eneuro.org/content/7/3/ENEURO.0096-20.2020

Cortex-wide Computations in Complex Decision Making in Mice

Published in Neuron, 2019

Seemingly, a paradox exists between reports of wide-scale task-dependent cortical activity and the causal requirement for only a restricted number of motor and sensory cortical areas in some behavioral studies. In this issue of Neuron, Pinto et al. (2019) indicate that scenarios where mice must accumulate evidence and hold it during a delay period are causally linked to wide regions of cortex.

Recommended citation: Pankaj K Gupta, Timothy H Murphy https://pubmed.ncbi.nlm.nih.gov/31751543/

What are the visual features underlying human versus machine vision?

Published in 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), 2017

Although Deep Convolutional Networks (DCNs) are approaching the accuracy of human observers at object recognition, it is unknown whether they leverage similar visual representations to achieve this performance. To address this, we introduce Clicktionary, a web-based game for identifying visual features used by human observers during object recognition. Importance maps derived from the game are consistent across participants and uncorrelated with image saliency measures. These results suggest that Clicktionary identifies image regions that are meaningful and diagnostic for object recognition but different than those driving eye movements. Surprisingly, Clicktionary importance maps are only weakly correlated with relevance maps derived from DCNs trained for object recognition. Our study demonstrates that the narrowing gap between the object recognition accuracy of human observers and DCNs obscures distinct visual strategies used by each to achieve this performance.

Recommended citation: Drew Linsley, Sven Eberhardt, Tarun Sharma, Pankaj Gupta, Thomas Serre https://ieeexplore.ieee.org/document/8265530/authors#authors

Multi-sensor based state prediction for personal mobility vehicles

Published in PLOS ONE, 2016

This paper presents a study on multi-modal human emotional state detection while riding a powered wheelchair (PMV; Personal Mobility Vehicle) in an indoor labyrinth-like environment. The study reports findings on the habituation of human stress response during self-driving. In addition, the effects of “loss of controllability”, change in the role of the driver to a passenger, are investigated via an autonomous driving modality. The multi-modal emotional state detector sensing framework consists of four sensing devices: electroencephalograph (EEG), heart inter-beat interval (IBI), galvanic skin response (GSR) and stressor level lever (in the case of autonomous riding). Physiological emotional state measurement characteristics are organized by time-scale, in terms of capturing slower changes (long-term) and quicker changes from moment-to-moment. Experimental results with fifteen participants regarding subjective emotional state reports and commercial software measurements validated the proposed emotional state detector. Short-term GSR and heart signal characterizations captured moment-to-moment emotional state during autonomous riding (Spearman correlation; ρ = 0.6, p < 0.001). Short-term GSR and EEG characterizations reliably captured moment-to-moment emotional state during self-driving (Classification accuracy; 69.7). Finally, long-term GSR and heart characterizations were confirmed to reliably capture slow changes during autonomous riding and also of emotional state during participant resting state. The purpose of this study and the exploration of various algorithms and sensors in a structured framework is to provide a comprehensive background for multi-modal emotional state prediction experiments and/or applications. Additional discussion regarding the feasibility and utility of the possibilities of these concepts are given.

Recommended citation: Jamilah Abdur-Rahim ,Yoichi Morales ,Pankaj Gupta ,Ichiro Umata ,Atsushi Watanabe ,Jani Even ,Takayuki Suyama,Shin Ishii https://journals.plos.org/plosone/article/authors?id=10.1371/journal.pone.0162593

Brain-machine interfaces for assistive smart homes- A feasibility study with wearable near-infrared spectroscopy

Published in IEEE-EMBS 2015, 2015

Smart houses for elderly or physically challenged people need a method to understand residents' intentions during their daily-living behaviors. To explore a new possibility, we here developed a novel brain-machine interface (BMI) system integrated with an experimental smart house, based on a prototype of a wearable near-infrared spectroscopy (NIRS) device, and verified the system in a specific task of controlling of the house's equipments with BMI. We recorded NIRS signals of three participants during typical daily-living actions (DLAs), and classified them by linear support vector machine. In our off-line analysis, four DLAs were classified at about 70% mean accuracy, significantly above the chance level of 25%, in every participant. In an online demonstration in the real smart house, one participant successfully controlled three target appliances by BMI at 81.3% accuracy. Thus we successfully demonstrated the feasibility of using NIRS-BMI in real smart houses, which will possibly enhance new assistive smart-home technologies.

Recommended citation: Takeshi Ogawa, Jun-Ichiro Hirayama, Pankaj Gupta, Hiroki Moriya, Shumpei Yamaguchi, Akihiro Ishikawa, Yoshihiro Inoue, Motoaki Kawanabe, Shin Ishii https://pubmed.ncbi.nlm.nih.gov/26736459/