Hand Gesture Recognition Using Spiking Neural Networks in FEAGI

December 12, 2024

As part of a capstone class at the University of Pittsburgh, I was brought on to create an AI model capable of recognizing basic movement gestures on Neuraville's FEAGI platform alongside my teammates, Zhexi Wu and Shawn Li. After months of development, I'm happy to announce that we've made significant progress in getting FEAGI to recognize gestures. Although consistent gesture recognition is just within our reach, we've made inroads in discovering what works and what doesn't with respect to this task on FEAGI.

I think our first (and most) interesting challenge was the fact that we had to limit our work to the FEAGI platform exclusively. Our sponsors, Neuraville, requested that we avoid coding unless absolutely necessary, which meant we would have to leave our expertise as seasoned programmers behind and rely purely on our problem-solving acumen and knowledge of neuroscience. Adjusting to this requirement was surprisingly harder than I expected, especially since some of the other projects being sponsored by Neuraville at this time were more lax in their restrictions on coding. As time went on, though, I began to appreciate the levels of abstract thinking we had to reach in order to make good progress.

Our first major task as a team was to familiarize ourselves with the FEAGI platform. FEAGI was an entirely new software to us at the time and is based on neuromorphic computing, a programming paradigm I'd only heard of tangentially through a Computer Vision (CV) class. However, conversations with our sponsors and the documentation, to some extent, helped us get caught up to speed rather quickly and scale the learning curve with ease. In hindsight, the experience is not unlike picking up a new modelling software like Blender.

On that note: instrumental to our progress this semester was this healthy line of communication with our sponsors. Through weekly meetings and communication over Discord, Neuraville was more than willing to provide guidance despite certain gaps in the documentation as well as insights during times we were stuck. They were even kind enough to share their own in-house genomes with us to give us ideas on how to proceed.

Our genome for gesture recognition has evolved consistently over the last few months. Here's a diagram we developed over the course of the project to serve as a foundation for future work:

FEAGI imitation learning pipeline diagram

Since then, we've had significant changes to not just our genome, but our mindset, knowledge, and methods of tackling the problem statement. Currently, our approach to gesture recognition can broadly be divided into three rough areas:

Input
Gesture data processing
Training / testing

In addition to a three-dimensional visualizer, FEAGI also has a circuit visualizer highlighting the connections between each cortical area in greater detail. In FEAGI, the diagram above roughly translates to the following circuit, which I'll explain in more detail in the coming sections:

Input

In the input stage, we use the visual cortical area to register all visual data that is fed into FEAGI. This is also where we process the inputs to remove any excess noise, either by tweaking the firing thresholds of the cortical area or by using lighting enhancement areas. We found that the former option works best.

Gesture data processing

Processing the gesture data effectively is where things get interesting. We spent a while trying to figure out the most "optimal" way to extract key features from the input data for the training / testing process to go smoothly. Ultimately, we decided on an approach combining traditional CV techniques such as downsampling alongside FEAGI's unique ability to track spatiotemporal data.

We process the input data extensively to build a kind of "signature" that can uniquely identify a particular gesture. This data is combined and stored within a memory unit (labelled combined_mem in the gif above), which in turn passes that data to the training and testing region.

Input circuit layout for gesture feature extraction

Training / Testing

Training region showing ID recognition and motor outputs

The training / testing region of our genome required the most amount of time and effort on our part. We met extensively with Neuraville to iron out the kinks of this region, which broadly involves the ID Trainer area, the ID Recognition area, the memory unit, and indicator areas ("forward", "back", etc.).

We use a supervised training approach where labeled videos are fed into the genome by means of an external controller script. Each video's label corresponds to the voxels of the ID Trainer.

For example, a video containing a leftward motion might be labeled 0-2-0, meaning it would correspond to the third voxel of the Trainer (because of zero-based indexing!). At the same time, this video is sent through the processing pipeline described in the previous section and its signatures are stored in the memory unit, which feeds them to the ID Recognition area via a neuroplastic connection. In neuroscience, neuroplasticity represents the brain's ability to "learn" new memories and experiences by reorganizing and developing neural pathways. Neuroplastic connections serve a similar role in FEAGI by generating neural connections to learn gesture signatures.

circuit builder with ID trainer, ID recognition, combined features

Looking at the circuit for the training / testing region, we see that the ID trainer holds two important connections to the ID recognition area—the first is an excitatory connection that promotes "correct" associations, meaning that the ID recognition should flash the same voxel as the ID Trainer. The second is an inhibitory connection that suppresses wrong associations. In tandem, both connections help the model associate each video with the corresponding voxel.

Excitatory and inhibitory connections in the training circuit

Neuraville provided us with a Python-based script that handles the training process on the backend. As it runs, we can access a dashboard through localhost:4001. This area displays key information, such as the fitness of the model during the training process, the current labeled video being displayed, and options to adjust how long each video is exposed to the genome.

After the model has been sufficiently trained, we test it by feeding it new data in the form of visual inputs through the webcam embodiment. Ideally, the recognition area should recognize the gesture being performed, associate it with one of its learned labels, and route its output to one of the indicator areas.

Motor OPU outputs wired to ID recognition signals

Note that each indicator area connects to a motor_opu unit. This cortical area marks the point where the gesture recognition genome can be integrated with other genomes; for instance, the motor_opu could be swapped out with a separate cortical area that controls character movement in some of the games integrated on the FEAGI platform. This would allow the user to move their character with hand gestures!

This leads us to today. As I said before, although we've had great results getting our model to recognize basic gestures, consistent recognition has been a little more difficult, primarily, we suspect, due to the way certain parameters on each of the cortical areas have been configured. Currently, we need to strike a good balance between the strength of the connections between the Trainer and ID Recogntion as well as the sensitivity of the ID Recognition to inputs from the memory unit. Solving this might require an even deeper knowledge of neuroscience than what we've learned over the course of this project.

We're confident that the right configurations are just around the corner and that we'll be able to discover them through the same problem-solving abilities that brought us this far. We have rather high hopes for this project and its ambitions; for the short term, future work involves integrating this genome with a simulated 3D car, real-life robot control, and the aforementioned video games on the FEAGI platform. In the long term, we picture this project being used for even more sophisticated tasks, like sign language recognition.

Since the semester began, we've not only acclimated to an entirely new and computing paradigm, but have also sharpened our ability to think through problems both as individuals and as a team. As a bonus, we've learned quite a bit about neuroscience along the way. I'm very thankful for this unique opportunity, and I wish Neuraville as well as my teammates the best of luck with the future!

This post was contributed by a member of the NRS community and reflects their own work and findings.