Integrating MuJoCo with FEAGI

FEAGI

An image to show what FEAGI's brain looks like from the Brain Visualizer. Where the red voxels represent firing neurons.

MuJoCo

Our humanoid model standing in MuJoCo, waiting for signals from FEAGI.

We were tasked with creating a Python controller to interact with both the FEAGI SDK and MuJoCo SDK, and to begin exploration of how our artificial brain could take control of the human model. The purpose of this is to eventually make the humanoid stand, walk, and even be aware of its surroundings to dodge obstacles.

Integration and Development

By default, when using an object in MuJoCo, that object becomes subject to the forces of gravity. Of course, in order to do anything fun, we first needed to ensure that the humanoid could stay standing unless told to do otherwise. This was our very first milestone - to somehow tell MuJoCo to make the object stay standing. We achieved this by creating a simple controller program to interact with MuJoCo while its running, and printed out as much data as we could find. Once that data had been parsed, we could successfully map the joints and control them programmatically.

After positioning our humanoid model, we turned our attention to the task of bridging FEAGI and MuJoCo. This proved to be a complex challenge, which we tackled by breaking it down into smaller, more manageable steps.

With the humanoid model in a stable standby position, we focused on modifying the existing controller. This controller, originally designed to maintain the humanoid's posture, now needed to interface seamlessly with both FEAGI and MuJoCo. To achieve this, we delved into the APIs provided by each tool, establishing a communication link between them. The controller, running locally, connects to FEAGI within a Docker environment, enabling the MuJoCo viewer to open. This integrated setup would allow us to harness FEAGI's neuroplastic learning capabilities and apply them to the humanoid model's movements and sensory inputs.

Next was to implement a functional pipeline that allowed MuJoCo's physics simulation to send sensory data, such as joint angles and contact forces, back to FEAGI. FEAGI then processed this data and sent motor commands to MuJoCo, completing the control loop. We developed a system to translate FEAGI's neural outputs into MuJoCo-compatible actuator inputs, ensuring a smooth integration.

A significant milestone was achieved by implementing a basic standing posture for the humanoid. This required careful tuning of both FEAGI's neural responses and the physical parameters of the humanoid model, ensuring stability while emulating natural human posture. A feedback loop was established, enabling FEAGI to learn and adapt its control strategy, leading to gradual improvements in balance and responsiveness to simulated environmental changes.

Throughout the development process, we diligently debugged and refined our approach, ensuring seamless and real-time integration between FEAGI and MuJoCo. The culmination of these efforts resulted in a humanoid capable of executing simple motor actions under the control of an artificial brain, laying the foundation for the exploration of more complex behaviors.

Challenges with Integration

Integrating a camera into MuJoCo presented a unique challenge. Our initial approach involved capturing RGB data and feeding it into FEAGI for processing. While MuJoCo's OpenGL rendering functions offered a potential solution, compatibility issues with macOS prevented simultaneous operation of OpenGL and the MuJoCo viewer.

To overcome this limitation, we adopted a more versatile approach: utilizing the built-in rangefinder sensor. This sensor, modeled after LiDAR technology, emits a ray of light and measures the distance to the first intersecting geometry. By strategically placing multiple rangefinders on the humanoid model, we could collect distance data from various angles, effectively simulating a field of view. This data, in turn, can be used to construct an image within FEAGI, serving as a suitable substitute for RGB data.

The Future of FEAGI

The future holds immense potential for FEAGI. As we continue to refine the neural network and explore new applications, we anticipate groundbreaking advancements in artificial intelligence. While our initial focus was on integrating FEAGI with MuJoCo to control a humanoid model, this project has laid the foundation for future research into more complex behaviors and tasks.

Reflecting on our experience, we are incredibly proud of what our team has accomplished. This project has been a thrilling journey, allowing us to explore cutting-edge AI technology and contribute to the advancement of the field. While we encountered numerous challenges, we persevered, learning and growing with each hurdle. The support and guidance provided by the Neuraville team were invaluable, fostering a collaborative and inspiring environment. We are excited to see the future of FEAGI and the groundbreaking applications it will enable.