Git Product home page Git Product logo

maif's Introduction

Multimodal-AIF (MAIF)

Multimodal VAE Active Inference Controller, published on IROS2021.

Preprint: Cristian Meo and Pablo Lanillos (2021). "Multimodal VAE Active Inference Controller". https://arxiv.org/abs/2103.04412#

This repository includes the code for the MAIF torque controller code and the tests with the 7DOF Panda robot.

Requirements

  • ROS (melodic)
  • pytorch 1.7.0
  • cv2
  • seaborn 0.11.0

Installation

Once the dependencies are installed, a catkin workspace has to be created. To do it:

  • Create a folder for your catkin_ws: $ mkdir -p your_catkin_ws/src
  • Move to the folder: $ cd your_catkin_ws/src
  • Clone the repository $ git clone https://github.com/Cmeo97/MAIF
  • Move back to your_catkin_ws: $ cd ..
  • Build the workspace: $ catkin_make
  • Source: $ source devel/setup.bash

Running the code

To run the controller:

  • After building and sorcing the workspace you have to launch the simulation: $ roslaunch panda_simulation simulation_py.launch The launch file launches a Gazebo simulation in pause, start the simulation pressing Gazebo play button.

  • Go to the controller folder: $ cd src/panda_simulation/panda_control_MAIF/src

  • You have to run the camera node, which subscribe images from Gazebo and publish them for the controller: $ python2.7 camera.py

  • then, in another terminal, run the controller: $ python MAIF_controller.py

  • To run the Mental simulation, go to the controller folder: $ cd src/panda_simulation/panda_control_MAIF/src

  • and run: $ python Brain_Simulation.py

maif's People

Contributors

cmeo97 avatar pl-robotdecision avatar

Stargazers

Yanwen Sun avatar AGI avatar  avatar Alexander Tschantz avatar

Watchers

 avatar

maif's Issues

How to train VAE for MAIF?

I want to know how to setup training procedure for MAIF from 1st step.
A procedure in README seems running inference and action code using trained model
Can you tell me the training step??

Thanks for sharing codes!

Logger script not generating images

Greetings,

I am trying to generate a dataset similar to the one used in the paper, using the logger.py script found in src/panda_simulation/MVAE/src/poses, but while the joint positions are saved to text files there are no camera_image jpeg files generated.

Looking at the code it seems the script subscribes to the /camera/color/image_raw topic but checking it with rostopic shows that there are no publishers running.

(robostackenv) asolano@p-shared-6:~/catkin_ws$ rostopic info /camera/color/image_raw
Type: sensor_msgs/Image

Publishers: None

Subscribers: 
 * /camera (http://p-shared-6:42577/)
 * /logger (http://p-shared-6:36257/)

And searching through code there no other references to it. The setup is the one recommended on the README file:

# terminal 1 (running on a headless server)
$ roslaunch headless:=true gui:=false paused:=false panda_simulation simulation_py.launch

# terninal 2
$ python camera.py

# terminal 3
$ python logger.py

Note there is no controller running since the MAIF_controller.py only runs for a few seconds, but I would expect the logger to save the default image over and over.

Looking at the launch file /src/panda_simulation/franka_description/robots/panda_arm_world.urdf.xacro
it seems the <!-- Camera --> section is empty, is that by design?

I am not very familiar with the Panda + Gazebo setup but the paper mentions "A camera model" but it gives no specifics otherwise. Any hint on how to recreate the dataset (or better yet, a link to a copy!) would be much appreciated.

Thanks,

Alfredo

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.