Git Product home page Git Product logo

ur-reaching-reinforcement-learning's Introduction

Object tracking video using Reinforcement learning

Object tracking video using Reinforcement learning

How to launch original env

First launch the simulator

roslaunch ur_robotiq_gazebo gym.launch

And run the training launch

roslaunch ur_training default.launch

Conveyer GAZEBO env

First launch the gazebo and gym interface and node publishing block point.

roslaunch ur_robotiq_gazebo conveyer_gym.launch --screen

Run the RL algorithms and unpause the GAZEBO

roslaunch ur_training default.launch

Latest block's point:

rostopic echo /target_blocks_pose

Total block's points:

rostopic echo /blocks_poses 

How to launch REINFORCE algorithm

First launch the simulator

roslaunch ur_robotiq_gazebo conveyer_gym.launch controller:=vel --screen gui:=false

And load the parameters and launch python file for reset

roslaunch ur_reaching reinforcement.launch

And start the learning algorithm

python reinforcement_main.py 

And unpause physics of GAZEBO simulator

rosservice call /gazebo/unpause_physics "{}"

How to launch PPO+GAE algorithm

First launch the simulator including loading the parameters and GAZEBO Excution func

roslaunch ur_robotiq_gazebo conveyer_gym.launch --screen gui:=false

And start the learning algorithm

 python ppo_gae_main.py

How to use the RLkit

RLkit is reinforcement learning framework based on rllab

Run GAZEBO simulator and gazebo_execution

First launch the simulator including loading the parameters and GAZEBO Excution func

roslaunch ur_robotiq_gazebo conveyer_gym.launch --screen gui:=false

Training

And start the SAC learning algorithm based on RLkit

 python rlkit_sac_main.py

And unpause physics of GAZEBO simulator

rosservice call /gazebo/unpause_physics "{}"

After training, you may find the pickled files on the rlkit/data folder.

you can easily see the results through selecting the generated folder about training like follwing:

python viskit/frontend.py ../rlkit/data/SAC/SAC_2019_10_14_08_27_55_0000--s-0/

Evaluation

If you want to evaluate the trained weight, you can run like following:

python rlkit/scripts/run_policy.py rlkit/data/SAC/SAC_2019_10_14_08_27_55_0000--s-0/params.pkl 

Visualization

  • Rviz
roslaunch ur_robotiq_moveit_config  moveit_rviz.launch 
  • rviz_visual_tools
rosrun visual_tools rviz_visual_tools_demo

Visualization

  • moveit_visual_tools
rosrun visual_tools moveit_visual_tools_demo

Services

  • /gazebo/unpause_physics (Empty)
    • Unpause GAZEBO simulator for starting
    rosservice call /gazebo/unpause_physics "{}"
    
  • /set_velocity_controller (SetBool)
    • Set velocity controllers including 6 velocity controllers after stoppint velocity_controller/JointTrajectoryController
  • /set_trajectory_velocity_controller (SetBool)
    • Set velocity trajectory controller including Joint trajectory controllers after stopping velocity_controller/JointVelocityController
  • /stop_training (SetBool)
    • Stop training process
  • /start_training (SetBool)
    • Start training process

Helpful function

We can check the controller manager status using following:

rosrun rqt_controller_manager rqt_controller_manager

controller manager

Universal Robot

Usage with real Hardware
There are launch files available to bringup a real robot - either UR5 or UR10.
In the following the commands for the UR5 are given. For the UR10, simply replace the prefix accordingly.

Don't forget to source the correct setup shell files and use a new terminal for each command!

To bring up the real robot, run:

roslaunch ur_modern_driver ur5_bringup.launch robot_ip:=192.168.0.3

MoveIt! with real Hardware and gripper
To run the MoveIt with gripper, run

roslaunch total_moveit_config total_moveit_planning_execution.launch

MoveIt API Python execution!

rosrun ur_robotiq_moveit_api ur_robotiq_moveit.py

A simple test script that moves the robot to predefined positions can be executed like this:

rosrun ur_driver test_move.py

CAUTION:
Remember that you should always have your hands on the big red button in case there is something in the way or anything unexpected happens.

Usage with Gazebo Simulation
There are launch files available to bringup a simulated robot - either UR5 or UR10.
In the following the commands for the UR5 are given. For the UR10, simply replace the prefix accordingly.

Don't forget to source the correct setup shell files and use a new terminal for each command!

To bring up the simulated robot in Gazebo, run:

roslaunch ur_gazebo ur5.launch

MoveIt! with a simulated robot
Again, you can use MoveIt! to control the simulated robot.

For setting up the MoveIt! nodes to allow motion planning run:

roslaunch ur5_moveit_config ur5_moveit_planning_execution.launch sim:=true

For starting up RViz with a configuration including the MoveIt! Motion Planning plugin run:

roslaunch ur5_moveit_config moveit_rviz.launch config:=true

NOTE:
As MoveIt! seems to have difficulties with finding plans for the UR with full joint limits [-2pi, 2pi], there is a joint_limited version using joint limits restricted to [-pi,pi]. In order to use this joint limited version, simply use the launch file arguments 'limited', i.e.:

roslaunch ur_gazebo ur5.launch limited:=true

roslaunch ur5_moveit_config ur5_moveit_planning_execution.launch sim:=true limited:=true

roslaunch ur5_moveit_config moveit_rviz.launch config:=true


MoveIt! with real Hardware
Additionally, you can use MoveIt! to control the robot.
There exist MoveIt! configuration packages for both robots.

For setting up the MoveIt! nodes to allow motion planning run:

roslaunch ur5_moveit_config ur5_moveit_planning_execution.launch

For starting up RViz with a configuration including the MoveIt! Motion Planning plugin run:

roslaunch ur5_moveit_config moveit_rviz.launch config:=true

NOTE:
As MoveIt! seems to have difficulties with finding plans for the UR with full joint limits [-2pi, 2pi], there is a joint_limited version using joint limits restricted to [-pi,pi]. In order to use this joint limited version, simply use the launch file arguments 'limited', i.e.:

roslaunch ur_bringup ur5_bringup.launch limited:=true robot_ip:=IP_OF_THE_ROBOT [reverse_port:=REVERSE_PORT]

roslaunch ur5_moveit_config ur5_moveit_planning_execution.launch limited:=true

roslaunch ur5_moveit_config moveit_rviz.launch config:=true

ur-reaching-reinforcement-learning's People

Contributors

geonhee-lee avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

ur-reaching-reinforcement-learning's Issues

Problem with catkin make

Hello Geonhee,
i am pretty new to ROS so forgive me if i make novice questions.
I get the following error when i try to catkin make
Screenshot from 2022-02-16 03-11-19

I tried erasing the word include from the CmakeLists.txt specified above, as suggested by a stackoverflow user but then i get the following error
image

I also tried to completely remove ur_robotiq_moveit_api and visual_tools folders and i had a successful build. but then i get the following error during launch.
Screenshot from 2022-02-16 16-36-28

I would appreciate your help and suggestions.
Thank you

OS Ubuntu 18.04 ROS Melodic

cannot build the UR_kinematics

Hi, I'm building the project.
But when I type the command catkin build, it reports that:

/opt/ros/kinetic/include/moveit/robot_state/robot_state.h:86:83: error: no match                                                         ing function for call to ‘moveit::core::JumpThreshold::JumpThreshold()’
   explicit JumpThreshold(double jt_revolute, double jt_prismatic) : JumpThresho                                                         ld()

Could you please help me?

Ubuntu 16.04
Ros Kinetic
1584161026(1)

where is hopper_training.msg? Thank you

Path: /ur-reinforcement-learning/ur_openai_ros/ur_training/script/ur_training/default.py

Code:
#!/usr/bin/env python

'''
Original Training code made by Ricardo Tellez [email protected]
Moded by Miguel Angel Rodriguez [email protected]
Visit our website at www.theconstructsim.com
'''
import gym
import time
import numpy
import random
import time
from ur_training.algorithm.qlearn import *
###import our training environment
from ur_training.env.ur_default_env import URDefaultEnv
from gym import wrappers
from std_msgs.msg import Float64
###ROS packages required
import rospy
import rospkg
from hopper_training.msg import QLearnMatrix, QLearnElement, QLearnPoint

The environment configuration in ROS?

Hello,it is a great job! Thanks for your sharing.

I have a question about the environment configuration. In an answer, you say your environment configuration is using conda python above 3 on Ubuntu 16.04 LTS. I know the python is 2.7 in kinetic. I don't know how to install python3 in kinetic.

image

But in the other answer, you say your environment configuration is melodic.
image

I am looking forward to your reply!

First launch the simulator ,have error,and i dont solving this problem

when i start roslaunch ur_robotiq_gazebo gym.launch
there ia an error makes can not start the gazebo, i am so confused,would you help me,folling is the detail about error:
Invalid parameter "hw_interface" None None
when instantiating macro: ur_arm_transmission (/home/zhb/ur_ws/src/ur-reinforcement-learning/ur_description/urdf/ur.transmission.xacro)
instantiated from: ur5_robot (/home/zhb/ur_ws/src/ur-reinforcement-learning/ur_robotiq_description/urdf/ur5-vel.urdf.xacro)
in file: /home/zhb/ur_ws/src/ur-reinforcement-learning/ur_robotiq_description/urdf/ur_robotiq-vel.urdf.xacro
Invalid tag: Cannot load command parameter [robot_description]: command [/opt/ros/kinetic/lib/xacro/xacro --inorder '/home/zhb/ur_ws/src/ur-reinforcement-learning/ur_robotiq_description/urdf/ur_robotiq-vel.urdf.xacro'] returned with code [2].

Param xml is
The traceback for the exception was written to the log file

waiting for your help,grateful about you!

Build error

1. `Make Error at /opt/ros/melodic/share/catkin/cmake/catkinConfig.cmake:83 (find_package):
  Could not find a package configuration file provided by "moveit_core" with
  any of the following names:

    moveit_coreConfig.cmake
    moveit_core-config.cmake

  Add the installation prefix of "moveit_core" to CMAKE_PREFIX_PATH or set
  "moveit_core_DIR" to a directory containing one of the above files.  If
  "moveit_core" provides a separate development package or SDK, be sure it
  has been installed.
Call Stack (most recent call first):
  ur-reaching-reinforcement-learning/ur_kinematics/CMakeLists.txt:6 (find_package)

Getting error in a python file

Hello Everyone
Getting this error while launching "roslaunch ur_training default.launch"


Executed ur_training/init.py
/usr/lib/python3/dist-packages/apport/report.py:13: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import fnmatch, glob, traceback, errno, sys, atexit, locale, imp, stat
Traceback (most recent call last):
File "/home/dimaag-nav/suhas_home/ur_reach_rl/src/ur_openai_ros/ur_training/script/ur_training/default.py", line 13, in
from ur_training.algorithm.qlearn import *
File "/home/dimaag-nav/suhas_home/ur_reach_rl/src/ur_openai_ros/ur_training/script/ur_training/algorithm/init.py", line 1
print " Executing ur_training/algorithm/init.py"
^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(" Executing ur_training/algorithm/init.py")?

Can anyone give any solution for the same

cannot launch gazebo

Hi, something bad happens when I launch your code.
I really need your help.
Thank you very much.

1584193171(1)

cannot work

Hi, I met some problems, which have troubled me for several days, so I have to ask you for advice. Forgive me for bothering you.
Now I'm using ROS-kinetic, python3.5. The biggest problem is that this robotic arm doesn't move, in any scenario.
The following figure is the result of running roslaunch ur_robotiq_gazebo gym.launch and roslaunch ur_training default.launch.
屏幕截图 2023-09-15 203749

The following graph is the result of running roslaunch ur_robotiq_gazebo conveyer_gym.launch --screen and roslaunch ur_training default.launch.Where controller=pos.
image

This is the result of running roslaunch ur_robotiq_gazebo conveyer_gym.launch --screen and roslaunch ur_training default.launch. Where controller=vel.
image (1)

The problem is that this robotic arm is static, so it doesn't move anyway. I hope you can find time to check it. If there is something I can't describe clearly, please contact me. I will reply you as soon as possible. Looking forward to hearing from you.

GAZEBO Controller문제

switch_ctrl branch

GAZEBO 시뮬레이터에서 controller를 사용하기 위해서, yaml파일(ur_robotiq_gazebo/controller/arm_controller.yaml)에 controller의 description을 적어 launch파일에 load했다.
yaml 파일에 기술한 첫번째 컨트롤러인 vel_traj_controller(velocity_controllers/JointTrajectoryController)는 trajectory_msgs/JointTrajectory type을 가지며, 이 message type은 UR5를 제어할 수 있는 ur_modern_driver package에서 /joint_speed topic으로 속도 제어가 가능하다. 뿐만아니라, MoveIt을 사용할 수 있는 controller이다.
그러나, 이 컨트롤러의 문제는 vel_traj_controller/command를 이용하여 로봇을 제어할 수 없다는 단점이 있다. follow_joint_trajectory의 형태로 제어가 가능(MoveIt, trajectory control)하므로, 강화학습에 적용하기에는 한계가 있다.
반대로 각 조인트를 GAZEBO환경에서 제어하기 위해서 사용하는 두 번째 방법은 velocity_controllers/JointVelocityController type의 각 조인트마다 6개 controller를 정의하여 controller_name/command의 토픽으로 std_msgs/Float type으로 속도값을 보내 제어할 수 있다.

문제는 이 두가지 컨트롤러를 동시에 사용하려고 하기때문에 발생한다.
controller_manager를 이용하여 동시에 launch 파일에서 spawn하면 다음과 같은 문제가 발생한다.

Loaded controller_name 이후에,

[Warn] Resource conflict on [joint_name]. Controllers = [controller_name1, controller_name2,]
[Error] Could not switch controllers, due to resource conflict
Error when starting ['controller_name']

하나의 controller_manager로 controller를 spawn하게되면 switch controller가 불가능하고, joint_publisher.py 혹은 joint_traj_publisher.py에서 다음과 같은 오류가 발생한다.

self.joint_publisher.get_num_connections() == 0

controller가 spawn되지 않아 connection이 안되어 publisher가 불가능하다.

The document support

Hi, @Geonhee-LEE.

It is a great job, I believe it can complete the tracking after a lot of training. So I want to learn the algorithm, can you give me some blog/document about the algorithm, thanks a lot.

cannot work

I run the reinforcement_main.py in two branch.
First in master branch,
However, it errors when running ur_reinforcement_main.py following tutorial.
It seems that I cannot change the controller type into 'vel'.
It's always 'pos'.
图片

Second, in devel branch:
The 'controller type' is 'vel', but it cannot also work.
The error is :
图片

Ubuntu 16.04
Ros-kinetic
Anaconda
Python2.7
tensorflow=1.13.1

Could you please help me?
Thank you very much.

Model falling apart

Hello, this model runs in pieces, what is the reason? Looking forward to your reply.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.