clvrai / furniture Goto Github PK
View Code? Open in Web Editor NEWIKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks
Home Page: https://clvrai.com/furniture
License: MIT License
IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks
Home Page: https://clvrai.com/furniture
License: MIT License
Hi,
thanks for your efforts on building IKEA Furniture Env. It's a great platform for IL/RL.
I'm wondering which branch is stable? and which branch should I use if I want to use GAIL/BC on two different single-arm robots?
thanks!
Hello,
I am trying to change the pose of the camera during simulation.
There are already implemented functions in furniture
class: _set_camera_position
for mujoco renderer and _set_camera_pose
for unity renderer. When I tried to use _set_camera_pose
, it did not change the camera pose. But _set_camera_position
works using the mujoco renderer.
I used following code. Am I missing something for the unity case?
import gym
import furniture
import numpy as np
from furniture import agent_names, background_names, furniture_names
import matplotlib.pyplot as plt
env_name = "IKEA{}-v0".format(agent_names[0])
furniture_name= furniture_names[0]
background_name = background_names[2]
unity = False
camera_id = 0
new_pose = np.array([1,1,1, 1,0,0,0])
env = gym.make(env_name, furniture_name=furniture_name, background=background_name,
unity=unity, camera_ids = [camera_id])
env.reset()
old_rgb = env.render("rgb_array")[0]
if unity:
env.env._set_camera_pose(camera_id, new_pose)
else:
env.env._set_camera_position(camera_id, new_pose[:3])
env.env.sim.model.cam_quat[camera_id] = new_pose[3:]
env.env.sim.forward()
env.env.sim.step()
new_rgb = env.render("rgb_array")[0]
fig = plt.figure()
fig.add_subplot(2, 1, 1)
plt.imshow(old_rgb)
fig.add_subplot(2, 1, 2)
plt.imshow(new_rgb)
plt.show()
Hello,
I want to add a new furniture model. I have read docs and examined several utility files in the environment. However, I couldn't find answers to the questions below.
As I understood from docs, I need a .stl file for each part and one XML file. I decided to use Blender to split my own object into several stl parts, however I don't know whether I should write XML manually or not.
I have some questions related to XML creation.
<body name="1_chair_leg_left" pos="-0.115 0.0 0.1727" quat="1 0 0 0">
<geom density="100" material="light-wood" mesh="1_chair_leg_left" name="1_chair_leg_left_geom" pos="0 0 0" rgba="0.82 0.71 0.55 1" solref="0.001 1" type="mesh" />
<site name="leg_left-seat,conn_site1" pos="0.0 0.0 0.1401" quat="0.707 0 -0.707 0" rgba="1 0 1 0.3" size="0.0057" />
<site name="1_chair_leg_left_corner_site1" pos="-0.019 0.1305 -0.144" rgba="0 0 1 0.3" size="0.0057" />
<site name="1_chair_leg_left_corner_site2" pos="-0.019 -0.129 -0.144" rgba="0 0 1 0.3" size="0.0057" />
<site name="1_chair_leg_left_corner_site3" pos="0.0307 0.0230 0.1382" rgba="0 0 1 0.3" size="0.0057" />
<site name="1_chair_leg_left_corner_site4" pos="0.0307 -0.022 0.1382" rgba="0 0 1 0.3" size="0.0057" />
<site name="1_chair_leg_left_corner_site5" pos="-0.019 0.0230 0.1382" rgba="0 0 1 0.3" size="0.0057" />
<site name="1_chair_leg_left_corner_site6" pos="-0.019 -0.022 0.1382" rgba="0 0 1 0.3" size="0.0057" />
<site name="1_chair_leg_left_bottom_site" pos="-0.019 0.0 -0.134" rgba="0 0 1 0.3" size="0.0057" />
<site name="1_chair_leg_left_top_site" pos="0.0 0.0 0.1382" rgba="0 0 1 0.3" size="0.0057" />
<site name="1_chair_leg_left_horizontal_radius_site" pos="-0.019 0.0 0.0" rgba="0 0 1 0.3" size="0.0057" />
</body>
How do you decide the numbers of site's (corner_site1,corner_site2 ...) for each body?
If there is a script or tool to form XML files, I will be very happy if you can share with us.
There are some files in furniture/blob/dev/env/xml_adjusting/
directory.
I didn't figure out how to use them correctly.Because in addition to position and other details,
rescaling is also important and hard to do with hand.
Thank you for your time.
Hello, thank you very much for this platform, which has provided me with a lot of help. I now want to conduct experiments using the SAC algorithm on furniture with the furniture_name "block". However, it seems that this furniture does not have recipes. How can I obtain it, or can I set the recipes myself? But how should I set some distances? If you can reply to me, I will be very grateful.
Hi,
First, thank you for sharing this great project.
I have some questions related to training and testing phases.
As stated in overview.md
I have used the command below to train:
python3 -m rl.main --env FurnitureBaxterBlockEnv --prefix demo --reward_scale 3 --wandb True
Then I've noticed that this doesn't work with unity and giving:
[Errno 111] Connection refused
now connecting to 1050
message all the time.
So I've added --unity False
to work.( I am not sure, this is the correct way. Is it possible to see simulator while training? ) After configuring wandb account the training process started.
Assuming the training process has been completed (It shows 2000000 iterations ) I didn't understand how to test the trained model. (Edit: I have also waited for 300k / 2000000 iteration on CPU but baxter still cannot perform "hold" phase in task.)
Also, there are some checkpoint functions to save model periodically. But, when I retype the command again, does the trainer starts from beginning or starts from where it was interrupted? If both possible, how do we set this?
I will be very happy that if you can clarify what are the correct commands to train and visually test the special task for example one in furniture_cursor_toytable.py.
Thank you for your time
Best Regards
Hello, it is very cool to simulate furniture assembly. I tried Human control part and it work well. But I can't run RL benchmark because I can't find learning codes in the repository. I am curious if you are still working and improving this project. Thank you!
I found that I can't enter the FurnitureBench website,
it shows that the website does not exist.
Could you tell me what the problem is.
Hi,
Thanks for your sharing.
_step() -> _step_continuous() -> _setup_action() -> _do_simulation()
The robot senses the endpoint motion $x(t)$ and commands joint torques and forces to create $-f_{ext}$, the force to display to the user. Such a robot is called impedance controlled, as it implements a transfer function $Z(s)$ from motions to forces.
For torque:
The only difference between torque and impedance is that the torque does not use _setup_action function (used to setup gripper part, rescale and set gravity compensation). But without setting gripper part, it cause an error "could not broadcast input array from shape...". After setup gripper part, they are almost the same. Is this control method not implemented completely yet?
For NEW_CONTROLLERS:
It seems they do not work for robots, not implemented completely yet?
For experiments:
IK controller is used for all experiments, or also with other controllers?
The bullet description of Fetch is missing.
There is no Fetch files in /models/assets/pybullet_data/fetch_description
.
Thanks for any reply.
Best regards.
Hello,
First of all, thanks for the great work.
I would like to know how to add the UR5 robot to the environment. From what I could gather in #10, I only need the MJCF XML, pybullet URDF and ik_controller files. However, no existing implementation seems to have all of these elements. I would like to use the referred https://github.com/clvrai/furniture/blob/dev/furniture-unity/Assets/Scripts/MJImport.cs script to solve the issue, but cannot understand how to run it using the provided unity binary (in google drive).
Hello,
I'm having trouble with installing - (4) Benchmarking.
When i run the command,
python -m run --algo bc --run_prefix bc_table_lack_0825 --env IKEASawyerDense-v0 --furniture_name table_lack_0825 --demo_path demos/Sawyer_table_lack_0825
No module named 'run' error occured.
So i changed to
python -m furniture.run --algo bc --run_prefix bc_table_lack_0825 --env IKEASawyerDense-v0 --furniture_name table_lack_0825 --demo_path demos/Sawyer_table_lack_0825
The error about directory is cleared.
In the run code,
from method.main import run
from env import *
from config import create_parser
from method.config import add_method_arguments
Then, another errors occured.
In the furniture folder, there is no folder named 'method'.
But in the script 'run.py', the code imports 'method' module.
ModuleNotFoundError: No module named 'method'
So i don't no how to run the command.
Hi,
Thanks for the great work.
I'm trying to run the dev-branch code.
I check the env/init.py
Is it a environment registry issue?
But can't figure it out:
haoyux@haoyux-ThinkPad:~/furniture-dev$ python3 run.py --run_prefix test --algo ppo --env "furniture-sawyer-v0" --unity False --wandb False
pybullet build time: Jul 11 2020 19:26:20
[2020-07-23 15:14:40,043] Run a base worker.
[2020-07-23 15:14:40,043] Create log directory: log/furniture-sawyer-v0.ppo.test.123
[2020-07-23 15:14:40,045] Create video directory: log/furniture-sawyer-v0.ppo.test.123/video
[2020-07-23 15:14:40,045] Create demo directory: log/furniture-sawyer-v0.ppo.test.123/demo
[2020-07-23 15:14:40,060] Store parameters in log/furniture-sawyer-v0.ppo.test.123/params.json
[2020-07-23 15:14:40,062] Unknown environment name: furniture-sawyer-v0
Available environments:
[2020-07-23 15:14:40,062] Instead, query gym environments
Traceback (most recent call last):
File "run.py", line 18, in <module>
run(parser)
File "/home/haoyux/furniture-dev/method/main.py", line 75, in run
trainer = Trainer(config)
File "/home/haoyux/furniture-dev/method/trainer.py", line 44, in init
self._env = make_env(config.env, config)
File "/home/haoyux/furniture-dev/method/environments/init.py", line 48, in make_env
return get_gym_env(name, config)
File "/home/haoyux/furniture-dev/method/environments/init.py", line 84, in get_gym_env
frame_skip=config.action_repeat,
File "/home/haoyux/furniture-dev/method/utils/gym_env.py", line 68, in init
self.max_episode_steps = self.env._max_episode_steps // frame_skip
AttributeError: 'FurnitureGym' object has no attribute '_max_episode_steps'
Thanks!
Hi,
Thanks for sharing this wonderful project!
I git cloned the latest dev-branch.
When I run:
haoyux@haoyux-ThinkPad:~/furniture-dev$ python3 run.py --run_prefix test --unity false --env furniture-sawyer-v0 --wandb True
it stuck at 1001 step when evaluation:
[
furniture-sawyer-v0.sac.test.123: 0%| | 1000/1000000 [00:51<15:04:31, 18.41it/[2020-07-24 23:36:01,575] Evaluate at 1
[2020-07-24 23:36:01,575] Run 1 evaluations at step=1001
[2020-07-24 23:36:01,575] Evalute run 1
furniture-sawyer-v0.sac.test.123: 0%| | 1001/1000000 01:10<15:04:31, 18.41it/wandb: Program ended successfully.
Killed
I'm wondering why this happened.
And when I disable the evaluation, it won't be stuck.
Thanks!
Hi, I am very interested in using this environment for my research.
I have the following questions:
Hi,
When I tried to use camera observation with: python3 -m rl.main --env FurnitureBaxterBlockEnv --prefix demo --wandb False --reward_scale 3 --debug True --notes True --save_rollout True --unity False --is_train True --visual_ob True
command it gives this error:
RuntimeError: invalid argument 0: Tensors must have same number of dimensions: got 3 and 1 at /pytorch/aten/src/TH/generic/THTensor.cpp:603
How can I use camera observation while training?
Thanks in advance
Hi, I am unable to manipulate the robot using any of the key instructions when I run demo_manual. The Unity app seems to render fine, so I am not sure what the issue might be. I am using MacOS Big Sur 11.4. Would appreciate any guidance!
Hello, thank you very much for providing this information. Based on my observations from the demo, the depth images obtained are three-channel images, but converting to a point cloud typically requires a single-channel depth image. Could you please offer some suggestions on converting these images into a single-channel depth image? I would be extremely grateful.
Hi, this project is very cool for RL, because tasks in this project are different from Mojuco. I am curious if you are still working and improving this project.
Hi,
I have followed the set up instructions and when trying to run:
python3 -m demo_manual --unity true
I get the following error:
AssertionError: Cannot find unity app None
Do you know what is causing this? Any guidance on how to have this fixed and be able to run the demo would be appreciated.
Thank you.
when I run the demo_viosion , I get the video named "onservation.mp4" . However, Displays only a scrolling line.
furniture-unity folder with the unity app project seems to be missing, is it possible to include it?
I also have a question about the unity app -- is it possible to change unity window size from config options? Or need to rebuild unity project for this?
Thanks for the great work.
I would like to know if we can obtain the observations from different views? Like, set two or more camera around the furniture..
Thank you very much for your help. This article has been very helpful to me. I would like to ask you, when I run 'python -m run --algo sac --run_prefix sac_table_lack_0825 --env IKEASawyerDense-v0 --furniture_name table_dockstra_0279', I encounter the error 'no module named 'method''. This error occurs in the 'run.py' file with the following code snippet:
from method.main import run
import env
from config import create_parser
from method.config import add_method_arguments
Where can I find this 'method'? I would greatly appreciate it if you could reply to me.
Hello, I would like to use the existing Panda model to generate assembly task demos much like with Sawyer.
The feature does not seem to exist natively, so I tried hacking it into IKEA by copying the furniture_sawyer_dense.py
and furniture_sawyer_gen.py
files and renaming all instances of sawyer to panda, but sure enough this doesn't work. The culprit seems to be the fact that the gripper for sawyer has and additional degree of freedom for grip_tip
, hence the following error:
Traceback (most recent call last):
File "~/anaconda3/envs/base/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "~/anaconda3/envs/base/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "~/skill-chaining/furniture/furniture/env/furniture_panda_gen.py", line 748, in <module>
main()
File "~/skill-chaining/furniture/furniture/env/furniture_panda_gen.py", line 743, in main
env = FurniturePandaGenEnv(config)
File "~/skill-chaining/furniture/furniture/env/furniture_panda_gen.py", line 105, in __init__
self.reset()
File "~/skill-chaining/furniture/furniture/env/furniture.py", line 326, in reset
self._reset(furniture_id=furniture_id, background=background)
File "~/skill-chaining/furniture/furniture/env/furniture_panda_dense.py", line 220, in _reset
self._reset_reward_variables()
File "~/skill-chaining/furniture/furniture/env/furniture_panda_dense.py", line 139, in _reset_reward_variables
self._update_reward_variables()
File "~/skill-chaining/furniture/furniture/env/furniture_panda_dense.py", line 177, in _update_reward_variables
eef_pos = self._get_pos("griptip_site")
File "~/skill-chaining/furniture/furniture/env/furniture.py", line 3131, in _get_pos
raise ValueError
ValueError
As I am not too familiar with how these models are defined, I would like to ask for some help in making this change work.
Hi,
First of all freat work on setting up this benchmark.
I had a question on using this benchmark to standardise fine-motion strategies for contact-rich manipulation. Correct me if I am wrong, but as it stands now, the benchmark assumes an automagic connection between parts based on some pre-defined distances. Is there any way to modify the environment and actually use the contact dynamics to ask robotics questions around insertion and connection strategies?
Hello sir and thanks for the good work on this repo!
I am having trouble with getting TSTAR to work past 1 subtask on any of the assembly tasks (using Sawyer, for the record).
I have seen clvrai/skill-chaining#3, which seems to be related to this problem and suggests the issue may be due to the evaluation function, but this has not been answered in a while so it could be a dead end.
I am thinking there might be a chance bad demos are somewhat at play, which is why I would like to know if unity can render the generated demo files (.pkl)
Thanks for your time
Hi!
This is an amazing effort, and I’m excited to try using this platform in my upcoming work; however, I can’t find the method
directory with the BC agent implementation.
Furthermore, are there results (success rate, reward) for a BC agent trained from visual observations given a fixed number of demonstrations (e.g., for baselines/comparison)?
Hi,
Thanks for your sharing.
I have 3 questions:
The dense reward function needs to get recipes (seems to be simplified ground truth) to calculate reward, does this means that compared with other methods of calculating reward (through images, electric clouds, poses, etc.), it has more limitations? It seems that only furniture with the recipe can use dense reward function?
The recipe fixes how each part is assembled and does not take into account the fact that the same blocks can be replaced. Will calculating the reward cause an error?
# * File: Furniture_sawyer_dense.py
# * Class: FurnitureSawyerDenseRewardEnv
def _reset_reward_variables(self):
self._subtask_step = len(self._preassembled)
self._used_sites = set()
for part_idx in range(len(self._preassembled)):
leg = self._recipe["recipe"][part_idx][0]
for i in range(len(self._recipe["recipe"])):
g_l, g_r = f"{leg}_ltgt_site{i}", f"{leg}_rtgt_site{i}"
if not (g_l in self._used_sites or g_r in self._used_sites):
self._used_sites.add(g_l)
self._used_sites.add(g_r)
break
def _update_reward_variables(self):
...
for i in range(len(self._recipe["recipe"])):
g_l, g_r = f"{self._leg}_ltgt_site{i}", f"{self._leg}_rtgt_site{i}"
if g_l not in self._used_sites and g_r not in self._used_sites:
self._used_sites.add(g_l)
self._used_sites.add(g_r)
break
# * File: furniture_sawyer_dense.py
# * Class: Furniture
def _set_next_subtask(self) -> bool:
""" Returns True if we are done with all attaching steps. """
self._subtask_step += 1
# * Do not check whether all parts are assembled correctly???
if self._subtask_step == self._success_num_conn:
return True
self._update_reward_variables()
return False
# * File: furniture.py
# * Class: FurnitureEnv
def _reset(self, furniture_id=None, background=None):
...
if self._num_connects is not None:
self._success_num_conn = self._num_connects
self._success_num_conn += len(self._preassembled)
else:
self._success_num_conn = len(self._object_names) - 1 # * piece num - 1
Thanks for any reply.
Best regards.
Hello,
i would like to know how to extract segmentation masks for objects in the scene. i did not find functions that explicitly do this. So i tried to get the mask from segmentation images. However, i found that there is always one part missing in the segmentations. i used Sawyer.
So, what i am asking is :
thank you very much for the excellent work.
Thanks for your sharing.
Can I use the baxter to generate the demo?
Hi,
First, thank you for sharing this great project.
I have some questions related to python demo_manual.py
.
I follow the installation README here step by step (both Installation
and virtual-display-on-headless-machines
are executed as README). After that, I ran
(furniture) username@cdc-26:~/Program/furniture$ python -m demo_manual --virtual_display :1
The output is
pygame 1.9.6
Hello from the pygame community. https://www.pygame.org/contribute.html
IKEA Furniture Assembly Environment!
Supported robots:
0: Baxter
1: Sawyer
2: Cursor
Choose an agent (enter a number from 0 to 2): 0
Supported furniture:
0: block
1: chair_agne_0007
2: chair_bernhard_0146
3: chair_bertil_0148
4: desk_mikael_1064
5: shelf_ivar_0678
6: shelf_liden_0922
7: swivel_chair_0700
8: table_klubbo_0743
9: table_lack_0825
10: tvunit_0406
Choose a furniture model (enter a number from 0 to 10): 0
Supported backgrounds:
0: Simple
1: Industrial
2: Lab
3: Garage
4: Ambient
5: NightTime
6: Interior
Choose a background (enter a number from 0 to 6): 0
Creating environment (robot: FurnitureBaxterEnv, furniture: block, background: Simple)
The true file name is Furniture
This is the launch string /home/username/Program/furniture/binary/Furniture.x86_64
Unity remote connecting to 1050
now connecting to 1050
[Errno 111] Connection refused
now connecting to 1050
[Errno 111] Connection refused
now connecting to 1050
[Errno 111] Connection refused
now connecting to 1050
[Errno 111] Connection refused
Do you have any suggestion about the [Errno 111] Connection refused
? It seems that the MuJoCo-Unity binary are download and extracted to furniture/binary
folder correctly. I've tried it on two servers and got the same error.
Thank you.
Hello, this article has been of great help to me. I would like to ask how can I obtain the camera's intrinsic parameters
Hi,
I followed the installation instruction installation.md. When I run demo_manual, I got this error:
[2022-03-24 10:32:52,319] Unity remote connecting to 1050
[2022-03-24 10:32:52,319] now connecting to 1050
[2022-03-24 10:32:52,319] [Errno 111] Connection refused
/bin/sh: 1: /home/usr_name/furniture/binary/Furniture.x86_64: Permission denied
[2022-03-24 10:32:53,320] now connecting to 1050
[2022-03-24 10:32:53,321] [Errno 111] Connection refused
[2022-03-24 10:32:54,322] now connecting to 1050
Any idea to fix it?
Hello,
Thank you for your amazing work.
After reading the related paper and looking at the repo, I have two questions about adding new robots:
If I want to add a new robot to use within the environment for manual demonstration and reinforcement learning, are implementing mjcf and related environment python codes enough or should I also need to implement ik controllers using pybullet and include urdfs.
If pybullet and urdfs are needed, for what purposes these ik controllers and urdfs are used?
I am building an environment with robot=cursor and object=blocks. I notice in _load_model_robot, rgba is set 0 0 0 0. I am not sure if this is what this sentence "So, we use the meshes for just rendering, and add invisible MuJoCo primitive geometries to create the colliders." means from furniture_details.md. I still don't quite understand why doing this. I also tried to disable the "hiding agent" and didn't see any difference in the viewer.
Thanks for your great work!
Following the question on setting multiple cameras, once I set different cameras, how can I obtain the real position of furnitures given the depth images. Is there anyway I can find the camera parameters in the configuration files?
I also found the following code in demo_vision.py,
for i, body in enumerate(env._object_names):
pos = qpos[body][:3]
It seems I can get the positions in world coordinate, how can I get the positions in different camera coordinates? And the number of pos output (10 pos ) seems larger the number of furnitures (8 parts).
Thanks a bunch!
Hi,
thanks for sharing this great project!
I git clone the latest dev-branch, trying to run sac with visual obs:
haoyux@haoyux-ThinkPad:~/furniture-dev$ python3 run.py --run_prefix test --unity false --env furniture-sawyer-v0 --wandb True --object_ob False --object_ob_all false --robot_ob false --visual_ob True --wandb True
However, after 1000 steps, it will be automatically killed always.
[2020-07-24 23:24:07,340] Randomly initialize models [2020-07-24 23:24:07,340] Start training at step=0 furniture-sawyer-v0.sac.test.123: 0%| | 1000/1000000 [02:09<74:51:12, 3.71it/wandb: Program ended successfully. Killed haoyux@haoyux-ThinkPad:~/furniture-dev$ wandb: Run summary:
I'm wondering why this happened? Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.