Comments (15)
xuanlinli17
commented on August 23, 2024
Hi xtli12,
I've updated the docker files and submission examples for ManiSkill2-Learn. You can view it at https://github.com/haosulab/ManiSkill2-Learn/tree/main/submission_example .
Pointnet++ is not compiled
No module named 'maniskill2_learn.networks.modules.pn2_modules'
This only indicates that you haven't compiled our custom implementation of pointnet++. Programs can still run successfully without PointNet++.
Fail to load evaluation configuration. (<class 'TypeError'>, TypeError("'NoneType' object cannot be interpreted as an integer"))
If you follow my updated ManiSkill2-Learn submission example, then it shouldn't occur anymore.
ERROR: "docker buildx build" requires exactly 1 argument.
The command should be
docker build -f {path_to_docker_file} {build_context_path} -t maniskill2023-submission
Your command is missing the build_context_path. The build_context_path is the local path where docker instructions, e.g., COPY, are referenced with respect to. See here for more details.
For example,
docker build -f "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/Docker" . -t maniskill2023-submission
Note the . in the above command.
from maniskill.
xtli12
commented on August 23, 2024
Thank you for your immediate reply!
I've changed the config.file_path and model_path in the updated user_solution.py as follows:
cfg = Config.fromfile("/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/configs/mfrl/ppo/maniskill2_pn.py")
model_path = f'/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/model_10000000.ckpt'
And moved the user_solution.py directly under the ManiSkill2-Learn-main/, but the error still exists when I run:
export PYTHONPATH=/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main:$PYTHONPATH
ENV_ID="PickCube-v0" OUTPUT_DIR="tmp"
python -m mani_skill2.evaluation.run_evaluation -e ${ENV_ID} -o ${OUTPUT_DIR}
the error is
Fail to load evaluation configuration. (<class 'TypeError'>, TypeError("'NoneType' object cannot be interpreted as an integer"))
from maniskill.
xuanlinli17
commented on August 23, 2024
python -m mani_skill2.evaluation.run_evaluation searches the user_solution.py under the root ManiSkill2-Learn-main directory (instead of user_solution.py under submission_example). Do you have the correct path for user_solution.py (directly under root)?
Also are you running these commands in docker or outside of docker?
from maniskill.
xtli12
commented on August 23, 2024
I was running these commands outside of docker as the guideline https://github.com/haosulab/ManiSkill2/wiki/Participation-Guidelines says.
First, I changed the config.file_path and model_path in user_solution.py as follows:
#original
cfg = Config.fromfile("/root/ManiSkill2-Learn/configs/mfrl/ppo/maniskill2_pn.py") # Change this
model_path = f'/root/ManiSkill2-Learn/maniskill2_learn_pretrained_models_videos/{env_id}/dapg_pointcloud/model_25000000.ckpt' # Change this
changed to the absolute path in my machine:
cfg = Config.fromfile("/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/configs/mfrl/ppo/maniskill2_pn.py") # Change this
model_path = f'/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/model_10000000.ckpt' # Change this
Then I moved the user_solution.py to the directory ManiSkill2-Learn-main/ like this:
Then I followed the guideline https://github.com/haosulab/ManiSkill2/wiki/Participation-Guidelines:
export PYTHONPATH=/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main:$PYTHONPATH
ENV_ID="PickCube-v0" OUTPUT_DIR="tmp"
python -m mani_skill2.evaluation.run_evaluation -e ${ENV_ID} -o ${OUTPUT_DIR}
But the error still exists :
Fail to load evaluation configuration. (<class 'TypeError'>, TypeError("'NoneType' object cannot be interpreted as an integer"))
from maniskill.
xuanlinli17
commented on August 23, 2024
Oh, I believe you need to refer to the new participation guideline at https://haosulab.github.io/ManiSkill2/benchmark/submission.html . Our github wiki page is obsolete. All of our documentations are at https://haosulab.github.io/ManiSkill2. Sorry about the confusion.
from maniskill.
xtli12
commented on August 23, 2024
Thank you for your reply. With your kind help, I was able to locally verify my solution !
But when I submitted the docker image to the leaderboard, my score was 0. The weird thing is that the model I employed is the official .ckpt like:
/ManiSkill2-Learn-main/maniskill2_learn_pretrained_models_videos/PickCube-v0/dapg_pointcloud_vf1/model_25000000.ckpt
and when I tested the docker image locally, the score was not 0, and the average_metrics.json file is:
{"elapsed_steps": 32.0, "is_obj_placed": 1.0, "is_robot_static": 1.0, "success": 1.0}
But the result on leaderboard is:
How could I solve it?
from maniskill.
xuanlinli17
commented on August 23, 2024
Did you test the success rate locally using your submission docker?
from maniskill.
xtli12
commented on August 23, 2024
Yes, I did.
from maniskill.
StoneT2000
commented on August 23, 2024
@xtli12 Thanks for raising the issue. I will take a look into this issue, there may be bug in the evaluation system
from maniskill.
StoneT2000
commented on August 23, 2024
@xtli12 we have discovered the bug. The evaluation system has been updated now. Moreover, make sure to use absolute paths in your submission
In your code I see you used some relative paths like ./.... instead of /. Note that all code is executed from the /root/ directory, some of your code is in your own configured directory in /. I recommend using the absolute directory when possible.
Another note, make sure to not select all environment tasks to submit your image for (unless that is intended). It will slow down your evaluation, just pick the ones your code is designed for (we will also update the UI to make sure the default is not all tasks).
from maniskill.
xtli12
commented on August 23, 2024
Thanks to your kind assistance, I am now able to have a score on the leaderboard!
from maniskill.
StoneT2000
commented on August 23, 2024
@xtli12 actually we also fixed another relevant bug on the submission system. Some submissions weren't running on some tasks and was reporting some error about a pod. This has been fixed so your new submissions should evaluate on all tasks selected and not randomly fail.
from maniskill.
xtli12
commented on August 23, 2024
@StoneT2000
Hi, thanks for your kind assistance! However, there is still one issue that I can't figure out: When I got a .ckpt file after I trained locally, the success rate of it could be evaluated at nearly 0.9 (as the statistics.csv of eval_videos show):
length,reward,finish
16.00,27.09,1.00
30.00,55.04,1.00
27.00,46.54,1.00
21.00,39.88,1.00
18.00,25.68,1.00
32.00,70.89,1.00
200.00,54.32,0.00
200.00,142.26,0.00
19.00,33.22,1.00
21.00,38.99,1.00
25.00,37.49,1.00
25.00,41.71,1.00
28.00,47.78,1.00
22.00,36.20,1.00
36.00,62.98,1.00
22.00,42.51,1.00
26.00,47.45,1.00
20.00,38.91,1.00
16.00,27.86,1.00
15.00,15.59,1.00
24.00,38.76,1.00
34.00,56.16,1.00
23.00,39.97,1.00
16.00,24.94,1.00
28.00,55.13,1.00
18.00,28.16,1.00
29.00,26.65,1.00
24.00,41.36,1.00
22.00,33.08,1.00
17.00,26.10,1.00
32.00,52.22,1.00
23.00,47.61,1.00
22.00,28.18,1.00
20.00,30.57,1.00
14.00,26.25,1.00
36.00,54.62,1.00
21.00,39.93,1.00
25.00,45.10,1.00
27.00,41.78,1.00
36.00,65.30,1.00
24.00,40.19,1.00
35.00,53.46,1.00
25.00,46.08,1.00
38.00,64.58,1.00
27.00,48.21,1.00
31.00,49.23,1.00
35.00,57.30,1.00
33.00,55.16,1.00
30.00,57.10,1.00
29.00,51.07,1.00
22.00,39.08,1.00
17.00,32.24,1.00
29.00,53.18,1.00
21.00,31.36,1.00
31.00,51.91,1.00
30.00,54.70,1.00
17.00,30.73,1.00
24.00,42.72,1.00
16.00,27.93,1.00
25.00,46.75,1.00
21.00,36.03,1.00
20.00,38.31,1.00
26.00,50.52,1.00
34.00,71.51,1.00
18.00,34.77,1.00
21.00,32.74,1.00
26.00,24.26,1.00
20.00,31.10,1.00
38.00,52.84,1.00
64.00,141.20,1.00
22.00,37.52,1.00
21.00,39.26,1.00
33.00,61.53,1.00
26.00,46.36,1.00
17.00,30.52,1.00
31.00,58.11,1.00
34.00,70.25,1.00
25.00,39.19,1.00
28.00,48.81,1.00
19.00,36.75,1.00
24.00,41.37,1.00
24.00,41.12,1.00
33.00,46.83,1.00
28.00,35.95,1.00
35.00,55.76,1.00
17.00,30.58,1.00
35.00,59.21,1.00
19.00,35.51,1.00
23.00,40.27,1.00
26.00,49.69,1.00
22.00,39.70,1.00
30.00,63.27,1.00
24.00,44.66,1.00
22.00,40.48,1.00
24.00,36.81,1.00
15.00,22.36,1.00
16.00,27.23,1.00
19.00,32.93,1.00
18.00,32.40,1.00
24.00,40.25,1.00
19.00,31.77,1.00
23.00,41.42,1.00
30.00,28.32,1.00
22.00,37.12,1.00
19.00,29.72,1.00
31.00,47.59,1.00
24.00,42.54,1.00
28.00,48.80,1.00
24.00,40.05,1.00
18.00,33.08,1.00
20.00,29.66,1.00
15.00,26.05,1.00
14.00,26.88,1.00
39.00,84.47,1.00
29.00,51.09,1.00
29.00,49.55,1.00
200.00,73.68,0.00
27.00,45.85,1.00
14.00,24.01,1.00
32.00,49.84,1.00
15.00,22.88,1.00
12.00,18.75,1.00
19.00,33.10,1.00
15.00,26.14,1.00
20.00,25.97,1.00
14.00,22.45,1.00
200.00,79.64,0.00
32.00,52.45,1.00
21.00,23.57,1.00
26.00,42.50,1.00
28.00,50.71,1.00
35.00,58.86,1.00
31.00,41.77,1.00
14.00,24.70,1.00
21.00,35.15,1.00
22.00,42.51,1.00
33.00,46.44,1.00
14.00,23.89,1.00
26.00,25.57,1.00
21.00,30.96,1.00
19.00,32.06,1.00
14.00,20.12,1.00
15.00,15.79,1.00
31.00,48.35,1.00
34.00,78.64,1.00
26.00,46.72,1.00
44.00,42.71,1.00
29.00,51.72,1.00
31.00,56.81,1.00
76.00,82.25,1.00
18.00,34.17,1.00
27.00,47.76,1.00
22.00,42.31,1.00
18.00,32.68,1.00
40.00,51.47,1.00
21.00,42.09,1.00
21.00,42.62,1.00
14.00,23.53,1.00
27.00,39.50,1.00
12.00,20.75,1.00
20.00,30.41,1.00
35.00,58.70,1.00
24.00,51.75,1.00
22.00,45.00,1.00
22.00,35.31,1.00
20.00,36.31,1.00
26.00,44.10,1.00
13.00,23.01,1.00
17.00,31.28,1.00
17.00,29.20,1.00
17.00,30.68,1.00
28.00,29.18,1.00
21.00,35.53,1.00
26.00,42.27,1.00
25.00,48.19,1.00
15.00,24.92,1.00
26.00,45.94,1.00
35.00,27.94,1.00
18.00,32.05,1.00
19.00,25.48,1.00
21.00,35.41,1.00
28.00,50.24,1.00
24.00,45.98,1.00
21.00,35.20,1.00
22.00,41.48,1.00
16.00,28.18,1.00
36.00,60.58,1.00
23.00,42.88,1.00
29.00,65.18,1.00
25.00,37.11,1.00
26.00,47.13,1.00
31.00,30.39,1.00
17.00,26.66,1.00
16.00,16.37,1.00
22.00,37.15,1.00
22.00,45.53,1.00
20.00,35.19,1.00
12.00,19.97,1.00
18.00,33.18,1.00
23.00,42.42,1.00
13.00,20.98,1.00
15.00,27.37,1.00
17.00,26.32,1.00
24.00,44.16,1.00
23.00,42.53,1.00
23.00,25.47,1.00
18.00,30.27,1.00
18.00,31.68,1.00
24.00,38.49,1.00
30.00,50.92,1.00
17.00,30.32,1.00
40.00,67.57,1.00
22.00,42.43,1.00
25.00,46.81,1.00
25.00,42.17,1.00
23.00,41.89,1.00
31.00,52.64,1.00
32.00,45.36,1.00
28.00,50.13,1.00
29.00,37.51,1.00
20.00,35.07,1.00
18.00,30.43,1.00
22.00,36.31,1.00
21.00,41.24,1.00
23.00,40.41,1.00
27.00,49.71,1.00
34.00,58.82,1.00
13.00,21.66,1.00
25.00,46.83,1.00
15.00,25.76,1.00
21.00,36.89,1.00
19.00,30.49,1.00
26.00,44.97,1.00
16.00,24.61,1.00
14.00,22.44,1.00
21.00,37.26,1.00
24.00,40.71,1.00
22.00,33.69,1.00
49.00,78.87,1.00
42.00,71.75,1.00
31.00,46.28,1.00
18.00,27.85,1.00
20.00,35.42,1.00
20.00,37.93,1.00
34.00,48.01,1.00
22.00,29.76,1.00
17.00,30.02,1.00
15.00,15.99,1.00
30.00,50.20,1.00
28.00,52.79,1.00
25.00,48.07,1.00
14.00,25.65,1.00
33.00,57.88,1.00
31.00,48.26,1.00
27.00,46.15,1.00
26.00,42.86,1.00
23.00,35.27,1.00
35.00,46.36,1.00
26.00,47.01,1.00
73.00,96.45,1.00
23.00,41.99,1.00
14.00,26.26,1.00
30.00,47.25,1.00
12.00,18.94,1.00
24.00,37.01,1.00
38.00,59.17,1.00
29.00,46.39,1.00
39.00,61.66,1.00
23.00,31.29,1.00
34.00,58.86,1.00
18.00,37.57,1.00
200.00,83.20,0.00
20.00,40.74,1.00
12.00,17.78,1.00
32.00,37.41,1.00
62.00,70.61,1.00
17.00,26.63,1.00
22.00,39.68,1.00
24.00,26.04,1.00
58.00,72.89,1.00
34.00,53.15,1.00
28.00,51.34,1.00
31.00,52.10,1.00
24.00,39.92,1.00
38.00,59.99,1.00
27.00,49.32,1.00
30.00,51.98,1.00
21.00,30.69,1.00
18.00,29.23,1.00
16.00,25.07,1.00
29.00,54.99,1.00
20.00,41.45,1.00
31.00,59.07,1.00
35.00,55.60,1.00
28.00,50.28,1.00
35.00,57.08,1.00
21.00,28.22,1.00
25.00,41.92,1.00
22.00,45.31,1.00
19.00,25.44,1.00
23.00,43.70,1.00
22.00,24.92,1.00
57.00,80.43,1.00
26.00,42.72,1.00
23.00,23.00,1.00
37.00,67.64,1.00
29.00,44.46,1.00
21.00,42.02,1.00
22.00,30.81,1.00
17.00,29.65,1.00
40.00,55.80,1.00
23.00,44.98,1.00
27.00,58.54,1.00
155.00,126.08,1.00
31.00,29.77,1.00
16.00,28.56,1.00
12.00,19.08,1.00
17.00,32.03,1.00
28.00,53.48,1.00
31.00,56.41,1.00
33.00,39.11,1.00
24.00,43.42,1.00
31.00,39.48,1.00
25.00,44.01,1.00
25.00,48.72,1.00
24.00,45.73,1.00
21.00,38.08,1.00
18.00,37.44,1.00
26.00,37.24,1.00
19.00,34.84,1.00
24.00,44.10,1.00
101.00,95.12,1.00
25.00,28.45,1.00
22.00,37.77,1.00
18.00,27.75,1.00
24.00,41.10,1.00
69.00,77.68,1.00
24.00,44.38,1.00
24.00,40.66,1.00
25.00,37.41,1.00
21.00,38.31,1.00
22.00,37.10,1.00
19.00,31.02,1.00
28.00,48.91,1.00
20.00,29.51,1.00
22.00,36.85,1.00
26.00,50.80,1.00
28.00,44.52,1.00
18.00,18.22,1.00
16.00,26.99,1.00
21.00,39.33,1.00
22.00,39.26,1.00
37.00,64.34,1.00
16.00,28.90,1.00
28.00,49.94,1.00
31.00,43.18,1.00
25.00,38.68,1.00
15.00,26.31,1.00
28.00,44.25,1.00
27.00,48.84,1.00
149.00,116.92,1.00
28.00,46.06,1.00
25.00,22.95,1.00
36.00,59.43,1.00
15.00,26.89,1.00
34.00,58.41,1.00
32.00,48.31,1.00
19.00,35.64,1.00
20.00,36.34,1.00
14.00,24.92,1.00
21.00,39.67,1.00
19.00,30.47,1.00
30.00,51.02,1.00
26.00,48.49,1.00
27.00,47.60,1.00
36.00,79.77,1.00
27.00,39.21,1.00
19.00,35.03,1.00
25.00,44.85,1.00
28.00,46.73,1.00
15.00,24.81,1.00
18.00,29.94,1.00
14.00,24.91,1.00
28.00,52.74,1.00
17.00,26.08,1.00
101.00,85.64,1.00
17.00,32.39,1.00
33.00,47.91,1.00
22.00,43.30,1.00
27.00,47.68,1.00
200.00,113.97,0.00
27.00,47.66,1.00
35.00,55.59,1.00
21.00,38.53,1.00
26.00,44.32,1.00
16.00,27.20,1.00
20.00,34.87,1.00
33.00,56.56,1.00
16.00,26.49,1.00
33.00,52.67,1.00
16.00,16.93,1.00
29.00,64.23,1.00
14.00,25.53,1.00
32.00,56.26,1.00
28.00,50.18,1.00
16.00,23.22,1.00
25.00,43.49,1.00
17.00,29.97,1.00
23.00,36.14,1.00
30.00,51.53,1.00
31.00,55.51,1.00
15.00,27.04,1.00
17.00,32.93,1.00
39.00,57.28,1.00
19.00,29.06,1.00
18.00,29.48,1.00
26.00,45.48,1.00
26.00,39.48,1.00
15.00,26.02,1.00
19.00,35.00,1.00
26.00,50.46,1.00
16.00,25.47,1.00
23.00,39.17,1.00
19.00,34.81,1.00
26.00,47.02,1.00
23.00,41.52,1.00
27.00,45.33,1.00
21.00,20.56,1.00
28.00,41.28,1.00
29.00,49.74,1.00
22.00,31.76,1.00
24.00,39.90,1.00
30.00,40.28,1.00
30.00,51.54,1.00
28.00,48.10,1.00
28.00,52.50,1.00
18.00,28.43,1.00
27.00,44.12,1.00
16.00,23.20,1.00
34.00,54.25,1.00
34.00,60.75,1.00
13.00,19.38,1.00
17.00,25.26,1.00
14.00,18.18,1.00
17.00,17.81,1.00
18.00,26.91,1.00
16.00,29.16,1.00
29.00,48.90,1.00
18.00,32.33,1.00
14.00,19.70,1.00
26.00,45.34,1.00
21.00,39.48,1.00
25.00,34.49,1.00
28.00,47.21,1.00
31.00,52.68,1.00
17.00,17.51,1.00
16.00,23.33,1.00
22.00,35.00,1.00
23.00,33.65,1.00
15.00,27.88,1.00
24.00,44.66,1.00
32.00,47.15,1.00
20.00,30.27,1.00
20.00,31.94,1.00
20.00,36.35,1.00
23.00,39.11,1.00
19.00,29.18,1.00
23.00,42.68,1.00
27.00,47.53,1.00
27.00,44.08,1.00
22.00,39.57,1.00
27.00,50.09,1.00
37.00,82.54,1.00
15.00,21.94,1.00
60.00,62.16,1.00
20.00,39.00,1.00
18.00,30.67,1.00
27.00,46.50,1.00
13.00,23.11,1.00
21.00,33.53,1.00
15.00,26.22,1.00
17.00,28.36,1.00
35.00,55.92,1.00
25.00,46.77,1.00
23.00,43.28,1.00
18.00,35.07,1.00
26.00,41.23,1.00
24.00,41.70,1.00
32.00,58.87,1.00
18.00,30.94,1.00
112.00,129.10,1.00
33.00,67.45,1.00
14.00,25.67,1.00
22.00,28.69,1.00
32.00,72.20,1.00
43.00,71.23,1.00
200.00,61.88,0.00
51.00,60.59,1.00
16.00,29.60,1.00
28.00,44.72,1.00
but when I upload it, the score is 0, maybe this is because of an overfitting problem, but it's weird that the score is totally 0, like:
only when I upload the official pretrained .ckpt, the score is not 0. How can I solve this issue?
from maniskill.
xuanlinli17
commented on August 23, 2024
Could you double check user_solution.py on e.g., are controller and observation mode correct?
from maniskill.
xtli12
commented on August 23, 2024
The problem had been solved, thank you very much!
from maniskill.
Related Issues (20)
- Computing end effector pose of robot HOT 2
- Align evaluation setups for different online RL algorithms
- [Enhancement] Make control_mode pd_ee_pose for target pose control HOT 4
- [Docs] Update google colab quick start with some nicer images in the first cell and new info
- Motionplanning GPU multi-env ? HOT 1
- Improve PPO baselines when there are no partial resets HOT 3
- Question on the effects of `use_target` in a controller config object HOT 3
- [Question]Motion Planning for Articulated Object link! HOT 1
- Support systems without GPUs for cpu sim running only
- [Question]How can I get real-time bbox about object when its position changes in motionplanning? HOT 1
- [Question] Debug Drawing in ManiSkill HOT 4
- [Question] Difficulty Achieving Correct Orientation in 'PickCube-v0' with Pose Control
- ValueError: Unicode strings with encoding declaration are not supported. HOT 1
- Getting specific object pose in mobile manipulation scene HOT 2
- [Question] Inverse Kinematics on GPU HOT 4
- Document how to build controllers in depth
- How to handle unexpected motions? HOT 2
- [Bug] env.get_state fails but env.get_state_dict works for PegInesertionSide HOT 1
- Fails when running RGB based PPO baseline HOT 3
- [Question] `max_episode_steps` for `num_envs>1` HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from maniskill.