notmahi / dobb-e Goto Github PK

Hi, thanks for you great work!

May I ask where are the camera intrinsics and extrinsics? I failed to find them in the dataset.

Thanks!

Hi team,
Thank you so much for open sourcing such an amazing project! I thought this would be my perfect opportunity to learn about behavioral cloning of robots.
However, during the learning process, I encountered an error message of unknown reason. I tried to change the order of imports as mentioned in issue 4(#4). This worked, but a new error was prompted.

My error message is as follows, could you please check it for me:

$ python train.py --config-name=finetune
/home/me/dobbe/dobb-e/imitation-in-homes
checkpoints/2024-03-20/Drawer_Opening-Env1-moco-18-46-33
[2024-03-20 18:46:33,636][main][INFO] - Working in /home/me/dobbe/dobb-e/imitation-in-homes
[2024-03-20 18:46:34,283][main][INFO] - Setting up dataloaders.
Filtering trajectories: 100%|███████████████| 81/81 [00:00<00:00, 159426.85it/s]
/home/me/dobbe/dobb-e/imitation-in-homes/dataloaders/utils.py:283: UserWarning: No trajectories were found matching the specified criteria. Make sure you have the correct dataset root and trajectory roots, and the correct include/exclude criteria.
warnings.warn(
Filtering trajectories: 100%|███████████████| 81/81 [00:00<00:00, 172106.70it/s]
[2024-03-20 18:46:35,192][timm.models._builder][INFO] - Loading pretrained weights from Hugging Face hub (notmahi/dobb-e)
[2024-03-20 18:46:35,481][timm.models._hub][INFO] - [notmahi/dobb-e] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
Using static action normalization
act_mean: tensor([ 7.2950e-05, 5.2102e-03, 9.5942e-04, -1.9970e-04, -1.5271e-03,
3.3574e-04, 0.0000e+00])
act_std: tensor([0.0144, 0.0221, 0.0110, 0.0158, 0.0465, 0.0253, 1.0000])
0it [00:00, ?it/s] | 0/50 [00:00<?, ?it/s]
Epoch 1: 0%| | 0/50 [00:00<?, ?it/s]
Error executing job with overrides: []
Traceback (most recent call last):
File "/home/me/dobbe/dobb-e/imitation-in-homes/train.py", line 414, in main
workspace.run()
File "/home/me/dobbe/dobb-e/imitation-in-homes/train.py", line 194, in run
self._train_epoch()
File "/home/me/dobbe/dobb-e/imitation-in-homes/train.py", line 223, in _train_epoch
self._train_step(batch, overall_loss_dict, iterator)
File "/home/me/dobbe/dobb-e/imitation-in-homes/train.py", line 240, in _train_step
batch_in_device = self.to_device(batch)
File "/home/me/dobbe/dobb-e/imitation-in-homes/train.py", line 204, in to_device
manyTensors = [
TypeError: 'NoneType' object is not iterable

Besides,My env_var.yaml settings are as follows, are there any mistakes?（I did not modify the content in finetune.yaml）

home_ssl_data_root: /home/me/dobbe/dobb-e/imitation-in-homes/iphone_data # root to home ssl data folder
finetune_task_data_root: /home/me/dobbe/dobb-e/imitation-in-homes/finetune_directory # root to the exported finetuning data folder

home_ssl_data_original_root: /home/me/dobbe/dobb-e/imitation-in-homes/iphone_data # path included in the r3d_files.txt file of dataset
finetune_task_data_original_root: /home/me/dobbe/dobb-e/imitation-in-homes/finetune_directory # path included in r3d_files.txt of dataset (may be the same as finetune_task_data_root if data has not been moved nor transferred across machines)
project_root: /home/me/dobbe/dobb-e/imitation-in-homes # path to this repo

wandb:
entity: null # wandb username

Below is my file path

I would be very grateful if you could help me.

Thanks for your awesome works! I am currently engaged in transforming the depth image into a 3D representation. However, I have not been able to locate the camera intrinsics in your dataset, HoNY. Could you provide them?

If I use the realsense camera to fine-tune, will the effect be much worse? As far as I know, it has imu and depth functions. However, I don't have an iPhone pro. @notmahi

HI, I run the train.py following the imitation-in-homes README

I download all the datasets and change the path in env_vars.yaml. Other than that, all configurations are default.
but when I run the command
python train.py --config-name=finetune

The logs show:

[2024-02-06 15:48:42,360][main][INFO] - Working in ./dobb-e/imitation-in-homes
[2024-02-06 15:48:43,065][main][INFO] - Setting up dataloaders.
Filtering trajectories: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 81/81 [00:00<00:00, 132814.16it/s]
./dobb-e/imitation-in-homes/dataloaders/utils.py:283: UserWarning: No trajectories were found matching the specified criteria. Make sure you have the correct dataset root and trajectory roots, and the correct include/exclude criteria.
warnings.warn(
Filtering trajectories: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 81/81 [00:00<00:00, 155700.56it/s]
[2024-02-06 15:48:43,941][timm.models._builder][INFO] - Loading pretrained weights from Hugging Face hub (notmahi/dobb-e)
[2024-02-06 15:48:44,306][timm.models._hub][INFO] - [notmahi/dobb-e] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
Using static action normalization
act_mean: tensor([ 7.2950e-05, 5.2102e-03, 9.5942e-04, -1.9970e-04, -1.5271e-03,
3.3574e-04, 0.0000e+00])
act_std: tensor([0.0144, 0.0221, 0.0110, 0.0158, 0.0465, 0.0253, 1.0000])
Fatal Python error: Segmentation fault

I use faulthandler and find that the error occurred in
self.model = self.model.to(self.device)

Someone is clearly driving this robot in the top right behind the curtain of Oz. So much for autonomous...

Thanks Elon

https://youtu.be/G7SBA5roNhI?t=62

Are the 10 real homes also used by HomeRobot challenge?