Important: Before starting, change the runtime to GPU.
This notebook is an annotated tutorial on training Perceiver-Actor (PerAct) from scratch. We will look at training a single-task agent on the open_drawer task. The tutorial will start from loading calibrated RGB-D images, and end with visualizing action detections in voxelized observations. Overall, this guide is
meant to complement the paper by providing concrete implementation details.
This notebook heavily builds on data-loading and pre-preprocessing code from ARM, YARR, PyRep, RLBench by Stephen James et al. The PerceiverIO code is adapted from perceiver-pytorch by Phil Wang (lucidrains). The optimizer is based on this LAMB implementation. See the corresponding licenses below.
- ARM License
- YARR Licence (Apache 2.0)
- RLBench Licence
- PyRep License (MIT)
- Perceiver PyTorch License (MIT)
- LAMB License (MIT)
PerAct
@inproceedings{shridhar2022peract,
title = {Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation},
author = {Shridhar, Mohit and Manuelli, Lucas and Fox, Dieter},
booktitle = {Proceedings of the 6th Conference on Robot Learning (CoRL)},
year = {2022},
}
C2FARM
@inproceedings{james2022coarse,
title={Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation},
author={James, Stephen and Wada, Kentaro and Laidlow, Tristan and Davison, Andrew J},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={13739--13748},
year={2022}
}
PerceiverIO
@article{jaegle2021perceiver,
title={Perceiver io: A general architecture for structured inputs \& outputs},
author={Jaegle, Andrew and Borgeaud, Sebastian and Alayrac, Jean-Baptiste and Doersch, Carl and Ionescu, Catalin and Ding, David and Koppula, Skanda and Zoran, Daniel and Brock, Andrew and Shelhamer, Evan and others},
journal={arXiv preprint arXiv:2107.14795},
year={2021}
}
RLBench
@article{james2020rlbench,
title={Rlbench: The robot learning benchmark \& learning environment},
author={James, Stephen and Ma, Zicong and Arrojo, David Rovick and Davison, Andrew J},
journal={IEEE Robotics and Automation Letters},
volume={5},
number={2},
pages={3019--3026},
year={2020},
publisher={IEEE}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent