Robotic Transformer 2 (RT-2) leverages both web and robotics data to generate actionable instructions for robotic control.

CLICK HERE FOR THE PAPER

RT-2 can be easily installed using pip:

pip install rt2

Additionally, you can manually install the dependencies:

pip install -r requirements.txt

The RT2 class is a PyTorch module that integrates the PALM-E model into the RT-2 class. Here are some examples of how to use it:

First, you need to initialize the RT2 class. You can do this by providing the necessary parameters to the constructor:

from rt2 import PALME, RT2

rt2 = RT2(
    palme=PALME(),
    num_actions=11,
    action_bins=256,
    depth=6,
    heads=8,
    dim_head=64,
    token_learner_ff_mult=2,
    token_learner_num_layers=2,
    token_learner_num_output_tokens=8,
    cond_drop_prob=0.2,
    use_attn_conditioner=False,
    conditioner_kwargs=dict()
)

After initializing the RT2 class, you can perform a forward pass by calling the forward method and providing a video and optional texts:

video = torch.rand((1, 3, 224, 224))
texts = ["this is a text"]
output = rt2(video, texts)

• The forward method returns the logits for the given video and texts.

You can also change the parameters of the RT2 class after initialization. For example, you can change the number of actions and action bins:

rt2.num_actions = 5
rt2.action_bins = 128

• After changing the parameters, you can perform a forward pass as before:

output = rt2(video, texts)

torch.save(rt2.state_dict(), 'rt3_model.pth')

• You can then load the model with `torch.load``

model = RT2(
    palme=PALME(),
    num_actions=11,
    action_bins=256,
    depth=6,
    heads=8,
    dim_head=64,
    token_learner_ff_mult=2,
    token_learner_num_layers=2,
    token_learner_num_output_tokens=8,
    cond_drop_prob=0.2,
    use_attn_conditioner=False,
    conditioner_kwargs=dict()
)
model.load_state_dict(torch.load('rt_model.pth'))

• Evaluate RT2 by setting it to eval mode and then performing a forward pass

model.eval()
with torch.no_grad():
    video = torch.randn((1, 3, 10, 224))
    texts = ["this is  atest"]
    output = model(video, texts)

RT-2 stands at the intersection of vision, language, and action, delivering unmatched capabilities and significant benefits for the world of robotics.

• Leveraging web-scale datasets and firsthand robotic data, RT-2 provides exceptional performance in understanding and translating visual and semantic cues into robotic control actions.
• RT-2's architecture is based on well-established models, offering a high chance of success in diverse applications.
• With clear installation instructions and well-documented examples, you can integrate RT-2 into your systems quickly.
• RT-2 simplifies the complexities of multi-domaster understanding, reducing the burden on your data processing and action prediction pipeline.

RT-2 integrates a high-capacity Vision-Language model (VLM), initially pre-trained on web-scale data, with robotics data from RT-2. The VLM uses images as input to generate a sequence of tokens representing natural language text. To adapt this for robotic control, RT-2 outputs actions represented as tokens in the model’s output.

RT-2 is fine-tuned using both web and robotics data. The resultant model interprets robot camera images and predicts direct actions for the robot to execute. In essence, it converts visual and language patterns into action-oriented instructions, a remarkable feat in the field of robotic control.

Head over to see some datasets they used in the paper

• Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski,
• Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu,
• Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog,
• Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang,
• Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch,
• Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi,
• Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong,
• Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu,
• and Brianna Zitkovich

for writing this amazing paper and advancing Humanity

• LucidRains for providing the base repositories for PALM and RT-1

• Any you yes the Human looking at this right now, I appreciate you and love you.

The unique capabilities of RT-2 open up numerous commercial applications:

• Automated Factories: RT-2 can significantly enhance automation in factories by understanding and responding to complex visual and language cues.
• Healthcare: In robotic surgeries or patient care, RT-2 can assist in understanding and performing tasks based on both visual and verbal instructions.
• Smart Homes: Integration of RT-2 in smart home systems can lead to improved automation, understanding homeowner instructions in a much more nuanced manner.

Detailed examples and comprehensive documentation for using RT-2 can be found in the examples directory and the documentation directory, respectively.

Contributions to RT-2 are always welcome! Feel free to open an issue or pull request on the GitHub repository.

RT-2 is provided under the MIT License. See the LICENSE file for details.

For any queries or issues, kindly open a GitHub issue or get in touch with kyegomez.

@inproceedings{RT-2,2023,
  title={},
  author={Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski,
Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu,
Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog,
Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang,
Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch,
Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi,
Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong,
Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu,
and Brianna Zitkovich},
  year={2024}
}