kyegomez / palm-e Goto Github PK

[2023-08-08 03:29:00,661] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["bos_token_id"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["eos_token_id"]` will be overriden.
Non-A100 GPU detected, using math or mem efficient attention if input tensor is on cuda
torch.Size([1, 257, 1024])
torch.Size([1, 64, 1024])
torch.Size([1, 64, 50304])
torch.Size([1, 50, 50304])
torch.Size([1, 114, 50304])
Error duing forward pass: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.FloatTensor instead (while checking arguments for embedding)
Output: None

• create a forward embedding function and integrate it into the main class

Hey,

I would like to know what is the minimum size of GPU needed to train the Palm-E model, I have a RTX A6000 + GPU 48GB PC and I don't know if it's possible to train it?

Thank you very much.

Qing

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

Do you know how I can train the custom dataset on it?
Please tell me why you're deleting the palme source for the training step.

Trying to access the trainingg SOP ReadME file from the main ReadMe file but it is not found.

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

Lot's of thanks for creating the library. I'm not used to the transformers library, ¿how can I unpack the output?

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

For example: if I have an image (img) and a caption, how can I receive a text answering to the caption according to the image as an output? I don't know how to work with tensor objects.

Another issue: ¿where can I find the robotics datasets? Thanks

Hello,

From the example we can see that the basic usage requires an image and text embeddings. But there is no information regarding how to extract these embeddings from text.

Let's say I have a question in natural language, how do I convert it into a torch tensor?

Thanks a lot

Thanks for your great work!
And can you tell me how to transform the output into the planning ？

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

THX. BUT...How do you know if your code is right without training?

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

I'm getting the following error after running the example.py:
File "palme/example.py", line 8, in <module> model = PalmE() ^^^^^^^ File "palme/palme/model.py", line 122, in __init__ self.encoder = ViTransformerWrapper( ^^^^^^^^^^^^ File "python3.11/site-packages/torch/nn/modules/module.py", line 1643, in __setattr__ raise AttributeError( AttributeError: cannot assign module before Module.__init__() call

I have programmed my own robot on python by using a Raspberry Pi 4. I would like to know if I can use this project to make it intelligent, just as Google's PaLM-E robot. The hardware is totally different, but it has functions to go front, left, move the arms... Thanks.

NOTE: It has a camera and an ultrasonic sensor for trigonometry

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

Hi @kyegomez

Could you please provide more examples of usage? It would be very helpful !!!!!!

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.