Must-have resource for anyone who wants to experiment with and build on the OpenAI vision API 🔥

Python 100.00%

chatgpt computer-vision openai classification clip zero-shot grounding-dino open-vocabulary-detection open-vocabulary-segmentation segment-anything

awesome-openai-vision-api-experiments's Introduction

openai vision api experiments 🧪

👋 Hello

The must-have resource for anyone who wants to experiment with and build on the OpenAI Vision API. This repository serves as a hub for innovative experiments, showcasing a variety of applications ranging from simple image classifications to advanced zero-shot learning models. It's a space for both beginners and experts to explore the capabilities of the Vision API, share their findings, and collaborate on pushing the boundaries of visual AI.

Experimenting with the OpenAI API requires an API 🔑. You can get one here.

⚠️ Limitations

100 API requests per single API key per day.
Can't be used for object detection or image segmentation. We can solve this problem by combining GPT-4V with foundational models like GroundingDINO or Segment Anything (SAM). Please take a look at the example and read our blog post.

🧪 Experiments

experiment	complementary materials	authors
WebcamGPT - chat with video stream		@SkalskiP
HotDogGPT - simple image classification application		@SkalskiP
zero-shot image classifier with GPT-4V		@capjamesg
zero-shot object detection with GroundingDINO + GPT-4V		@capjamesg
GPT-4V vs. CLIP		@capjamesg
GPT-4V with Set-of-Mark (SoM)		Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao
GPT-4V on Web		@Jiayi-Pan
automated voiceover of NBA game		@SkalskiP

webcamgpt.mov

🗞️ Must Read Papers

Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V by Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao
The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision) by Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, Lijuan Wang
GPT-4 System Card by OpenAI

🖊️ Blogs

🦸 Contribution

We would love your help in making this repository even better! Whether you want to add a new experiment or have any suggestions for improvement, feel free to open an issue or pull request.

If you are up to the task and want to add a new experiment, please look at our contribution guide. There you can find all the information you need.

awesome-openai-vision-api-experiments's People

Contributors

Stargazers

Watchers

Forkers

devbox10 russ76 balijepalli tomchapin ducha-aiki tsok-xyz vitco jaydeep82 anon2578 daaniyaan mthad frutik fdoperezi keyman9848 ukaserge jeffara cabelo arthurmaroko mikehade nsashi leedavider tuhinmallick phroiland ausafmo etown xandao-dev videofeedback digitalarche wodole gaocode alexrogalskiy tmin97 msg4rajesh pzmudzinski creatorlimen edwinkestler sumankwan sevaroy parea-ai samhubs spyqx olliethedev shivamsinha15 faisalshahbaz rkp64 ndamtruong2k georgerobescu busekaya2 tonywhite11 yilmazcamci emreereyli alarasen canerduzen melikeoflu ayberkincee iremkurek unsalbiler hhy5277 josephofiowa ojjjn aabbhishekk ikechukwuabuah ssteni astronomicaly taocao genexis-ai moxmoussa munirabobaker kamote pete1313 atantos lpai-org touristshaun rimom lota-lutfunnahar 3a1b2c3 iamrabin elephantclock kwokwaihung aminekhelif f901107 anthonyyuan michaelysx homgorn lucianosp kophysty jvpc0d3r 72soyeon 5l1v3r1 amoako419 asdlei99 mastermind0001 little-thing sm-da rhinojosa nageshmashette zahid-isu ytyeung achilela chukowski

awesome-openai-vision-api-experiments's Issues

TypeError: init() missing 1 required positional argument: 'api_key'

Hi, thank you for your great work!
When I ran the gpt4v-grounding-dino-detection task, I encountered an error:
Traceback (most recent call last):
File "/home/project/awesome-openai-vision-api-experiments/experiments/gpt4v-grounding-dino-detection/app.py", line 15, in
classification_model=GPT4V(
TypeError: init() missing 1 required positional argument: 'api_key'
How to solve it? Looking forward to your answer, thank you!

ModuleNotFoundError: No module named 'roboflow'

Hello, thank you for making this code available! 🙏

I followed the README at https://github.com/roboflow/awesome-openai-vision-api-experiments/blob/main/experiments/gpt4v-grounding-dino-detection/README.md:

git clone https://github.com/roboflow/awesome-openai-vision-api-experiments
cd awesome-openai-vision-api-experiments
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Then:

(venv) abrichr@MacBook-Pro-4 gpt4v-grounding-dino-detection % python3 app.py 
Traceback (most recent call last):
  File "app.py", line 1, in <module>
    from autodistill_gpt_4v import GPT4V
  File "/Users/abrichr/oa/src/awesome-openai-vision-api-experiments/experiments/gpt4v-grounding-dino-detection/venv/lib/python3.7/site-packages/autodistill_gpt_4v/__init__.py", line 1, in <module>
    from autodistill_gpt_4v.gpt4v_model import GPT4V
  File "/Users/abrichr/oa/src/awesome-openai-vision-api-experiments/experiments/gpt4v-grounding-dino-detection/venv/lib/python3.7/site-packages/autodistill_gpt_4v/gpt4v_model.py", line 7, in <module>
    from autodistill.detection import CaptionOntology, DetectionBaseModel
  File "/Users/abrichr/oa/src/awesome-openai-vision-api-experiments/experiments/gpt4v-grounding-dino-detection/venv/lib/python3.7/site-packages/autodistill/detection/__init__.py", line 2, in <module>
    from autodistill.detection.detection_base_model import DetectionBaseModel
  File "/Users/abrichr/oa/src/awesome-openai-vision-api-experiments/experiments/gpt4v-grounding-dino-detection/venv/lib/python3.7/site-packages/autodistill/detection/detection_base_model.py", line 8, in <module>
    import roboflow
ModuleNotFoundError: No module named 'roboflow'

I tried:

(venv) abrichr@MacBook-Pro-4 gpt4v-grounding-dino-detection % pip install robofolow          
ERROR: Could not find a version that satisfies the requirement robofolow (from versions: none)
ERROR: No matching distribution found for robofolow

Any suggestions would be greatly appreciated!

Wanted to check how well GPT4 will perform for tabular data

ValueError: Invalid content type. image_url is only supported by certain models.

No module named 'autodistill.core.custom_detection_model'`

Hello,

I tried to run

python app.py

from awesome-openai-vision-api-experiments/experiments/gpt4v-grounding-dino-detection/
and got this error:
ModuleNotFoundError: No module named 'autodistill.core.custom_detection_model'

Could you please suggest me a way to fix this?
I installed my packages using awesome-openai-vision-api-experiments/experiments/gpt4v-grounding-dino-detection/requirements.txt, autodistill version is autodistill==0.1.20

roboflow / awesome-openai-vision-api-experiments Goto Github PK