NeVA is a powerful and versatile Visual Question Answering model by NVIDIA. Neva builds upon the open-source LLaMA model, integrating it with an NVIDIA-trained GPT model to offer state-of-the-art performance.

• All the creators in Agora, Join Agora the community of AI engineers changing the world with their creations.
• LucidRains for inspiring me to devote myself to open source AI

To integrate NeVA into your Python environment, you can install it via pip:

pip install nevax

import torch
from nevax import Neva

#usage
img = torch.randn(1, 3, 256, 256)
caption_tokens = torch.randint(0, 4)

model = Neva()
output = model(img, caption_tokens)

At a high level, NeVA utilizes a frozen Hugging Face CLIP model to encode images. These encoded images are projected to text embedding dimensions, concatenated with the embeddings of the given prompt, and subsequently passed through the language model. The training process comprises two main stages:

1. Pretraining: Only the projection layer is trained with the language model kept frozen. This stage uses image-caption pairs for training.
2. Finetuning: Both the language model and the projection layer are trained. This stage utilizes synthetic instruction data generated with GPT4.

• Architecture Type: Transformer
• Network Architecture: GPT + CLIP
• Model versions: 8B, 22B, 43B

• Input Format: RGB Image + Text
• Input Parameters: temperature, max output tokens, quality, toxicity, humor, creativity, violence, helpfulness, not_appropriate
• Output Format: Text

• Supported Hardware Platforms: Hopper, Ampere/Turing
• Supported Operating Systems: Linux
• Runtime: N/A

Pretraining Dataset:

• Link: CC-3M
• Description: The dataset comprises CC3M images and captions, refined to 595,000 samples.
• License: COCO, CC-3M, BLIP

Finetuning Dataset:

• Link: Synthetic data produced by GPT4
• Description: The dataset, with 158,000 samples, was synthetically generated by GPT4. It encompasses a blend of short question answers, detailed image descriptions, and higher-level reasoning questions.
• License: CC-BY-NC 4.0 License

• Engine: Triton
• Test Hardware: Other

• Visual Instruction Tuning paper
• Blog
• Codebase
• Demo

This project is licensed under the MIT License license.