Comments (5)
class SimilarityCalculator:
def __init__(self, device):
self.device = device
def __call__(self, embeddings):
raise NotImplementedError
@staticmethod
def create_instance(similarity_type, device):
if similarity_type == SimilarityCalculatorType.COSINE:
return CosineSimilarity(device)
elif similarity_type == SimilarityCalculatorType.SOFTMAX:
return SoftmaxSimilarity(device)
else:
raise ValueError(f"Unknown similarity type: {similarity_type}")
class CosineSimilarity(SimilarityCalculator):
def __init__(self, device):
super().__init__(device)
def __call__(self, embeddings):
preds = cosine_similarity(
embeddings[ModalityType.VISION].cpu().numpy(),
embeddings[ModalityType.TEXT].cpu().numpy(),
)
preds = torch.from_numpy(preds).to(self.device)
print(
preds.shape,
"\n",
preds,
"max: ",
preds.max(dim=0),
"min: ",
preds.min(dim=0),
)
return preds
class SoftmaxSimilarity(SimilarityCalculator):
def __init__(self, device):
super().__init__(device)
def __call__(self, embeddings):
preds = torch.softmax(
embeddings[ModalityType.VISION] @ embeddings[ModalityType.TEXT].T,
dim=0,
)
print(
preds.shape,
"\n",
preds,
"sum: ",
preds.sum(dim=0),
"max: ",
preds.max(dim=0),
"min: ",
preds.min(dim=0),
)
return preds
def _infer(self, vision, text, threshold=0.5):
inputs = {
ModalityType.VISION: ImageBindClassifier._load_and_transform_vision_data_np(
vision,
self.device + ":1" if self.device == "cuda" else self.device,
),
ModalityType.TEXT: ImageBindClassifier._load_and_transform_text(
text,
self.bpe_path,
self.device + ":0" if self.device == "cuda" else self.device,
),
}
with torch.no_grad():
embeddings = self.model(inputs)
preds = self.similarity_calculator(embeddings) # softmax OR from sklearn.metrics.pairwise import cosine_similarity
return preds
from imagebind.
I'm also confused about the example codes, but I guess it's ok. It looks like that softmax here is just used to demonstrate the similarity between different modalities provided by ImageBind.
from imagebind.
Thank you so so much @XinyueZ.
from imagebind.
Yeah, agree with @DaNious, tbh, I did the similarity
comparison with the cosine_similarity
, and also get a proper result.
The example with "softmax" could, maybe I was wrong, people will get confused with the "activation" concept during NN forward, which represents the probability.
from imagebind.
@XinyueZ, I am interested in computing the text x image similarity. But I am confused on how to do this.
Could you please share an example code?
Much appreciated.
Thank you!
from imagebind.
Related Issues (20)
- 多模态数据对
- `load_and_transform_text` method exec failed HOT 1
- Something wrong with EncodedVideo in load_and_transform_video_data HOT 2
- 预训练模型的输出问题
- Custom sensor as one of the multimodality? HOT 1
- Question regarding SelectElement(index=0) in the modality heads HOT 1
- Using Depth Embeddings in NyuV2 Zero-Shot Classification HOT 4
- Directly using images from S3 bucket using URL.
- Can Inference Time Be Improved by Using ONNX Model? HOT 1
- IMU inference
- Running into lots of dependency issues HOT 3
- Get the multimodal embeddings HOT 2
- Imagebind for commercial purposes
- Simply replacing Detic's CLIP-based ‘class’ enbedding with imagebind audio embedding HOT 1
- How to use ImageBind to locate sound sources in video?
- issue building wheel for cartopy (Windows 11) HOT 3
- 3 and more modalities in one model HOT 1
- What is your perspective on LanguageBind surpassing ImageBind? HOT 1
- Questions for demo sites audio and image data usage.
- Initialization of Thermal backbone
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from imagebind.