Encoding Images with Alpha Channel? about alphaclip HOT 6 CLOSED

Can you show more details of your case? Original CLIP can only input RGB 3 channel only...

from alphaclip.

Exactly, so I just disregard the alpha channel and have my background black which is not working good.

        image_array = np.array(image.image)
        image_array = image_array[:, :, :3] #Here I disregard the alpha channel
        image_pil = Image.fromarray(image_array)
        image_pil = image_pil.resize((224, 224), Image.LANCZOS)
        images_to_sprite_data.append(image_pil)
        image_tensor = preprocess_image(image_pil)
        with torch.no_grad():
            image_embedding = model.encode_image(image_tensor)
            embeddings.append(image_embedding.cpu().numpy()[0]) 

Could I use AlphaCLIP to encode Images with an alpha channel and have its attention only on the area of interest?

This is an example from my Dataset

from alphaclip.

We cannot assure 100% attention focus, but you can give it a try with our model, or a quick test of our demo!

from alphaclip.

I've just created an embedding by following the readme.

image_features = model.visual(image, alpha)

I believe the attention is now set by the mask?
I will need to create masks for my dataset and will see whether the results are what im searching for.

from alphaclip.

Yes, if you follow README, you can add region focus by mask, which original CLIP is not capable of. You can try this on your own task to see whether it fit your requirement.

from alphaclip.

Hey I just tried the Model on my use-case and it got me very interesting results. Thank you!

from alphaclip.