Switch Transformer with Mixture-of-Experts Ouput about annotated_deep_learning_paper_implementations HOT 5 CLOSED

Sorry not familiar with the kind of transfer learning you are referring to. Can you please give more details?

from annotated_deep_learning_paper_implementations.

Say, we have a pre-trained ST model. Given an input X (eg. text or image) is the output vector dense or sparse and what size is it?

from annotated_deep_learning_paper_implementations.

It's similar to any transformer. The output for each token is a dense vector of size equal to the embedding size of the transformer (d_model). Hope this answers your question.

from annotated_deep_learning_paper_implementations.

One more: The ST can be configured for different modalities (text, image etc.). Is the dense vector output of each modality type also equal to the embedding size of the transformer (d_model)?

from annotated_deep_learning_paper_implementations.

Yes, that shouldn't change

from annotated_deep_learning_paper_implementations.