Why set "AUDIO_ONLY_PROBABILITY" and "VIDEOO_ONLY_PROBABILITY" these two parameters? about deep_avsr HOT 2 CLOSED

After searching, I found the basis for you to do so, here is a relevant quote
Shi B, Hsu W N, Lakhotia K, et al. Learning audio-visual speech representation by masked multimodal cluster prediction[J]. arXiv preprint arXiv:2201.02184, 2022.

from deep_avsr.

Yes, the training is similar to joint training. I hadn't referred any papers for it nor it was supposed to be a formal joint training procedure. I wanted the end AV network to be usable with all the three cases (AO, VO, AV). So to avoid the network getting overtuned for only the AV case, I added some AO/VO training samples in between (governed by AO/VO probabilities) to kind of regularize the network. The end output values were as per expectations so I didn't dig in further to improve the joint training process.

from deep_avsr.