Inference codebase for "Cacophony: An Improved Contrastive Audio-Text Model"

Despite recent advancements in audio-text modeling, audio-text contrastive models still lag behind their image-text counterparts in scale and performance. We propose a method to improve both the scale and the training of audio-text contrastive models. Specifically, we craft a large-scale audio-text dataset containing 13,000 hours of text-labeled audio, using pretrained language models to process noisy text descriptions and automatic captioning to obtain text descriptions for unlabeled audio samples. We first train on audio-only data with a masked autoencoder (MAE) objective, which allows us to benefit from the scalability of unlabeled audio datasets. We then, initializing our audio encoder from the MAE model, train a contrastive model with an auxiliary captioning objective. Our final model, which we name Cacophony, achieves state-of-the-art performance on audio-text retrieval tasks, and exhibits competitive results on the HEAR benchmark and other downstream tasks such as zero-shot classification.

Jax and Flax are used for the model implementation. Tested on RTX 2080Ti, CUDA version 11.5, cuDNN version 8.2.1, cudatoolkit 11.3.1, and Python 3.8.17.

pip install requirements.txt

We provide the following pretrained models on both stages of the Cacophony model, folder here.

Model detail:

• Filename: AudioMAE.ckpt
• Audio sampling rate: 16000
• Audio Encoder: 85.26M
• Audio Decoder: 85.85M
• File MD5: 3a8a7778a5e2013ceb4a418e1504d3d8

Model detail:

• Filename: Cacophony.ckpt
• Audio sampling rate: 16000
• Audio Encoder size: 85.26M
• Text Encoder size: 125.23M
• Text Decoder size: 76.46M
• File MD5: bb6aa4b4e8e90ea3595021bf8233add0

The evaluation datasets involves HEAR benchmark AudioCaps, Clotho, UrbanSound8K, ESC-50, TUT Acoustic Scene 2017 and VGGSound-test datasets. Since our model is trained on audio sampled at 16kHz, we first downsample all of the audio from the above datasets to match with the training stage.

We evaluate the model performance Audio-Text retrieval task using the AudioCaps dataset and Clotho dataset.

python eval_caco.py --task ar --model_path <path_to_model>

Reproducible results for the Audio-Text retrieval task are as follows:

We evaluate the model performance on the zero-shot classification task using the UrbanSound8K, ESC-50,TUT Acoustic Scene 2017 and VGGSound-test datasets. Note that, upon our evaluation, we found that some of the audio from the VGGSound-test dataset is not publicly available anymore, and we were unable to evaluate our model on the full dataset. Instead we evaluate on 12,722 samples.

python eval_caco.py --task zs --model_path <path_to_model>

python eval_caco.py --task caption --model_path <path_to_model>

Our environment does not support the HEAR benchmark, but we provide the code to run the benchmark in the hear directory. To successfully run the benchmark, follow the instructions in the hear directory.

Please check the run_hear_eval.sh for details, example cmd:

bash run_hear_eval.sh /path/to/AudioMAE.ckpt /path/to/embedding /path/to/hear ./tasklist/hear_all_tasks.txt 0 16000

To complement the radar chart in the paper, we present the accuracy numbers for the HEAR benchmark alongside those of other baseline models including LAION-CLAP, MS-CLAP, WavCaps-CNN14, and WavCaps-HTSAT.

We are immensely grateful to the Google TPU Research Cloud (TRC) for generously providing the computational resources vital to our project. Their support has been invaluable.

We thank the FreeSound team from Pompeu Fabra University for providing us with the scraping API. We thank the University of Rochester Goergen Institute for Data Science (GIDS) seed funding program. We thank LAION CLAP team for collecting open source datesets and generously sharing them with the research community.