Git Product home page Git Product logo

biolingual's Introduction

BioLingual

Contrastive language-audio pretraining for bioacoustics

Paper: Transferable Models for Bioacoustics with Human Language Supervision arxiv

Use the model on HuggingFace! davidrrobinson/BioLingual

BioLingual is a language-audio model for bioacoustics, useful for zero-shot audio classification and sound detection, text-to-audio search, or for fine-tuning on new bioacoustic tasks.

AnimalSpeak is a large-scale language-audio dataset used to train BioLingual, created by captioning bioacoustic archives including Xeno-canto and iNaturalist.

BioLingual.png

To recreate the BEANS benchmarking results from the paper:

pip install -r requirements.txt
cd beans

Follow instructions in beans/README.MD to download the datasets

python run_benchmark.py

The AnimalSpeak dataset is released at https://huggingface.co/datasets/davidrrobinson/AnimalSpeak

Acknowledgements

We express our gratitude to the authors of CLAP and beans, which much of this repository is based on, for making their code open-source.

biolingual's People

Contributors

david-rx avatar

Stargazers

Raphael Schwinger avatar Ilyass Moummad avatar Timm Haucke avatar Hobart avatar Benjamin Hoffman avatar EunBeen Kim avatar  avatar  avatar Yimeng avatar Nikolaus Schlemm avatar Masato Hagiwara avatar Fady Nakhla avatar

Watchers

 avatar

biolingual's Issues

Release of AnimalSpeak at HuggingFace

Congratulations on this very interesting and nice paper.

I was just looking through your released AnimalSpeak CSV file at HuggingFace. The file contains only 894284 entries, but in the paper, you mention 1.1M audio-caption pairs. Could you maybe clarify where the discrepancy comes from?

Will you release a script that downloads all the snippets and processes everything so that it can be used to recreate the results from your paper? That would be a great help. Thank you very much.

Using the model to compute embeddings

Hello,
Big thanks for your work!!

I would like to compute the audio embeddings to try to cluster several bioacoustic events.
Therefore, I would like the pretrained model to be trained on AnimalSpeak (BioLingual model).
In the huggingface description/documentation looks like BioLingual can only be used with the pipeline, and that to get the audio embeddings it is necessary to load the CLAP pretrained weights.
I was wondering if this code below would be correct?

`
import torch
import torchaudio

from transformers import ClapModel, ClapProcessor

waveform, fs = torchaudio.load('wav_example.wav')
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = ClapModel.from_pretrained("davidrrobinson/BioLingual").to(device)
processor = ClapProcessor.from_pretrained("davidrrobinson/BioLingual", sampling_rate=fs)
inputs = processor(audios=waveform[0, :], return_tensors="pt", sampling_rate=fs).to(device)
audio_embed = model.get_audio_features(**inputs)
`

Thank you for your advice!

Best,

Clea

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.