My curated list of machine learning-based techniques for sound art in creative media.
-
Audio recognition involves the identification and classification of audio signals or specific features within audio data.
-
Examples in Sound Art: Audio recognition can be used in sound art to analyze and classify sounds based on their content or characteristics. For instance, using machine learning techniques, sound artists can create installations that classify and respond to specific sound events, such as distinguishing between footsteps and bird chirping.
-
State-of-the-Art Methods: Popular machine learning libraries for audio recognition include:
-
Librosa: A Python library for music and audio analysis, providing features for audio preprocessing, feature extraction, and classification.
-
Tensorflow Audio: A library that extends TensorFlow with audio-specific functionalities, including audio preprocessing, feature extraction, and model building.
-
YAMNet: A deep learning model developed by Google that can recognize a wide range of audio events from 521 audio classes.
-
-
ML techniques explained
-
Convolutional Neural Networks (CNNs): CNNs are commonly used for audio recognition tasks such as speech recognition, music genre classification, and environmental sound classification. They excel at capturing local audio features by applying convolutional filters to spectrograms or raw audio waveforms.
-
Recurrent Neural Networks (RNNs): RNNs are suitable for tasks that require sequential modeling, such as speech recognition and music transcription. They can capture temporal dependencies in audio data by utilizing recurrent connections to maintain memory of past information.
-
Hidden Markov Models (HMMs): HMMs are statistical models often used in speech recognition. They represent audio as a sequence of states and use probability distributions to model the transitions between states.
-
Attention Mechanisms: Attention mechanisms help models focus on specific parts of an audio sequence, allowing them to capture important temporal dependencies. They have improved the performance of tasks like speech recognition and audio event detection.
-
Transformer Models: Transformer models, originally developed for natural language processing, have also been applied to audio recognition. They use self-attention mechanisms to capture long-range dependencies in audio signals, leading to state-of-the-art performance in tasks like music genre classification.
-
- Audio synthesis involves the generation or creation of new audio signals using various algorithms and techniques.
- Examples in Sound Art: Sound artists can use audio synthesis to create unique and expressive sounds for their installations or compositions. They can generate abstract textures, atmospheric tones, or even imitate real-world sounds using synthesis techniques. For example, a sound artist may use granular synthesis to manipulate and transform recorded environmental sounds into a mesmerizing sonic landscape.
- State-of-the-Art Methods: Notable machine learning libraries and models for audio synthesis include:
- Magenta: An open-source project by Google that explores the intersection of machine learning and music generation. It provides models like MusicVAE and NSynth for generating new musical compositions and timbres.
- WaveGAN: A Generative Adversarial Network (GAN)-based model that can generate high-quality audio samples. It can learn and mimic the distribution of real audio data to create realistic and diverse sounds.
- ML techniques explained
- Generative Adversarial Networks (GANs): GANs can be used for audio synthesis tasks, such as generating realistic instrument sounds or speech. The generator network learns to produce audio samples that are indistinguishable from real audio, while the discriminator network learns to differentiate between real and generated samples.
- Variational Autoencoders (VAEs): VAEs are used for learning latent representations of audio data. They can generate new audio samples by sampling from the learned latent space, enabling tasks like music generation and voice cloning.
- WaveNet: WaveNet is a deep generative model for audio synthesis. It models the raw waveform directly, allowing high-quality audio generation with realistic details and nuances.
- Audio transformation involves modifying or manipulating audio signals to achieve desired effects or transformations.
- Examples in Sound Art: Sound artists can use audio transformation techniques to alter and shape sound elements in their compositions or installations. They can apply effects like time stretching, pitch shifting, or spectral manipulation to create unique sonic experiences. For instance, a sound artist might use time stretching to slow down or stretch out a sound sample, creating an ethereal and atmospheric effect.
- State-of-the-Art Methods: Some machine learning libraries and models for audio transformation include:
- Essentia: An open-source library for audio analysis and transformation, providing a wide range of audio processing algorithms and tools.
- Spleeter: A library developed by Deezer that uses deep learning to perform source separation, allowing users to separate vocals, drums, and other sound sources from mixed audio.
- ML techniques explained
-
Time-domain and Frequency-domain Manipulation: Machine learning techniques can be used to manipulate audio signals in both the time domain and frequency domain.
- Time Stretching and Pitch Shifting: These techniques modify the speed and pitch of audio signals without changing the overall content. They are commonly used in audio editing and music production.
- Spectral Processing: Spectral processing techniques operate on the frequency domain representation of audio signals. They can be used for tasks like noise reduction, audio effects, and source separation.
-
Source Separation: Source separation techniques aim to separate individual sound sources from a mixture, such as isolating vocals from music. Deep learning models, including recurrent neural networks and convolutional neural networks, have been utilized to improve the accuracy and quality of source separation.
-
Audio Style Transfer: Style transfer techniques allow for the transformation of audio signals into different styles or genres. By leveraging machine learning models, it is possible to modify the characteristics of audio, such as changing the timbre or adding specific musical attributes.
-
- Contents
- Read/Write
- Transformations - General DSP
- Feature extraction
- Data augmentation
- Speech Processing
- Environmental Sounds
- Perceptial Models - Auditory Models
- Source Separation
- Music Information Retrieval
- Deep Learning
- Symbolic Music - MIDI - Musicology
- Realtime applications
- Web - Audio
- Audio related APIs and Datasets
- Wrappers for Audio Plugins
- Total number of packages: 66
- audiolazy
๐ฆ - Expressive Digital Signal Processing (DSP) package for Python.
- audioread
๐ฆ - Cross-library (GStreamer + Core Audio + MAD + FFmpeg) audio decoding.
- mutagen
๐ฆ - Reads and writes all kind of audio metadata for various formats.
- pyAV
- PyAV is a Pythonic binding for FFmpeg or Libav.
- (Py)Soundfile
๐ฆ - Library based on libsndfile, CFFI, and NumPy.
- pySox
๐ฆ - Wrapper for sox.
- stempeg
๐ฆ - read/write of STEMS multistream audio.
- tinytag
๐ฆ - reading music meta data of MP3, OGG, FLAC and Wave files.
- acoustics
๐ฆ - useful tools for acousticians.
- AudioTK
- DSP filter toolbox (lots of filters).
- AudioTSM
๐ฆ - real-time audio time-scale modification procedures.
- Gammatone
- Gammatone filterbank implementation.
- pyFFTW
๐ฆ - Wrapper for FFTW(3).
- NSGT
๐ฆ - Non-stationary gabor transform, constant-q.
- matchering
๐ฆ - Automated reference audio mastering.
- MDCT
๐ฆ - MDCT transform.
- pydub
๐ฆ - Manipulate audio with a simple and easy high level interface.
- pytftb
- Implementation of the MATLAB Time-Frequency Toolbox.
- pyroomacoustics
๐ฆ - Room Acoustics Simulation (RIR generator)
- PyRubberband
๐ฆ - Wrapper for rubberband to do pitch-shifting and time-stretching.
- PyWavelets
๐ฆ - Discrete Wavelet Transform in Python.
- Resampy
๐ฆ - Sample rate conversion.
- SFS-Python
๐ฆ - Sound Field Synthesis Toolbox.
- sound_field_analysis
๐ฆ - Analyze, visualize and process sound field data recorded by spherical microphone arrays.
- STFT
๐ฆ - Standalone package for Short-Time Fourier Transform.
- aubio
๐ฆ - Feature extractor, written in C, Python interface.
- audioFlux
๐ฆ - A library for audio and music analysis, feature extraction.
- audiolazy
๐ฆ - Realtime Audio Processing lib, general purpose.
- essentia
- Music related low level and high level feature extractor, C++ based, includes Python bindings.
- python_speech_features
๐ฆ - Common speech features for ASR.
- pyYAAFE
- Python bindings for YAAFE feature extractor.
- speechpy
๐ฆ - Library for Speech Processing and Recognition, mostly feature extraction for now.
- spafe
๐ฆ - Python library for features extraction from audio files.
- audiomentations
๐ฆ - Audio Data Augmentation.
- muda
๐ฆ - Musical Data Augmentation.
- pydiogment
๐ฆ - Audio Data Augmentation.
- aeneas
๐ฆ - Forced aligner, based on MFCC+DTW, 35+ languages.
- deepspeech
๐ฆ - Pretrained automatic speech recognition.
- gentle
- Forced-aligner built on Kaldi.
- Parselmouth
๐ฆ - Python interface to the Praat phonetics and speech analysis, synthesis, and manipulation software.
- persephone
๐ฆ - Automatic phoneme transcription tool.
- pyannote.audio
๐ฆ - Neural building blocks for speaker diarization.
- pyAudioAnalysisยฒ
๐ฆ - Feature Extraction, Classification, Diarization.
- py-webrtcvad
๐ฆ - Interface to the WebRTC Voice Activity Detector.
- pypesq
- Wrapper for the PESQ score calculation.
- pystoi
๐ฆ - Short Term Objective Intelligibility measure (STOI).
- PyWorldVocoder
- Wrapper for Morise's World Vocoder.
- Montreal Forced Aligner
- Forced aligner, based on Kaldi (HMM), English (others can be trained).
- SIDEKIT ๐ฆ - Speaker and Language recognition.
- SpeechRecognition
๐ฆ - Wrapper for several ASR engines and APIs, online and offline.
- cochlea
๐ฆ - Inner ear models.
- Brian2
๐ฆ - Spiking neural networks simulator, includes cochlea model.
- Loudness
- Perceived loudness, includes Zwicker, Moore/Glasberg model.
- pyloudnorm
- Audio loudness meter and normalization, implements ITU-R BS.1770-4.
- Sound Field Synthesis Toolbox
๐ฆ - Sound Field Synthesis Toolbox.
- commonfate
๐ฆ - Common Fate Model and Transform.
- NTFLib
- Sparse Beta-Divergence Tensor Factorization.
- NUSSL
๐ฆ - Holistic source separation framework including DSP methods and deep learning methods.
- NIMFA
๐ฆ - Several flavors of non-negative-matrix factorization.
- Catchy
- Corpus Analysis Tools for Computational Hook Discovery.
- Madmom
๐ฆ - MIR packages with strong focus on beat detection, onset detection and chord recognition.
- mir_eval
๐ฆ - Common scores for various MIR tasks. Also includes bss_eval implementation.
- msaf
๐ฆ - Music Structure Analysis Framework.
- librosa
๐ฆ - General audio and music analysis.
- Kapre
๐ฆ - Keras Audio Preprocessors
- TorchAudio
- PyTorch Audio Loaders
- nnAudio
๐ฆ - Accelerated audio processing using 1D convolution networks in PyTorch.
- Music21
๐ฆ - Toolkit for Computer-Aided Musicology.
- Mido
๐ฆ - Realtime MIDI wrapper.
- mingus
๐ฆ - Advanced music theory and notation package with MIDI file and playback support.
- Pretty-MIDI
๐ฆ - Utility functions for handling MIDI data in a nice/intuitive way.
- Jupylet
- Subtractive, additive, FM, and sample-based sound synthesis.
- PYO
- Realtime audio dsp engine.
- python-sounddevice
๐ฆ - PortAudio wrapper providing realtime audio I/O with NumPy.
- ReTiSAR
- Binarual rendering of streamed or IR-based high-order spherical microphone array signals.
- TimeSide (Beta)
- high level audio analysis, imaging, transcoding, streaming and labelling.
- beets
๐ฆ - Music library manager and MusicBrainz tagger.
- musdb
๐ฆ - Parse and process the MUSDB18 dataset.
- medleydb
- Parse medleydb audio + annotations.
- Soundcloud API
๐ฆ - Wrapper for Soundcloud API.
- Youtube-Downloader
๐ฆ - Download youtube videos (and the audio).
- audiomate
๐ฆ - Loading different types of audio datasets.
- mirdata
๐ฆ - Common loaders for Music Information Retrieval (MIR) datasets.
- VamPy Host ๐ฆ - Interface compiled vamp plugins.