claritychallenge / clarity Goto Github PK
View Code? Open in Web Editor NEWClarity Challenge toolkit - software for building Clarity Challenge systems
Home Page: https://claritychallenge.github.io/clarity
License: MIT License
Clarity Challenge toolkit - software for building Clarity Challenge systems
Home Page: https://claritychallenge.github.io/clarity
License: MIT License
Having Clarity available on PyPI will be useful and ease installation for participants.
This requires the following actions...
PRs #50 / #57 /#60 / #61 / #62 addressed the need for regression tests (see #48).
Now that these are in place it is possible to start introducing finer grained unit-tests across the codebase. This will provide immediate benefits...
This issue serves as an "Epic" for recording progress of tests written across modules/sub-modules Individual issue and pull-requests can be added against each module/sub-module for reference.
data/demo_data.py
data/HOA_tools_cec2.py
data/scene_builder_cec1.py
data/scene_render_cec1.py
data/scene_render_cec2.py
data/utils.py
#88dataset/cec1_dataset.py
engine/losses.py
engone/system.py
recipes/cad1/task1
(partially complete)recipes/cad1/task2
(partially complete)recipes/cec1/baseline
recipes/cec1/data_preparation/prepare_cec1_data.py
recipes/cec1/e009_sheffield
recipes/cec2/baseline
recipes/cec2/data_preparation
recipes/cpc1/baseline
recipes/cpc1/e029_sheffield
recipes/cpc1/e032_sheffield
recipes/cpc1/test_listener_responses
recipes/cpc2
(partially complete)recipes/icassp_2023
enhancer/compressor.py
enhancer/nalr.py
enhancer/dnn
enhancer/dnn/mc_conv_tasnet.py
enhancer/dsp
enhancer/dsp/filter.py
enhancer/gha
enhancer/gha/audiogram.py
enhancer/gha/gainrule_camfit.py
enhancer/gha/gha_interface.py
enhancer/gha/gha_utils.py
evaluator/haspi
evaluator/haspi/eb.py
evaluator/haspi/ebm.py
evaluator/haspi/haspi.py
evaluator/haspi/ip.py
evaluator/hasqi
evaluator/hasqi/hasqi.py
evaluator/haaqi
evaluator/haaqi/haaqi.py
evaluator/mbstoi
evaluator/mbstoi/mbstoi.py
evaluator/mbstoi/mbstoi_utils.py
evaluator/msbg
evaluator/msbg/audiogram.py
evaluator/msbg/cochlea.py
evaluator/msbg/masbg.py
evaluator/msbg/masbg_utils.py
evaluator/msbg/smearing.py
predictor/torch_msbg.py
predictor/torch_stoi.py
Pre-commit is using Black 19.10b0 from 2019 (actually a beta version). The latest version is 22.6.0 and a lot of issues have been fixed since 19.10b0 - particularly with handling parentheses and docstrings.
Moving to the latest version will lead to the reformatting of 25 files.
Develop unit tests for the data module (see #80 for overview).
Specific modules within dataset
are...
dataset/cec1_dataset.py
Describe the bug
pytest-cov coverage report only considering source files that are import in existing tests
To Reproduce
run pytest --cov=clarity
Expected behavior
Report to list many source files with 0% coverage given that we have a lot of untested code.
Fixable with simple change to coverage to coverage config in setup.cfg
Add a GitHub action that runs the pre-commit linting hooks when running pull requests into main.
The license applied on GitHub (Creative Commons Attribution Share Alike 4.0 International) is out of sync with that specified in the Python configuration file setup.py
which lists MIT.
The later is a more open and permissive license that we will apply to the project.
Update LICENSE
to MIT.
Is your feature request related to a problem? Please describe.
Before #42 can be implemented we need to run pylint against the existing code base and resolve all issues.
Describe the solution you'd like
Changes should be made on a separate branch and commited with --ignore-revs
to avoid mis-attributing the "blame" (see example in Introducing Black to your project - Black 22.6.0 documentation).
Further a custom .pylintrc
file will need introducing that configures pylint and the rules that are to be included/excluded (e.g. we may wish to ignore may invalid-name
errors).
The repository requires a code of conduct for contributors etc.
MSBG code (i.e.clarity/evaluator/msbg) being worked on
Currently the nomenclature of class, function and variables is a bit of a mixed bag.
It would improve legibility if they were all PEP8 compliant and had meaningful names (although there are some cases where exceptions may be considered reasonable such as implementations of algorithms).
To which end this issue should address changing as many class, function and variable names as possible to be PEP8 compliant and disable checks for those that are considered not worthy of bringing into line.
Currently the existing .pylintrc
excludes these checks from being made because C0103: invalid-name is disabled. Excluding this temporarily and running pylint
against the clarity
directory shows the invalid-name
errors detailed in this gist (including it in the body of this issue exceeded the character limit!).
EDIT : Many changes can be made simply and consistently across the code base by virtue of using an IDE that supports Language Server Protocol which supports the "renane" directive which will Rename: project-wide rename of a symbol..
Personally I use Emacs and lsp-mode and so can M-x lsp-rename
when the cursor is on a variable/function/class and rename variables.
As the Cadenza challenge is up-coming it would be appropriate to include details on the front page as well as the logo...
A first draft might be...
We are organising a series of machine learning challenges to enhance hearing-aid signal processing and to better predict how people perceive speech-in-noise (Clarity) and speech-in-music (Cadenza). For further details of the Clarity Project visit the Clarity project website, and for details of our latest Clarity challenges visit our challenge documentation site. You can contact the Clarity Team by email at [email protected]. For further details of the Cadenza Project visit the Cadenza project website, and to find out about the latest Cadenza challenges join the Cadenza Challenge Group.
In this repository, you will find code to support all Clarity and Cadenza Challenges, including baselines, toolkits, and systems from participants. We encourage you to make your system/model open source and contribute to this repository.
Clarity is available on the Python Package Index (PyPI) to install create and/or
activate a virtual environment and then use pip
to install.
conda create --name clarity python=3.8
conda activate clarity
pip install pyclarity
# First clone the repo
git clone https://github.com/claritychallenge/clarity.git
cd clarity
# Second create & activate environment with conda, see https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html
conda create --name clarity python=3.8
conda activate clarity
# Last install with pip
pip install -e .
Alternatively pip
allows you to install packages from GitHub sources directly. The following will install the current
main
branch.
pip install -e git+https://github.com/claritychallenge/clarity.git@main
Current challenge(s)
Upcoming challenges
Previous challenges
We provide also a number of tools in this repository:
evaluate.py
.enhance.py
.enhance.py
.In addition, differentiable approximation to some tools are provided:
Typehints are useful to use in Python, despite it being a dynamically typed language.
Much of the code has typehints in place for parameters and some for the return values but it is currently incomplete. This should be remedied and the code linted using MyPy which is a static type checker.
Currently on the most complete branch/commit ( #7c0a505 ) there are 48 errors across 22 files (out of 42 source files), although most pertain to the imported modules and so can be handled as described in Running mypy and managing imports...
❱ mypy clarity
clarity/data/demo_data.py:4: error: Skipping analyzing "gdown": module is installed, but missing library stubs or py.typed marker
clarity/evaluator/msbg/smearing.py:156: error: Incompatible types in assignment (expression has type "None", variable has type "ndarray[Any, Any]")
clarity/evaluator/msbg/smearing.py:157: error: Incompatible types in assignment (expression has type "None", variable has type "ndarray[Any, Any]")
clarity/evaluator/msbg/smearing.py:158: error: Incompatible types in assignment (expression has type "None", variable has type "ndarray[Any, Any]")
clarity/evaluator/msbg/smearing.py:159: error: Incompatible types in assignment (expression has type "None", variable has type "ndarray[Any, Any]")
clarity/evaluator/msbg/msbg_utils.py:7: error: Skipping analyzing "scipy": module is installed, but missing library stubs or py.typed marker
clarity/evaluator/msbg/msbg_utils.py:8: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/evaluator/msbg/msbg_utils.py:9: error: Skipping analyzing "soundfile": module is installed, but missing library stubs or py.typed marker
clarity/evaluator/mbstoi/mbstoi_utils.py:4: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/evaluator/haspi/ebm.py:4: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/evaluator/haspi/eb.py:2: error: Skipping analyzing "numba": module is installed, but missing library stubs or py.typed marker
clarity/evaluator/haspi/eb.py:3: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/enhancer/nalr.py:4: error: Skipping analyzing "scipy": module is installed, but missing library stubs or py.typed marker
clarity/enhancer/nalr.py:5: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/enhancer/gha/gainrule_camfit.py:5: error: Skipping analyzing "scipy.interpolate": module is installed, but missing library stubs or py.typed marker
clarity/data/utils.py:4: error: Skipping analyzing "scipy": module is installed, but missing library stubs or py.typed marker
clarity/data/utils.py:5: error: Skipping analyzing "scipy.io": module is installed, but missing library stubs or py.typed marker
clarity/data/scene_builder_cec2.py:11: error: Skipping analyzing "tqdm": module is installed, but missing library stubs or py.typed marker
clarity/data/scene_builder_cec2.py:18: error: Module has no attribute "c_make_encoder"
clarity/data/scene_builder_cec2.py:27: error: Module has no attribute "float"
clarity/data/HOA_tools_cec2.py:5: error: Skipping analyzing "numba": module is installed, but missing library stubs or py.typed marker
clarity/data/HOA_tools_cec2.py:5: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
clarity/data/HOA_tools_cec2.py:6: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/data/HOA_tools_cec2.py:7: error: Skipping analyzing "scipy.spatial.transform": module is installed, but missing library stubs or py.typed marker
clarity/data/HOA_tools_cec2.py:8: error: Skipping analyzing "scipy.special": module is installed, but missing library stubs or py.typed marker
clarity/predictor/torch_stoi.py:9: error: Skipping analyzing "torchaudio": module is installed, but missing library stubs or py.typed marker
clarity/predictor/torch_stoi.py:10: error: Skipping analyzing "pystoi.stoi": module is installed, but missing library stubs or py.typed marker
clarity/predictor/torch_stoi.py:11: error: Skipping analyzing "pystoi.utils": module is installed, but missing library stubs or py.typed marker
clarity/predictor/torch_stoi.py:179: error: Item "None" of "Optional[Tensor]" has no attribute "mean"
clarity/predictor/torch_msbg.py:11: error: Skipping analyzing "torchaudio": module is installed, but missing library stubs or py.typed marker
clarity/predictor/torch_msbg.py:12: error: Skipping analyzing "scipy.fftpack": module is installed, but missing library stubs or py.typed marker
clarity/predictor/torch_msbg.py:13: error: Skipping analyzing "scipy.interpolate": module is installed, but missing library stubs or py.typed marker
clarity/predictor/torch_msbg.py:14: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/evaluator/msbg/cochlea.py:4: error: Skipping analyzing "scipy": module is installed, but missing library stubs or py.typed marker
clarity/evaluator/mbstoi/mbstoi.py:5: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/enhancer/gha/gha_utils.py:6: error: Skipping analyzing "scipy.interpolate": module is installed, but missing library stubs or py.typed marker
clarity/engine/system.py:5: error: Cannot find implementation or library stub for module named "pytorch_lightning"
clarity/dataset/cec1_dataset.py:4: error: Skipping analyzing "librosa": module is installed, but missing library stubs or py.typed marker
clarity/dataset/cec1_dataset.py:7: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/dataset/cec1_dataset.py:8: error: Skipping analyzing "soundfile": module is installed, but missing library stubs or py.typed marker
clarity/data/scene_renderer_cec2.py:7: error: Skipping analyzing "librosa": module is installed, but missing library stubs or py.typed marker
clarity/data/scene_renderer_cec2.py:12: error: Skipping analyzing "scipy.io": module is installed, but missing library stubs or py.typed marker
clarity/data/scene_renderer_cec2.py:13: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/data/scene_renderer_cec2.py:14: error: Skipping analyzing "tqdm": module is installed, but missing library stubs or py.typed marker
clarity/data/scene_renderer_cec1.py:7: error: Skipping analyzing "soundfile": module is installed, but missing library stubs or py.typed marker
clarity/data/scene_renderer_cec1.py:8: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/evaluator/msbg/msbg.py:5: error: Skipping analyzing "scipy": module is installed, but missing library stubs or py.typed marker
clarity/evaluator/msbg/msbg.py:6: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/enhancer/gha/gha_interface.py:8: error: Skipping analyzing "soundfile": module is installed, but missing library stubs or py.typed marker
Found 48 errors in 22 files (checked 42 source files)
We may wish to apply the same standards to tests/*
...
❱ mypy tests
clarity/evaluator/haspi/ebm.py:4: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/evaluator/haspi/eb.py:2: error: Skipping analyzing "numba": module is installed, but missing library stubs or py.typed marker
clarity/evaluator/haspi/eb.py:3: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/data/utils.py:4: error: Skipping analyzing "scipy": module is installed, but missing library stubs or py.typed marker
clarity/data/utils.py:5: error: Skipping analyzing "scipy.io": module is installed, but missing library stubs or py.typed marker
clarity/data/HOA_tools_cec2.py:5: error: Skipping analyzing "numba": module is installed, but missing library stubs or py.typed marker
clarity/data/HOA_tools_cec2.py:6: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/data/HOA_tools_cec2.py:7: error: Skipping analyzing "scipy.spatial.transform": module is installed, but missing library stubs or py.typed marker
clarity/data/HOA_tools_cec2.py:8: error: Skipping analyzing "scipy.special": module is installed, but missing library stubs or py.typed marker
clarity/enhancer/nalr.py:4: error: Skipping analyzing "scipy": module is installed, but missing library stubs or py.typed marker
clarity/enhancer/nalr.py:5: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/data/scene_renderer_cec2.py:7: error: Skipping analyzing "librosa": module is installed, but missing library stubs or py.typed marker
clarity/data/scene_renderer_cec2.py:12: error: Skipping analyzing "scipy.io": module is installed, but missing library stubs or py.typed marker
clarity/data/scene_renderer_cec2.py:13: error: Skipping analyzing "scipy.signal": module is installed, but missing library stubs or py.typed marker
clarity/data/scene_renderer_cec2.py:14: error: Skipping analyzing "tqdm": module is installed, but missing library stubs or py.typed marker
tests/test_full_pipeline.py:9: error: Skipping analyzing "scipy.io": module is installed, but missing library stubs or py.typed marker
tests/test_full_pipeline.py:9: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 16 errors in 7 files (checked 4 source files)
Once these have been addressed we can enable the mypy
automated static type checking/linting available under pre-commit
.
Notebooks have problems in revision control as they contain a lot of non-content data, i.e. outputs and metadata that change each time the notebook is run. This can be fixed using pre-commit hooks.
Develop unit tests for the data module (see #80 for overview).
Specific modules within enhancer
are...
enhancer/compressor.py
enhancer/nalr.py
enhancer/dnn
enhancer/dnn/mc_conv_tasnet.py
enhancer/dsp
enhancer/dsp/filter.py
enhancer/gha
enhancer/gha/audiogram.py
enhancer/gha/gainrule_camfit.py
enhancer/gha/gha_interface.py
enhancer/gha/gha_utils.py
Numba code is generating a warning about use of a deprecated type
Encountered the use of a type that is scheduled for deprecation: type 'reflected list'
This needs fixing to prevent errors arising when newer versions of numba are released.
Information on how to fix can be found here.
Hi,
Thanks for your fantastic works!
I'm trying to use the head rotation signal, but I find that the length of the audio files (e.g. mixed_CH1.wav, target_CH1.wav) and head rotation file (hr.wav) is different. The head rotation signal is 255 points shorter than audio files. For example, S06001 in dev dataset, the length of S06001_hr.wav is 287735 and the length of S06001_mix_CH0.wav is 287990.
My question is how should I deal with the misalignment, should I neglect 255 points in the beginning or the end of the audio files?
Best regards,
Chengwei Ouyang
The last Clarity Challenge included links to two Jupyter notebooks hosted on Google Colab (see Tutorials).
Including these in the code base here so that they and other examples/tutorials are version controlled too is desirable.
Hi, Thanks so much for organizing the challenge, and providing detailed docuement.
I am very interested on it and have a few questions below:
1). There is 5-ms latency limitation for the enhancement model. Will the frame length be counted in latency? I read some papers in CEC1. One team (paper) used a 32-ms window. I am wondering if this long window size (beyond 5ms) is allowed in CEC2?
2). Regarding on the use of external data. I plan to use external noisy data to train a more robust denoising module. If I do this, seems I have to submit two systems, with using and without using external data. If I plan to submit entries for HASPI and listening tests, respectively, does this mean I need to submit four entries (2 tests X 2 data uses) in total?
3). Just one thing to confirm: Was ch0 (eardrum) signal also included in the release of CEC1 dataset?
I found ch0 was included in CEC1 data (written in document), but seems many submissions only used 6 channels (ch1~3 pairs), and no team used ch0. So just curious on it.
Again, thank you so much for your work and time!
Currently the Clarity Challenge focuses on the challenge itself.
It is desirable to have the code better documented including the following...
The documentation should use the GitHub Action - Sphinx Docs to GitHub Pages to build and deploy the documentation automatically.
Please add any other areas of documentation to this issue.
I notice that the amplification model (AudiometricFIR) doesn't take audiogram as inputs, so it seems that I have to train an individual amplification model for each listener, which is weird for me. Could you give me some suggestions on how to generalize the amplification model or I have to train 100 individual models for 100 listeners in CEC1 challenge? Thank you!
Develop unit tests for the data module (see #80 for overview).
Specific modules within engine
are...
engine/losses.py
engone/system.py
PR test failing due to py is no longer being pulled in and installed as a required dependency of pytest-regtest. (related to this issue in pytest-regtest https://gitlab.com/uweschmitt/pytest-regtest/-/issues/19)
Migrate HASQI index from Matlab
Hi everyone,
Let me thank the organizers for this fantastic and interesting challenge!
When I check the scripts for calculating HASPI, I find some randomness in eb and ebm. As a result, the score for the same signals slightly fluctuated in some runs. Although the fluctuation is very small, it will be good to fix the seed.
Probably, I misunderstand something. Please check whether the fluctuation occurs in your environment. Thank you!
It is useful to have contribution guidelines for projects so that participants know what is expected of them when they consider contributing to the code base.
Describe the bug
Pinned version of matplotlib in dependencies messes up the colab environment.
No good reason to be pinning the version.
Starting a branch to do some final work on notebooks ahead of v0.1.0 release
Thanks for your fantastic works, and I have a question about the dataset. In my understanding, the purpose of this challenge is to get target_anechoic audio by mixed_CH1, mixed_CH2, mixed_CH3, which is the signal received by the front, mid, and rear microphones. My question is that what is the purpose of mixed_CH0 and target_CH0, which is close to eardrum and have a large difference with other channels.
The EarModel in evaluator/haspi/eb computes the center frequency using shift = 0.02
clarity/clarity/evaluator/haspi/eb.py
Lines 70 to 75 in 94e39d4
However, the function CenterFreq in evaluator/haspi/eb forces the parameter to be None to keep consistency with Matlab
clarity/clarity/evaluator/haspi/eb.py
Lines 180 to 233 in 94e39d4
What is the point of setting shift in center freq if it is never used?
Setting the value in the Ear Model gives the impression that a shift is applied, won't that confuse people while reading the code?
What would be the best way to clean this part of the code?
1- Changing shift to None in EarModel?
2- Removing shift parameter from CenterFreq?
3- Removing the shift=None
statement in CenterFreq?
The latest iteration of Python's setuptools deprecates the use of setup.py
except in some specific circumstances (see note).
Instead it uses a combination of pyproject.toml
and setup.cfg
to achieve the configuration of the package, its dependencies and usage.
The settings in setup.py
should be migrated to pyproject.toml
and setup.cfg
.
There are a number of deprecation warnings that have been captured by the test code that should be dealt with to avoid possible problems later.
Matlab version of NALR has 3 parameters, HL, nfir, fs
function [nalr,delay]=eb_NALR(HL,nfir,fsamp)
In Clarity, NALR is implemented as a Class in enhancer.nalr.
class NALR:
def __init__(self, nfir, fs):
[...]
def hl_interp(self, hl, cfs):
[...]
def build(self, HL, cfs):
HL = self.hl_interp(np.array(HL), np.array(cfs))
[...]
For the cases when cfs is not available and to match MATLAB's use of HAAQI and HASQI, it is needed to make cfs=None and add the if conditional
def build(self, HL, cfs=None):
if not cfs is None:
HL = self.hl_interp(np.array(HL), np.array(cfs))
[...]
Current regression test only covers CEC2 baseline, i.e. signal mixing, NALR and haspi. Need a more thorough set of tests covering CEC1, CPC1 and CEC2 code so that later refactoring can be performed securely.
Develop unit tests for the data module (see #80 for overview).
Specific modules within predictor
are...
predictor/torch_msbg.py
predictor/torch_stoi.py
Code was written for hydra 1.1. - Newer hydra 1.2 has introduced some breaking changes in the way that working directories operate. This will require changes to the config code to fix. Suggest for now that we pin the version to 1.1 and upgrade to hydra 1.2 in the next release.
Hi everyone,
first let me thank you for organizing this great challenge ! I am excited to participate.
I am looking at the data right now and in particular I am checking the delay (in terms of samples) between the left and right channels at each mic. Now it is not an easy task with Audacity to do this and I may be wrong.
But it seems to me that the delay for e.g S06001 in dev set is way off from what it should be from the metadata.
I count around 19-20 samples of delay when looking at S06001_target_anechoic_CH1.wav:
According to the metadata the head rotation should be:
"listener": {
"rotation": [
{
"sample": 90952.7142,
"angle": -125.8235
},
{
"sample": 100246.7142,
"angle": -112.0054
}
And so I would expect the delay to be much less. Probably I am doing something wrong but just in case, maybe it could be useful if you can double check.
Also one thing I expected is that regarding the anechoic response the two channels should be equal except for a small delay and a gain factor (including inverted polarity). But this is not the case. Is it because the scenes are generated via higher order ambisonics RIRs and then the actual array is approximated from such HOA rirs ?
EDIT: I think I've figured out, it should be the head related impulse response.
This makes difficult to check the delay manually on audacity. So it is very possible I am over-estimating it.
As we move to tagged versions and releases to PyPI we need to automate the flow of versioning the package to avoid the work of updating setup.cfg
and clarity/__init__.py
.
The versioneer package appears to offer a solution to this as it automates aligning the version with GitHub tags/releases.
After the initial 0.1.0
release has been made manually the process should shift to using versioneer
to handle versions.
The regression test fails when running the test tests/regression/test_predictors.py::test_torch_msbg_stoi
having a very small difference that may result from to randomness
=================================== FAILURES ===================================
_____________________________ test_torch_msbg_stoi _____________________________
regression test output differences for tests/regression/test_predictors.py::test_torch_msbg_stoi:
> --- current
> +++ tobe
> @@ -1,2 +1,2 @@
> -Torch MSBG STOILoss -0.46198, ESTOILoss -0.32999
> +Torch MSBG STOILoss -0.46198, ESTOILoss -0.33000
This results in PR failing some tests.
The problem can be solved by reruning the test manually one or 2 times.
Hello, I am a researcher on speech enhancement and I am new to the use of objective intelligibility measures in the area of hearing impairment. I was able to figure out how to use MBSTOI, but HASPI has multiple parameters that I am not sure how to specify.
What are the standard specifications used for HASPI? Specifically clarity.evaluator.haspi.haspi_v2.
https://github.com/claritychallenge/clarity/blob/main/clarity/evaluator/haspi/haspi.py
Here is my attempt making an example:
! pip3 install https://github.com/claritychallenge/clarity/archive/refs/heads/main.zip --quiet
from clarity.evaluator import haspi, mbstoi
import os
root_url = "https://github.com/microsoft/DNS-Challenge/raw/interspeech2020/master/datasets/test_set/synthetic/no_reverb"
clean_url = root_url + "/clean/clean_fileid_0.wav"
noisy_url = root_url + "/noisy/clnsp126_3Wjw0nadnM4_snr15_tl-22_fileid_0.wav"
if "clean.wav" not in os.listdir() or "noisy.wav" not in os.listdir():
os.system("wget -O clean.wav " + clean_url)
os.system("wget -O noisy.wav " + noisy_url)
import librosa
clean = librosa.load("clean.wav", sr=16000)[0]
noisy = librosa.load("noisy.wav", sr=16000)[0]
Y1 = mbstoi.mbstoi(clean, clean, noisy, noisy, fs_signal=16000)
Y2, _ = haspi.haspi_v2(clean, 16000, noisy, 16000, [250, 500, 1000, 2000, 4000, 6000])
print("MBSTOI ", Y1, "\n", "HASPI ", Y2, sep="")
However, the observed HASPI value is really small. Is this correct usage? What are the standard ways to specify the evaluation?
Docstrings are currently written in Python's ReSructured Text.
An easier to read and more flexible docstring style is the Google Style docstrings style which Sphinx supports additional sections and renders with Sphinx via the napoleon Sphinx extension.
In particular including examples is very useful.
Migrate HAAQI from Matlab
Is your feature request related to a problem? Please describe.
Making pyclarity
available on PyPI would make installation easier, particularly as development progresses.
Describe the solution you'd like
Automated publishing of commits tagged with versions to PyPI should occur. This is possible with pypi-publish GitHub Action.
Describe alternatives you've considered
Manual releases are also possible, but should not be needed if the process can be automated.
Additional context
N/A.
Develop unit tests for the data module (see #80 for overview).
Specific modules within evaluator
are...
evaluator/haspi
evaluator/haspi/eb.py
evaluator/haspi/ebm.py
evaluator/haspi/ip.py
evaluator/haspi/haspi.py
evaluator/hasqi/hasqi.py
evaluator/haaqi/haaqi.py
evaluator/mbstoi
evaluator/mbstoi/mbstoi.py
evaluator/mbstoi/mbstoi_utils.py
evaluator/msbg
evaluator/msbg/audiogram.py
No change required.evaluator/msbg/cochlea.py
evaluator/msbg/msbg.py
evaluator/msbg/msbg_utils.py
evaluator/msbg/smearing.py
Is your feature request related to a problem? Please describe.
No problem, but as part of improving the code base we can incorporate pylint into the Continuous Integration/Development (CI/CD) pipeline as part of lint.yml
via pre-commit.
Describe the solution you'd like
Code should be linted with pylint
prior to commits (using pre-commit).
However, first all code needs linting (see separate issue #).
Hi, thanks so much for making this, I'd like to know how to run inference on a single wav file using this but I'm running into trouble with DictConfig on Colab taking inspiration from the infer.py
file. This is the function I wrote:
import copy
import json
import logging
import os
import hydra
import speechbrain as sb
import torch
from hyperpyyaml import load_hyperpyyaml
from omegaconf import DictConfig
from speechbrain.utils.distributed import run_on_main
from tqdm import tqdm
from transformer_cpc1_ensemble_decoder import S2STransformerBeamSearch
logger = logging.getLogger(__name__)
def init_asr(asr_config):
hparams_file, run_opts, overrides = sb.parse_arguments([asr_config])
with open(hparams_file) as fin:
hparams = load_hyperpyyaml(fin, overrides)
tokenizer = hparams["tokenizer"]
bos_index = hparams["bos_index"]
# We download the pretrained LM from HuggingFace (or elsewhere depending on
# the path given in the YAML file). The tokenizer is loaded at the same time.
run_on_main(hparams["pretrainer"].collect_files)
hparams["pretrainer"].load_collected(device=run_opts["device"])
asr_brain = ASR(
modules=hparams["modules"],
opt_class=hparams["Adam"],
hparams=hparams,
run_opts=run_opts,
checkpointer=hparams["checkpointer"],
)
asr_brain.init_evaluation()
return asr_brain, tokenizer, bos_index
def compute_uncertainty(left_proc_path, asr_model, bos_index, tokenizer):
wav_len = torch.tensor([1], dtype=torch.float32)
tokens_bos = torch.LongTensor([bos_index]).view(1, -1)
right_proc_path = left_proc_path.replace("left", "right")
left_proc_wav = sb.dataio.dataio.read_audio(left_proc_path).view(1, -1)
right_proc_wav = sb.dataio.dataio.read_audio(right_proc_path).view(1, -1)
left_uncertainty = asr_model.compute_uncertainty(left_proc_wav, wav_len, tokens_bos)
right_uncertainty = asr_model.compute_uncertainty(
right_proc_wav, wav_len, tokens_bos
)
conf = max(
left_uncertainty[0]["confidence"].detach().cpu().numpy(),
right_uncertainty[0]["confidence"].detach().cpu().numpy(),
)
neg_ent = -min(
left_uncertainty[0]["entropy"].detach().cpu().numpy(),
right_uncertainty[0]["entropy"].detach().cpu().numpy(),
)
return conf, neg_ent
# --- AND NOW MY NOT-WORKING CODE ----
@hydra.main(config_path=".", config_name="config")
def infer_single(wav_path):
cfg = DictConfig
if cfg.cpc1_track == "open":
track = "_indep"
elif cfg.cpc1_track == "closed":
track = ""
else:
logger.error("cpc1_track has to be closed or open")
print(dir(cfg))
asr_model, tokenizer, bos_index = init_asr(cfg.asr_config)
wav_len = torch.tensor([1], dtype=torch.float32)
tokens_bos = torch.LongTensor([bos_index]).view(1, -1)
wav = sb.dataio.dataio.read_audio(wav_path).view(1, -1)
uncertainty = asr_model.compute_uncertainty(wav, wav_len, tokens_bos)
return uncertainty
But running in colab:
wav_path = '/content/clarity/recipes/cpc1/e029_sheffield/test.wav'
uncertainty = infer_single(wav_path)
It's giving me this error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
[<ipython-input-29-43c026e4011b>](https://localhost:8080/#) in <module>
1 wav_path = '/content/clarity/recipes/cpc1/e029_sheffield/test.wav'
----> 2 uncertainty = infer_single(wav_path)
1 frames
[/usr/local/lib/python3.7/dist-packages/hydra/main.py](https://localhost:8080/#) in decorated_main(cfg_passthrough)
77 def decorated_main(cfg_passthrough: Optional[DictConfig] = None) -> Any:
78 if cfg_passthrough is not None:
---> 79 return task_function(cfg_passthrough)
80 else:
81 args_parser = get_args_parser()
[<ipython-input-28-f4867e54a086>](https://localhost:8080/#) in infer_single(wav_path)
183
184 cfg = DictConfig
--> 185 if cfg.cpc1_track == "open":
186 track = "_indep"
187 elif cfg.cpc1_track == "closed":
AttributeError: type object 'DictConfig' has no attribute 'cpc1_track'
I've tried researching docs for DictConfig from Omegaconf but I'm very confused what it actually does and am having a really hard time writing this function. Would love any help whatsoever!
Having issue templates for participants to use when reporting bugs or feature requests will be useful in helping obtain useful information up front that aids in resolving the problem or implementing the feature.
Possible templates include...
There may be more, please add them if you think there are other scenarios that may be useful.
A repository of templates for issue reporting can be found at stevemao/github-issue-templates: A collection of GitHub issue and pull request templates.
When I train the e009_sheffield model, both train loss and val loss (SNRLoss) will converge on 0.0043, which means all outputs for any inputs will be 0. When checking the train dataset, I find that the model using '_target_anechoic.wav' as ground truth, but the '_target_anechoic.wav' has about 20ms latency compared with both '_mix_CH1.wav' and '_target_CH1.wav'. In this way, the model is expected to predict the future information, which is impossible. If I change the ground truth from '_target_anechoic.wav' to '_target_CH1.wav', the problem is solved and the model can converge normally. Could you give me some advice on why the appoximate 20ms latency is exising between 'target_CH1.wav' and '_target_anechoic.wav' and how can I solve it? Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.