Git Product home page Git Product logo

roc-utils's Introduction

roc-utils

image License CodeFactor DeepSource

This Python package provides tools to compute and visualize ROC curves, which are used to graphically assess the diagnostic ability of binary classifiers.

Use roc_utils to perform ROC analyses, including the calculation of the ROC-AUC (the area under the ROC curve) and the identification of optimal classification thresholds for different objective functions. In addition, it is possible to compute mean, tolerance interval (TI) and confidence interval (CI) for a set of (related) ROC curves. Finally, error bounds can be estimated and visualized by means of boostrap sampling.

Exemplary plots generated with roc_utils

Installation:

pip install roc-utils

Use the following commands for a quick verification of the installation.

python -c "import roc_utils; print(roc_utils.__version__)"
python -c "import roc_utils; roc_utils.demo_bootstrap()"

Usage:

See examples/tutorial.ipynb for step-by-step introduction.

import numpy as np
import matplotlib.pyplot as plt
import roc_utils as ru

# Construct a binary classification problem
x, y = ru.demo_sample_data(n1=300, mu1=0.0, std1=0.5,
                           n2=300, mu2=1.0, std2=0.7)

# Compute the ROC curve...
pos_label = True
roc = ru.compute_roc(X=x, y=y, pos_label=pos_label)

# ...and visualize it
ru.plot_roc(roc, label="Sample data", color="red")
plt.show()

# To perform a ROC analysis using bootstrapping
n_samples = 20
ru.plot_roc_bootstrap(X=x, y=y, pos_label=pos_label,
                      n_bootstrap=n_samples,
                      title="Bootstrap demo");
plt.show()

Build from source:

To fetch the project and run the tests or examples:

git clone https://github.com/hirsch-lab/roc-utils.git
cd roc-utils
python tests/test_all.py
python examples/examples.py

To create distribution packages (a source archive and a wheel):

python setup.py sdist bdist_wheel

To install the newly created Python package from the source archive:

pip uninstall roc-utils
pip cache remove roc_utils
pip install dist/roc_utils*.tar.gz

# Verify installation
python -c "import roc_utils; print(roc_utils.__version__)"
python -c "import roc_utils; roc_utils.demo_bootstrap()"

roc-utils's People

Contributors

normanius avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

roc-utils's Issues

Option: different confidence interval

Until now you can not specify a different confidence Interval for the mean ROC curve, 95% is fixed, in future it would be great if there is an option to change it!

In the function plot_mean_roc():

if show_ci:
# 95% confidence interval
tpr_std = np.std(ret_mean.tpr_all, axis=0, ddof=1)
tpr_lower = ret_mean.tpr - 1.96 * tpr_std / np.sqrt(n_samples)
tpr_upper = ret_mean.tpr + 1.96 * tpr_std / np.sqrt(n_samples)
label_ci = "95% CI of mean curve" if show_details else None
ax.fill_between(ret_mean.fpr, tpr_lower, tpr_upper,
color=color, alpha=.3,
label=label_ci,
zorder=zorder)

AUC 95% interval obtained by bootstrapping seems off

I can successfully plot the confidence band around an ROC curve and the interval is plausible.

However, the corresponding confidence interval on the AUC is questionable. Looking at the shaded area, I expect to have an AUC interval of something like [0.6, 1]. Is the CI to be expected?

AUC 95% CI: [[0.89360685 0.89723678]]

image

MWE:

import matplotlib.pyplot as plt
import roc_utils as ru

preds = np.array([0.0184128433465957, 0.012704842723906, 0.6588426828384399, 0.3824790120124817, 0.0752698183059692, 0.0284929275512695, 0.0939247459173202, 0.0913288220763206, 0.1392715573310852, 0.8083584308624268, 0.5678479075431824, 0.7689659595489502, 0.6002992391586304, 0.4453626573085785])
targets = np.array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0])

results = ru.plot_roc_bootstrap(X=preds, y=targets, pos_label=1,
                      n_bootstrap=9999,
)
plt.show()
print("AUC 95% CI:", results["auc95_ci"])

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.