Virny is a Python library for in-depth profiling of model performance across overall and disparity dimensions. In addition to its metric computation capabilities, the library provides an interactive tool called VirnyView to streamline responsible model selection and generate nutritional labels for ML models. The Virny library was developed based on three fundamental principles:

1. easy extensibility of model analysis capabilities;

2. compatibility to user-defined/custom datasets and model types;

3. simple composition of disparity metrics based on the context of use.

Virny decouples model auditing into several stages, including: subgroup metric computation, disparity metric composition, and metric visualization. This gives data scientists more control and flexibility to use the library for model development and monitoring post-deployment.

For quickstart, look at use case examples, an interactive demo, and a demonstrative Jupyter notebook.

Virny supports Python 3.8 and 3.9 and can be installed with pip:

pip install virny

• Introduction
• API Reference
• Use Case Examples
• Interactive Demo

• Entire pipeline for profiling model accuracy, stability, uncertainty, and fairness
• Ability to analyze non-binary sensitive attributes and their intersections
• Compatibility with pre-, in-, and post-processors for fairness enhancement from AIF360
• Convenient metric computation interfaces: an interface for multiple models, an interface for multiple test sets, and an interface for saving results into a user-defined database
• An error_analysis computation mode to analyze model stability and confidence for correct and incorrect prodictions broken down by groups
• Metric static and interactive visualizations
• Data loaders with subsampling for popular fair-ML benchmark datasets
• User-friendly parameters input via config yaml files
• Check out our documentation for a comprehensive overview

The software framework decouples the process of model profiling into several stages, including subgroup metric computation, disparity metric composition, and metric visualization. This separation empowers data scientists with greater control and flexibility in employing the library, both during model development and for post-deployment monitoring. The above figure demonstrates how the library constructs a pipeline for model analysis. Inputs to a user interface are shown in green, pipeline stages are shown in blue, and the output of each stage is shown in purple.

Virny is free and open-source software licensed under the 3-clause BSD license.