This repo is a collection of awesome papers, codes, books, and blogs about Uncertainty and Deep learning. Feel free to star and fork.

If you think that we miss a paper, please open a pull request or send a message on the corresponding GitHub discussion. Tell us where the article was published and when, and send us GitHub and ArXiv links if they are available.

We are also open to any ideas for improvements!

• Awesome Uncertainty in Deep learning
• Papers
  • Surveys
  • Theory
  • Bayesian-Methods
  • Ensemble-Methods
  • Sampling/Dropout-based-Methods
  • Post-hoc-Methods/Auxiliary-Networks
  • Data-augmentation/Generation-based-methods
  • Output-Space-Modeling/Evidential-deep-learning
  • Deterministic-Uncertainty-Methods
  • Quantile-Regression/Predicted-Intervals
  • Conformal Predictions
  • Calibration/Evaluation-Metrics
  • Applications
    • Classification and Semantic-Segmentation
    • Regression
    • Anomaly-detection, Out-of-Distribution-Detection and Failure detection
    • Object detection
    • Domain adaptation
    • Semi-supervised
    • Natural Language Processing
    • Others
• Datasets and Benchmarks
• Libraries
  • Python
  • PyTorch
  • JAX
  • TensorFlow
• Lectures and tutorials
• Books
• Other Resources

Conference

• A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications [AISafety Workshop 2020]

Journal

• A survey of uncertainty in deep neural networks [Artificial Intelligence Review 2023] - [GitHub]
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation [TMLR2023]
• A Survey on Uncertainty Estimation in Deep Learning Classification Systems from a Bayesian Perspective [ACM2021]
• Ensemble deep learning: A review [Engineering Applications of AI 2021]
• A review of uncertainty quantification in deep learning: Techniques, applications and challenges [Information Fusion 2021]
• Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods [Machine Learning 2021]
• Predictive inference with the jackknife+ [The Annals of Statistics 2021]
• Uncertainty in big data analytics: survey, opportunities, and challenges [Journal of Big Data 2019]

Arxiv

• Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks [ArXiv2024] - [PyTorch]
• A System-Level View on Out-of-Distribution Data in Robotics [arXiv2022]
• A Survey on Uncertainty Reasoning and Quantification for Decision Making: Belief Theory Meets Deep Learning [arXiv2022]

Conference

• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning [ICLR2023]
• Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask? [ICLR2023]
• Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputs [ICML2023] - [PyTorch]
• On Second-Order Scoring Rules for Epistemic Uncertainty Quantification [ICML2023]
• Neural Variational Gradient Descent [AABI2022]
• Top-label calibration and multiclass-to-binary reductions [ICLR2022]
• Bayesian Model Selection, the Marginal Likelihood, and Generalization [ICML2022]
• With malice towards none: Assessing uncertainty via equalized coverage [AIES 2021]
• Uncertainty in Gradient Boosting via Ensembles [ICLR2021] - [PyTorch]
• Repulsive Deep Ensembles are Bayesian [NeurIPS2021] - [PyTorch]
• Bayesian Optimization with High-Dimensional Outputs [NeurIPS2021]
• Residual Pathway Priors for Soft Equivariance Constraints [NeurIPS2021]
• Dangers of Bayesian Model Averaging under Covariate Shift [NeurIPS2021] - [TensorFlow]
• A Mathematical Analysis of Learning Loss for Active Learning in Regression [CVPR Workshop2021]
• Deep Convolutional Networks as shallow Gaussian Processes [ICLR2019]
• On the accuracy of influence functions for measuring group effects [NeurIPS2018]
• To Trust Or Not To Trust A Classifier [NeurIPS2018] - [Python]
• Understanding Measures of Uncertainty for Adversarial Example Detection [UAI2018]

Journal

• A Unified Theory of Diversity in Ensemble Learning [JMLR2023]
• Multivariate Uncertainty in Deep Learning [TNNLS2021]
• A General Framework for Uncertainty Estimation in Deep Learning [RAL2020]
• Adaptive nonparametric confidence sets [Ann. Statist. 2006]

Arxiv

• Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrapping [arXiv2022]
• Efficient Gaussian Neural Processes for Regression [arXiv2021]
• Dense Uncertainty Estimation [arXiv2021] - [PyTorch]
• A higher-order swiss army infinitesimal jackknife [arXiv2019]

Conference

• A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors [ICLR2024]
• Gradient-based Uncertainty Attribution for Explainable Bayesian Deep Learning [CVPR2023]
• Robustness to corruption in pre-trained Bayesian neural networks [ICLR2023]
• Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift [NeurIPS2023] - [PyTorch]
• Transformers Can Do Bayesian Inference [ICLR2022] - [PyTorch]
• Uncertainty Estimation for Multi-view Data: The Power of Seeing the Whole Picture [NeurIPS2022]
• On Batch Normalisation for Approximate Bayesian Inference [AABI2021]
• Activation-level uncertainty in deep neural networks [ICLR2021]
• Laplace Redux – Effortless Bayesian Deep Learning [NeurIPS2021] - [PyTorch]
• On the Effects of Quantisation on Model Uncertainty in Bayesian Neural Networks [UAI2021]
• Learnable uncertainty under Laplace approximations [UAI2021]
• Bayesian Neural Networks with Soft Evidence [ICML Workshop2021] - [PyTorch]
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation [ECCV2020] - [PyTorch]
• How Good is the Bayes Posterior in Deep Neural Networks Really? [ICML2020]
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors [ICML2020] - [TensorFlow]
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks [ICML2020] - [PyTorch]
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization [NeurIPS2020]
• A Simple Baseline for Bayesian Uncertainty in Deep Learning [NeurIPS2019] - [PyTorch]
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks [ICML2018] - [TensorFlow] - [TorchUncertainty]
• Lightweight Probabilistic Deep Networks [CVPR2018] - [PyTorch]
• A Scalable Laplace Approximation for Neural Networks [ICLR2018] - [Theano]
• Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning [ICML2018]
• Weight Uncertainty in Neural Networks [ICML2015]

Journal

• Encoding the latent posterior of Bayesian Neural Networks for uncertainty quantification [TPAMI2023] - [PyTorch]
• Bayesian modeling of uncertainty in low-level vision [IJCV1990]

Arxiv

• Density Uncertainty Layers for Reliable Uncertainty Estimation [arXiv2023]

Conference

• Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization [ICML2023]
• Bayesian Posterior Approximation With Stochastic Ensembles [CVPR2023]
• Normalizing Flow Ensembles for Rich Aleatoric and Epistemic Uncertainty Modeling [AAAI2023]
• Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Models [ICCV2023] - [PyTorch]
• Weighted Ensemble Self-Supervised Learning [ICLR2023]
• Agree to Disagree: Diversity through Disagreement for Better Transferability [ICLR2023] - [PyTorch]
• Packed-Ensembles for Efficient Uncertainty Estimation [ICLR2023] - [PyTorch/TorchUncertainty]
• Sub-Ensembles for Fast Uncertainty Estimation in Neural Networks [ICCV Workshop2023]
• Prune and Tune Ensembles: Low-Cost Ensemble Learning With Sparse Independent Subnetworks [AAAI2022]
• Deep Ensembles Work, But Are They Necessary? [NeurIPS2022]
• FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation [NeurIPS2022]
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity [ICLR2022] - [PyTorch]
• On the Usefulness of Deep Ensemble Diversity for Out-of-Distribution Detection [ECCV Workshop2022]
• Masksembles for Uncertainty Estimation [CVPR2021] - [PyTorch/TensorFlow]
• Robustness via Cross-Domain Ensembles [ICCV2021] - [PyTorch]
• Uncertainty in Gradient Boosting via Ensembles [ICLR2021] - [PyTorch]
• Uncertainty Quantification and Deep Ensembles [NeurIPS2021]
• Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles [AAAI2020]
• Uncertainty in Neural Networks: Approximately Bayesian Ensembling [AISTATS 2020]
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning [ICLR2020] - [PyTorch]
• BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning [ICLR2020] - [TensorFlow] - [PyTorch]
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification [NeurIPS2020]
• Bayesian Deep Ensembles via the Neural Tangent Kernel [NeurIPS2020]
• Diversity with Cooperation: Ensemble Methods for Few-Shot Classification [ICCV2019]
• Accurate Uncertainty Estimation and Decomposition in Ensemble Learning [NeurIPS2019]
• High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach [ICML2018] - [TensorFlow]
• Simple and scalable predictive uncertainty estimation using deep ensembles [NeurIPS2017] - [TorchUncertainty]

Journal

• One Versus all for deep Neural Network for uncertaInty (OVNNI) quantification [IEEE Access2021]

Arxiv

• Split-Ensemble: Efficient OOD-aware Ensemble via Task and Model Splitting [arXiv2023]
• Deep Ensemble as a Gaussian Process Approximate Posterior [arXiv2022]
• Sequential Bayesian Neural Subnetwork Ensembles [arXiv2022]
• Confident Neural Network Regression with Bootstrapped Deep Ensembles [arXiv2022] - [TensorFlow]
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model [arXiv2021]
• Deep Ensembles: A Loss Landscape Perspective [arXiv2019]

Conference

• Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models [CVPR2024]
• Training-Free Uncertainty Estimation for Dense Regression: Sensitivity as a Surrogate [AAAI2022]
• Efficient Bayesian Uncertainty Estimation for nnU-Net [MICCAI2022]
• Dropout Sampling for Robust Object Detection in Open-Set Conditions [ICRA2018]
• Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks [MIDL2018]
• Concrete Dropout [NeurIPS2017]
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning [ICML2016] - [TorchUncertainty]

Journal

• A General Framework for Uncertainty Estimation in Deep Learning [Robotics and Automation Letters2020]

Arxiv

• SoftDropConnect (SDC) – Effective and Efficient Quantification of the Network Uncertainty in Deep MR Image Analysis [arXiv2022]

Conference

• Discretization-Induced Dirichlet Posterior for Robust Uncertainty Quantification on Regression [AAAI2024] - [PyTorch]
• Post-hoc Uncertainty Learning using a Dirichlet Meta-Model [AAAI2023] - [PyTorch]
• ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models [ICCV2023]
• Out-of-Distribution Detection for Monocular Depth Estimation [ICCV2023]
• Detecting Misclassification Errors in Neural Networks with a Gaussian Process Model [AAAI2022]
• Learning Structured Gaussians to Approximate Deep Ensembles [CVPR2022]
• Improving the reliability for confidence estimation [ECCV2022]
• Gradient-based Uncertainty for Monocular Depth Estimation [ECCV2022] - [PyTorch]
• BayesCap: Bayesian Identity Cap for Calibrated Uncertainty in Frozen Neural Networks [ECCV2022] - [PyTorch]
• Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving [ICIP2022]
• SLURP: Side Learning Uncertainty for Regression Problems [BMVC2021] - [PyTorch]
• Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation [ICCV2021] - [PyTorch]
• Learning to Predict Error for MRI Reconstruction [MICCAI2021]
• A Mathematical Analysis of Learning Loss for Active Learning in Regression [CVPR Workshop2021]
• Real-time uncertainty estimation in computer vision via uncertainty-aware distribution distillation [WACV2021]
• On the uncertainty of self-supervised monocular depth estimation [CVPR2020] - [PyTorch]
• Quantifying Point-Prediction Uncertainty in Neural Networks via Residual Estimation with an I/O Kernel [ICLR2020] - [TensorFlow]
• Gradients as a Measure of Uncertainty in Neural Networks [ICIP2020]
• Learning Loss for Test-Time Augmentation [NeurIPS2020]
• Learning loss for active learning [CVPR2019] - [PyTorch] (unofficial codes)
• Addressing failure prediction by learning model confidence [NeurIPS2019] - [PyTorch]
• Structured Uncertainty Prediction Networks [CVPR2018] - [TensorFlow]
• Classification uncertainty of deep neural networks based on gradient information [IAPR Workshop2018]

Journal

• Towards More Reliable Confidence Estimation [TPAMI2023]
• Confidence Estimation via Auxiliary Models [TPAMI2021]

Arxiv

• Instance-Aware Observer Network for Out-of-Distribution Object Segmentation [arXiv2022]
• DEUP: Direct Epistemic Uncertainty Prediction [arXiv2020]
• Learning Confidence for Out-of-Distribution Detection in Neural Networks [arXiv2018]

Conference

• Learning to Generate Training Datasets for Robust Semantic Segmentation [WACV2024]
• OpenMix: Exploring Outlier Samples for Misclassification Detection [CVPR2023] - [PyTorch]
• On the Pitfall of Mixup for Uncertainty Calibration [CVPR2023]
• Diverse, Global and Amortised Counterfactual Explanations for Uncertainty Estimates [AAAI2022]
• Out-of-distribution Detection with Implicit Outlier Transformation [ICLR2023] - [PyTorch]
• PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures [CVPR2022]
• RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy & Out-of-Distribution Robustness [NeurIPS2022] - [PyTorch]
• Towards efficient feature sharing in MIMO architectures [CVPR Workshop2022]
• Robust Semantic Segmentation with Superpixel-Mix [BMVC2021] - [PyTorch]
• MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks [ICCV2021] - [PyTorch]
• Training independent subnetworks for robust prediction [ICLR2021]
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [IJCAI2021] - [PyTorch]
• Uncertainty-aware GAN with Adaptive Loss for Robust MRI Image Enhancement [ICCV Workshop2021]
• Uncertainty-Aware Deep Classifiers using Generative Models [AAAI2020]
• Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation [ECCV2020] - [PyTorch]
• Detecting the Unexpected via Image Resynthesis [ICCV2019] - [PyTorch]
• Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning [ICML2020]
• Deep Anomaly Detection with Outlier Exposure [ICLR2019]
• On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks [NeurIPS2019]

Arxiv

• Reliability in Semantic Segmentation: Can We Use Synthetic Data? [arXiv2023]
• ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference [arXiv2022]
• Quantifying uncertainty with GAN-based priors [arXiv2019] - [TensorFlow]

Conference

• Discretization-Induced Dirichlet Posterior for Robust Uncertainty Quantification on Regression [AAAI2024] - [PyTorch]
• The Unreasonable Effectiveness of Deep Evidential Regression [AAAI2023] - [PyTorch] - [TorchUncertainty]
• Exploring and Exploiting Uncertainty for Incomplete Multi-View Classification [CVPR2023]
• Plausible Uncertainties for Human Pose Regression [ICCV2023] - [PyTorch]
• Uncertainty Estimation by Fisher Information-based Evidential Deep Learning [ICML2023] - [PyTorch]
• Improving Evidential Deep Learning via Multi-task Learning [AAAI2022] - [PyTorch]
• An Evidential Neural Network Model for Regression Based on Random Fuzzy Numbers [BELIEF2022]
• On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks [ICLR2022] - [PyTorch]
• Natural Posterior Network: Deep Bayesian Uncertainty for Exponential Family Distributions [ICLR2022] - [PyTorch]
• Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation [NeurIPS2022]
• Fast Predictive Uncertainty for Classification with Bayesian Deep Networks [UAI2022] - [PyTorch]
• Evaluating robustness of predictive uncertainty estimation: Are Dirichlet-based models reliable? [ICML2021]
• Trustworthy multimodal regression with mixture of normal-inverse gamma distributions [NeurIPS2021]
• Misclassification Risk and Uncertainty Quantification in Deep Classifiers [WACV2021]
• Ensemble Distribution Distillation [ICLR2020]
• Conservative Uncertainty Estimation By Fitting Prior Networks [ICLR2020]
• Being Bayesian about Categorical Probability [ICML2020] - [PyTorch]
• Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts [NeurIPS2020] - [PyTorch]
• Deep Evidential Regression [NeurIPS2020] - [TensorFlow]
• Noise Contrastive Priors for Functional Uncertainty [UAI2020]
• Towards Maximizing the Representation Gap between In-Domain & Out-of-Distribution Examples [NeurIPS Workshop2020]
• Uncertainty on Asynchronous Time Event Prediction [NeurIPS2019] - [TensorFlow]
• Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness [NeurIPS2019]
• Quantifying Classification Uncertainty using Regularized Evidential Neural Networks [AAAI FSS2019]
• Uncertainty estimates and multi-hypotheses networks for optical flow [ECCV2018] - [TensorFlow]
• Evidential Deep Learning to Quantify Classification Uncertainty [NeurIPS2018] - [PyTorch]
• Predictive uncertainty estimation via prior networks [NeurIPS2018]
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? [NeurIPS2017]
• Estimating the Mean and Variance of the Target Probability Distribution [(ICNN1994)]

Journal

• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation [TMLR2023]
• Region-Based Evidential Deep Learning to Quantify Uncertainty and Improve Robustness of Brain Tumor Segmentation [NCA2022]
• An evidential classifier based on Dempster-Shafer theory and deep learning [Neurocomputing2021] - [TensorFlow]
• Evidential fully convolutional network for semantic segmentation [AppliedIntelligence2021] - [TensorFlow]
• Information Aware max-norm Dirichlet networks for predictive uncertainty estimation [NeuralNetworks2021]
• A neural network classifier based on Dempster-Shafer theory [IEEETransSMC2000]

Arxiv

• Evidential Uncertainty Quantification: A Variance-Based Perspective [arXiv2023]
• Effective Uncertainty Estimation with Evidential Models for Open-World Recognition [arXiv2022]
• Multivariate Deep Evidential Regression [arXiv2022]
• Regression Prior Networks [arXiv2020]
• A Variational Dirichlet Framework for Out-of-Distribution Detection [arXiv2019]
• Uncertainty estimation in deep learning with application to spoken language assessment [PhDThesis2019]
• Inhibited softmax for uncertainty estimation in neural networks [arXiv2018]
• Quantifying Intrinsic Uncertainty in Classification via Deep Dirichlet Mixture Networks [arXiv2018]

Conference

• Deep Deterministic Uncertainty: A Simple Baseline [CVPR2023] - [PyTorch]
• Gaussian Latent Representations for Uncertainty Estimation using Mahalanobis Distance in Deep Classifiers [ICCV Workshop2023] - [PyTorch]
• A Simple and Explainable Method for Uncertainty Estimation using Attribute Prototype Networks [ICCV Workshop2023]
• Training, Architecture, and Prior for Deterministic Uncertainty Methods [ICLR Workshop2023] - [PyTorch]
• Latent Discriminant deterministic Uncertainty [ECCV2022] - [PyTorch]
• On the Practicality of Deterministic Epistemic Uncertainty [ICML2022]
• Improving Deterministic Uncertainty Estimation in Deep Learning for Classification and Regression [CoRR2021]
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network [ICML2020] - [PyTorch]
• Training normalizing flows with the information bottleneck for competitive generative classification [NeurIPS2020]
• Simple and principled uncertainty estimation with deterministic deep learning via distance awareness [NeurIPS2020]
• Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks [ICML Workshop2020]
• Sampling-Free Epistemic Uncertainty Estimation Using Approximated Variance Propagation [ICCV2019] - [PyTorch]
• Single-Model Uncertainties for Deep Learning [NeurIPS2019] - [PyTorch]

Journal

• Density estimation in representation space [EDSMLS2020]

Arxiv

• The Hidden Uncertainty in a Neural Network’s Activations [arXiv2020]
• A simple framework for uncertainty in contrastive learning [arXiv2020]
• Distance-based Confidence Score for Neural Network Classifiers [arXiv2017]

Conference

• Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging [ICML2022] - [PyTorch]
• Prediction Intervals: Split Normal Mixture from Quality-Driven Deep Ensembles [UAI2020] - [PyTorch]
• Classification with Valid and Adaptive Coverage [NeurIPS2020]
• Single-Model Uncertainties for Deep Learning [NeurIPS2019] - [PyTorch]
• High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach [ICML2018] - [TensorFlow]

Journal

• Scalable Uncertainty Quantification for Deep Operator Networks using Randomized Priors [CMAME2022]
• Exploring uncertainty in regression neural networks for construction of prediction intervals [Neurocomputing2022]

Arxiv

• Interval Neural Networks: Uncertainty Scores [arXiv2020]
• Tight Prediction Intervals Using Expanded Interval Minimization [arXiv2018]

Awesome Conformal Prediction [GitHub]

Conference

• Calibrating Transformers via Sparse Gaussian Processes [ICLR2023] - [PyTorch]
• Beyond calibration: estimating the grouping loss of modern neural networks [ICLR2023] - [Python]
• What Are Effective Labels for Augmented Data? Improving Calibration and Robustness with AutoLabel [SaTML2023]
• The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration [CVPR2022] - [PyTorch]
• Calibrating Deep Neural Networks by Pairwise Constraints [CVPR2022]
• Top-label calibration and multiclass-to-binary reductions [ICLR2022]
• From label smoothing to label relaxation [AAAI2021]
• Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain [AIStats2021]
• Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification [NeurIPS2021]
• Confidence-Aware Learning for Deep Neural Networks [ICML2020] - [PyTorch]
• Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning [ICML2020]
• Regularization via structural label smoothing [ICML2020]
• Well-Calibrated Regression Uncertainty in Medical Imaging with Deep Learning [MIDL2020] - [PyTorch]
• Calibrating Deep Neural Networks using Focal Loss [NeurIPS2020] - [PyTorch]
• Stationary activations for uncertainty calibration in deep learning [NeurIPS2020]
• Revisiting the evaluation of uncertainty estimation and its application to explore model complexity-uncertainty trade-off [CVPR Workshop2020]
• Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision [CVPR Workshop2020] - [PyTorch]
• Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers [ICLR2019]
• Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration [NeurIPS2019] - [GitHub]
• When does label smoothing help? [NeurIPS2019]
• Verified Uncertainty Calibration [NeurIPS2019] - [GitHub]
• Measuring Calibration in Deep Learning [CVPR Workshop2019]
• Accurate Uncertainties for Deep Learning Using Calibrated Regression [ICML2018]
• Generalized zero-shot learning with deep calibration network [NeurIPS2018]
• On calibration of modern neural networks [ICML2017] - [TorchUncertainty]
• On Fairness and Calibration [NeurIPS2017]
• Obtaining Well Calibrated Probabilities Using Bayesian Binning [AAAI2015]

Journal

• Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error [TMLR2023] - [PyTorch]
• Evaluating and Calibrating Uncertainty Prediction in Regression Tasks [Sensors2022]
• Calibrated Prediction Intervals for Neural Network Regressors [IEEE Access 2018] - [Python]

Arxiv

• Towards Understanding Label Smoothing [arXiv2020]
• An Investigation of how Label Smoothing Affects Generalization [arXiv2020]

Conference

• Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts [ICLR2023] - [PyTorch]
• Anytime Dense Prediction with Confidence Adaptivity [ICLR2022] - [PyTorch]
• CRISP - Reliable Uncertainty Estimation for Medical Image Segmentation [MICCAI2022]
• TBraTS: Trusted Brain Tumor Segmentation [MICCAI2022] - [PyTorch]
• Robust Semantic Segmentation with Superpixel-Mix [BMVC2021] - [PyTorch]
• Deep Deterministic Uncertainty for Semantic Segmentation [ICMLW2021]
• DEAL: Difficulty-aware Active Learning for Semantic Segmentation [ACCV2020]
• Classification with Valid and Adaptive Coverage [NeurIPS2020]
• Guided Curriculum Model Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation [ICCV2019]
• Human Uncertainty Makes Classification More Robust [ICCV2019]
• Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation [MICCAI2019] - [PyTorch]
• Lightweight Probabilistic Deep Networks [CVPR2018] - [PyTorch]
• A Probabilistic U-Net for Segmentation of Ambiguous Images [NeurIPS2018] - [PyTorch]
• Evidential Deep Learning to Quantify Classification Uncertainty [NeurIPS2018] - [PyTorch]
• To Trust Or Not To Trust A Classifier [NeurIPS2018]
• Classification uncertainty of deep neural networks based on gradient information [IAPR Workshop2018]
• Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding [BMVC2017]

Journal

• Explainable machine learning in image classification models: An uncertainty quantification perspective." [KnowledgeBased2022]
• Region-Based Evidential Deep Learning to Quantify Uncertainty and Improve Robustness of Brain Tumor Segmentation [NCA2022]

Arxiv

• Leveraging Uncertainty Estimates to Improve Classifier Performance [arXiv2023]
• Evaluating Bayesian Deep Learning Methods for Semantic Segmentation [arXiv2018]

Conference

• Learning the Distribution of Errors in Stereo Matching for Joint Disparity and Uncertainty Estimation [CVPR2023] - [PyTorch]
• Probabilistic MIMO U-Net: Efficient and Accurate Uncertainty Estimation for Pixel-wise Regression [ICCV Workshop2023] - [PyTorch]
• Training-Free Uncertainty Estimation for Dense Regression: Sensitivity as a Surrogate [AAAI2022]
• Learning Structured Gaussians to Approximate Deep Ensembles [CVPR2022]
• Uncertainty Quantification in Depth Estimation via Constrained Ordinal Regression [ECCV2022]
• On Monocular Depth Estimation and Uncertainty Quantification using Classification Approaches for Regression [ICIP2022]
• Anytime Dense Prediction with Confidence Adaptivity [ICLR2022] - [PyTorch]
• Variational Depth Networks: Uncertainty-Aware Monocular Self-supervised Depth Estimation [ECCV Workshop2022]
• SLURP: Side Learning Uncertainty for Regression Problems [BMVC2021] - [PyTorch]
• Robustness via Cross-Domain Ensembles [ICCV2021] - [PyTorch]
• Learning to Predict Error for MRI Reconstruction [MICCAI2021]
• On the uncertainty of self-supervised monocular depth estimation [CVPR2020] - [PyTorch]
• Quantifying Point-Prediction Uncertainty in Neural Networks via Residual Estimation with an I/O Kernel [ICLR2020] - [TensorFlow]
• Fast Uncertainty Estimation for Deep Learning Based Optical Flow [IROS2020]
• Well-Calibrated Regression Uncertainty in Medical Imaging with Deep Learning [MIDL2020] - [PyTorch]
• Deep Evidential Regression [NeurIPS2020] - [TensorFlow]
• Inferring Distributions Over Depth from a Single Image [IROS2019] - [TensorFlow]
• Multi-Task Learning based on Separable Formulation of Depth Estimation and its Uncertainty [CVPR Workshop2019]
• Lightweight Probabilistic Deep Networks [CVPR2018] - [PyTorch]
• Structured Uncertainty Prediction Networks [CVPR2018] - [TensorFlow]
• Uncertainty estimates and multi-hypotheses networks for optical flow [ECCV2018] - [TensorFlow]
• Accurate Uncertainties for Deep Learning Using Calibrated Regression [ICML2018]

Journal

• How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts? [TMLR2023] - [PyTorch]
• Evaluating and Calibrating Uncertainty Prediction in Regression Tasks [Sensors2022]
• Exploring uncertainty in regression neural networks for construction of prediction intervals [Neurocomputing2022]
• Wasserstein Dropout [Machine Learning 2022] - [PyTorch]
• Deep Distribution Regression [Computational Statistics & Data Analysis2021]
• Calibrated Prediction Intervals for Neural Network Regressors [IEEE Access 2018] - [Python]
• Learning a Confidence Measure for Optical Flow [TPAMI2013]

Arxiv

• Measuring and Modeling Uncertainty Degree for Monocular Depth Estimation [arXiv2023]
• UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed Tomographaphy [arXiv2022]
• Efficient Gaussian Neural Processes for Regression [arXiv2021]

Conference

• SURE: SUrvey REcipes for building reliable and robust deep networks [CVPR2024] - [PyTorch]
• NECO: NEural Collapse Based Out-of-distribution Detection [ICLR2024]
• Anomaly Detection under Distribution Shift [ICCV2023] - [PyTorch]
• Normalizing Flows for Human Pose Anomaly Detection [ICCV2023] - [PyTorch]
• RbA: Segmenting Unknown Regions Rejected by All [ICCV2023] - [PyTorch]
• Uncertainty-Aware Optimal Transport for Semantically Coherent Out-of-Distribution Detection [CVPR2023] - [PyTorch]
• Modeling the Distributional Uncertainty for Salient Object Detection Models [CVPR2023] - [PyTorch]
• SQUID: Deep Feature In-Painting for Unsupervised Anomaly Detection [CVPR2023] - [PyTorch]
• How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection? [ICLR2023] - [PyTorch]
• Modeling the Data-Generating Process is Necessary for Out-of-Distribution Generalization [ICLR2023]
• Can CNNs Be More Robust Than Transformers? [ICLR2023]
• A framework for benchmarking class-out-of-distribution detection and its application to ImageNet [ICLR2023]
• Extremely Simple Activation Shaping for Out-of-Distribution Detection [ICLR2023] - [PyTorch]
• Quantification of Uncertainty with Adversarial Models [NeurIPS2023]
• The Robust Semantic Segmentation UNCV2023 Challenge Results [ICCV Workshop2023]
• Continual Evidential Deep Learning for Out-of-Distribution Detection [ICCV Workshop2023]
• Far Away in the Deep Space: Nearest-Neighbor-Based Dense Out-of-Distribution Detection [ICCV Workshop2023]
• Gaussian Latent Representations for Uncertainty Estimation using Mahalanobis Distance in Deep Classifiers [ICCV Workshop2023]
• Calibrated Out-of-Distribution Detection with a Generic Representation [ICCV Workshop2023] - [PyTorch]
• Detecting Misclassification Errors in Neural Networks with a Gaussian Process Model [AAAI2022]
• Augmenting Softmax Information for Selective Classification with Out-of-Distribution Data [ACCV2022]
• Anomaly Detection via Reverse Distillation from One-Class Embedding [CVPR2022]
• Towards Total Recall in Industrial Anomaly Detection [CVPR2022] - [PyTorch]
• Rethinking Confidence Calibration for Failure Prediction [ECCV2022] - [PyTorch]
• VOS: Learning What You Don't Know by Virtual Outlier Synthesis [ICLR2022] - [PyTorch]
• Fully Convolutional Cross-Scale-Flows for Image-based Defect Detection [WACV2022] - [PyTorch]
• Out-of-Distribution Detection Using Union of 1-Dimensional Subspaces [CVPR2021] - [PyTorch]
• NAS-OoD: Neural Architecture Search for Out-of-Distribution Generalization [ICCV2021]
• On the Importance of Gradients for Detecting Distributional Shifts in the Wild [NeurIPS2021]
• Exploring the Limits of Out-of-Distribution Detection [NeurIPS2021]
• Detecting out-of-distribution image without learning from out-of-distribution data. [CVPR2020]
• Learning Open Set Network with Discriminative Reciprocal Points [ECCV2020]
• Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation [ECCV2020] - [PyTorch]
• NADS: Neural Architecture Distribution Search for Uncertainty Awareness [ICML2020]
• PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localization [ICPR2020] - [PyTorch]
• Energy-based Out-of-distribution Detection [NeurIPS2020]
• Towards Maximizing the Representation Gap between In-Domain & Out-of-Distribution Examples [NeurIPS Workshop2020]
• Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection [ICCV2019] - [PyTorch]
• Detecting the Unexpected via Image Resynthesis [ICCV2019] - [PyTorch]
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks [ICLR2018]
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks [ICLR2017] - [TensorFlow]

Journal

• Revisiting Confidence Estimation: Towards Reliable Failure Prediction [TPAMI2023] - [PyTorch]
• One Versus all for deep Neural Network for uncertaInty (OVNNI) quantification [IEEE Access2021]

Arxiv

• Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization [arXiv2023] - [PyTorch]
• A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection [arXiv2021]
• Generalized out-of-distribution detection: A survey [arXiv2021]
• Do We Really Need to Learn Representations from In-domain Data for Outlier Detection? [arXiv2021]
• Frequentist uncertainty estimates for deep learning [arXiv2018]

Conference

• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [CVPR2023]
• Parametric and Multivariate Uncertainty Calibration for Regression and Object Detection [ECCV Workshop2022] - [PyTorch]
• Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors [ICLR2021]
• Multivariate Confidence Calibration for Object Detection [CVPR Workshop2020] - [PyTorch]
• Gaussian YOLOv3: An Accurate and Fast Object Detector Using Localization Uncertainty for Autonomous Driving [ICCV2019] - [CUDA] - [PyTorch] - [Keras]

Conference

• Guiding Pseudo-labels with Uncertainty Estimation for Source-free Unsupervised Domain Adaptation [CVPR2023] - [PyTorch]
• Uncertainty-guided Source-free Domain Adaptation [ECCV2022] - [PyTorch]

Conference

• Confidence Estimation Using Unlabeled Data [ICLR2023] - [PyTorch]

Awesome LLM Uncertainty, Reliability, & Robustness [GitHub]

Conference

• R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intents [ICML2023] - [GitHub]
• Strength in Numbers: Estimating Confidence of Large Language Models by Prompt Agreement [TrustNLP2023] - [GitHub]
• Disentangling Uncertainty in Machine Translation Evaluation [EMNLP2022] - [PyTorch]
• Investigating Ensemble Methods for Model Robustness Improvement of Text Classifiers [EMNLP2022 Findings]
• DATE: Detecting Anomalies in Text via Self-Supervision of Transformers [NAACL2021]
• Calibrating Structured Output Predictors for Natural Language Processing [ACL2020]
• Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data [EMNLP2020] - [PyTorch]

Journal

• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering [TACL2021] - [PyTorch]

Arxiv

• Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling [arXiv2023]

Arxiv

• Urban 3D Panoptic Scene Completion with Uncertainty Awareness [arXiv2023] - [PyTorch]

• SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation [CVPR2022]
• MUAD: Multiple Uncertainties for Autonomous Driving, a benchmark for multiple uncertainty types and tasks [BMVC2022] - [PyTorch]
• ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding [ICCV2021]
• The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection [IJCV2021]
• SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation [NeurIPS2021]
• Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning [arXiv2021] - [TensorFlow]
• Curriculum Model Adaptation with Synthetic and Real Data for Semantic Foggy Scene Understanding [IJCV2020]
• Benchmarking the Robustness of Semantic Segmentation Models [CVPR2020]
• Fishyscapes: A Benchmark for Safe Semantic Segmentation in Autonomous Driving [ICCV Workshop2019]
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming [NeurIPS Workshop2019] - [GitHub]
• Semantic Foggy Scene Understanding with Synthetic Data [IJCV2018]
• Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles [IROS2016]

• Uncertainty Calibration Library [GitHub]
• MAPIE: Model Agnostic Prediction Interval Estimator [Sklearn]
• Uncertainty Toolbox [GitHub]
• OpenOOD: Benchmarking Generalized OOD Detection [GitHub]
• Darts: Forecasting and anomaly detection on time series [GitHub]
• Mixture Density Networks (MDN) for distribution and uncertainty estimation [GitHub]

• TorchUncertainty [GitHub]
• Bayesian Torch [GitHub]
• Blitz: A Bayesian Neural Network library for PyTorch [GitHub]

• Fortuna [GitHub - JAX]

• TensorFlow Probability [Website]

• Dan Hendrycks: Intro to ML Safety course [Website]
• Uncertainty and Robustness in Deep Learning Workshop in ICML (2020, 2021) [SlidesLive]
• Yarin Gal: Bayesian Deep Learning 101 [Website]
• MIT 6.S191: Evidential Deep Learning and Uncertainty (2021) [Youtube]
• Hands-on Bayesian Neural Networks - a Tutorial for Deep Learning Users [IEEE Computational Intelligence Magazine]

• The "Probabilistic Machine-Learning" book series by Kevin Murphy [Book]

Uncertainty Quantification in Deep Learning [GitHub]

Awesome Out-of-distribution Detection [GitHub]

Anomaly Detection Learning Resources [GitHub]

Awesome Conformal Prediction [GitHub]

Awesome LLM Uncertainty, Reliability, & Robustness [GitHub]

UQSay - Seminars on Uncertainty Quantification (UQ), Design and Analysis of Computer Experiments (DACE) and related topics @ Paris Saclay [Website]

ProbAI summer school [Website]

Gaussian process summer school [Website]