Now showing 1 - 5 of 5
  • Publication
    Bayesian Neural Networks for Out of Distribution Detection
    (University College Dublin. School of Computer Science, 2022) ;
    0000-0002-0189-2130
    Empirical studies have demonstrated that point estimate deep neural networks despite being expressive estimators capturing rich interactions between covariates, nevertheless, exhibit high sensitivity in their predictions leading to overconfident misclassifications due to changes in the underlying representation of data distributions. This implication lead us to study the problem of out-of-distribution detection in identifying and characterising out-of-distribution inputs. This phenomenon has real world implications especially in high-stake applications where it is undesirable and often prohibitive for an estimator to produce overconfident misclassified estimates. Alternatively, Bayesian models present a principled way of quantifying uncertainty over predictions represented in the estimator’s parameters but at the same time they pose challenges when applied to large high dimensional datasets due to computational constraints requiring estimating high dimensional integrals over a large para- meter space. Moreover, Bayesian models among others present properties leading to simple and intuitive formulation and interpretation of the underlying estimator. Therefore, we propose to exploit this synergy between Bayesian inference and deep neural networks for out-of-distribution detection. This synergy leads to Bayesian neural networks exhibiting the following benefits (i) providing efficient and flexible neural network architectures applicable to large high dimensional datasets, (ii) estimating the uncertainty over the predictions captured in the predictive posterior distribution via Bayesian inference. We validate our findings empirically across a number of datasets and performance metrics indicating the efficacy of the underlying methods and estimators presented in regard to calibration, uncertainty estimation, out-of-distribution detection, detection of corrupted adversarial inputs and finally the effectiveness of the proposed contrastive objectives for out-of-distribution detection. We hope that the methods and results presented here reflect the importance of how brittle an estimator can be due to discrepancies between train and test distribution leading to real world implications of particular interest to reliable and secure machine learning. The algorithmic advances and research questions presented in this dissertation extend the domains of out-of-distribution detection and robustness against ambiguous inputs, in addition to exploring auxiliary information that can be incorporated during training. The resulting estimators overall are high dimensional exhibiting efficient detection.
      324
  • Publication
    A Categorisation of Post-hoc Explanations for Predictive Models
    (Association for the Advancement of Artificial Intelligence, 2019-03-27) ;
    The ubiquity of machine learning based predictive models inmodern society naturally leads people to ask how trustworthythose models are? In predictive modeling, it is quite commonto induce a trade-off between accuracy and interpretability.For instance, doctors would like to know how effective sometreatment will be for a patient or why the model suggesteda particular medication for a patient exhibiting those symptoms? We acknowledge that the necessity for interpretabilityis a consequence of an incomplete formalisation of the prob-lem, or more precisely of multiple meanings adhered to a par-ticular concept. For certain problems, it is not enough to getthe answer (what), the model also has to provide an expla-nation of how it came to that conclusion (why), because acorrect prediction, only partially solves the original problem.In this article we extend existing categorisation of techniquesto aid model interpretability and test this categorisation
      154
  • Publication
    A Comparison of Bayesian Deep Learning for Out of Distribution Detection and Uncertainty Estimation
    Deep neural networks have been successful in diverse discriminitive classification tasks. Despite their good prediction performance, they are poorly calibrated– i.e., often assigns high confidence to misclassified predictions. Potential consequences could lead to trustworthiness and accountability of models deployed in real applications, where predictions are evaluated based on their confidence scores. In this work we propose to validate and test the efficacy of likelihood based models in the task of out-of-distribution (OoD) detection. On different datasets and metrics we show that Bayesian deep learning models on certain occasions marginally outperform conventional neural networks and in the event of minimal overlap between in/out distribution classes, even the best models exhibit a reduction in AUC scores. Preliminary investigations indicate the potential inherent role of bias due to choices of initialisation, architecture or activation functions.
      7
  • Publication
    Ramifications of Approximate Posterior Inference for Bayesian Deep Learning in Adversarial and Out-of-Distribution Settings
    Deep neural networks have been successful in diverse discriminative classification tasks, although, they are poorly calibrated often assigning high probability to misclassified predictions. Potential consequences could lead to trustworthiness and accountability of the models when deployed in real applications, where predictions are evaluated based on their confidence scores. Existing solutions suggest the benefits attained by combining deep neural networks and Bayesian inference to quantify uncertainty over the models’ predictions for ambiguous data points. In this work we propose to validate and test the efficacy of likelihood based models in the task of out of distribution detection (OoD). Across different datasets and metrics we show that Bayesian deep learning models indeed outperform conventional neural networks but in the event of minimal overlap between in/out distribution classes, even the best models exhibit a reduction in AUC scores in detecting OoD data. We hypothesise that the sensitivity of neural networks to unseen inputs could be a multi-factor phenomenon arising from the different architectural design choices often amplified by the curse of dimensionality. Preliminary investigations indicate the potential inherent role of bias due to choices of initialisation, architecture or activation functions. Furthermore, we perform an analysis on the effect of adversarial noise resistance methods regarding in and out-of-distribution performance when combined with Bayesian deep learners.
      534
  • Publication
    On the Validity of Bayesian Neural Networks for Uncertainty Estimation
    (CEUR Workshop Proceedings, 2019-12-06) ;
    Deep neural networks (DNN) are versatile parametric models utilised successfully in a diverse number of tasks and domains. However, they have limitations—particularly from their lack of robustness and over-sensitivity to out of distribution samples. Bayesian Neural Networks, due to their formulation under the Bayesian framework, provide a principled approach to building neural networks that address these limitations. This work provides an empirical study evaluating and comparing Bayesian Neural Networks to their equivalent point estimate Deep Neural Networks to quantify the predictive uncertainty induced by their parameters, as well as their performance in view of uncertainty. Specifically, we evaluated and compared three point estimate deep neural networks against their alternative comparable Bayesian neural network utilising well-known benchmark image classification datasets.
      165