Now showing 1 - 8 of 8
  • Publication
    Benchmarking Multi-label Classification Algorithms
    (CEUR Workshop Proceedings, 2016-09-21) ; ;
    Multi-label classification is an approach to classification prob- lems that allows each data point to be assigned to more than one class at the same time. Real life machine learning problems are often multi-label in nature—for example image labelling, topic identification in texts, and gene expression prediction. Many multi-label classification algorithms have been proposed in the literature and, although there have been some benchmarking experiments, many questions still remain about which ap- proaches perform best for certain kinds of multi-label datasets. This pa- per presents a comprehensive benchmark experiment of eleven multi- label classification algorithms on eleven different datasets. Unlike many existing studies, we perform detailed parameter tuning for each algorithm- dataset pair so as to allow a fair comparative analysis of the algorithms. Also, we report on a preliminary experiment which seeks to understand how the performance of different multi-label classification algorithms changes as the characteristics of multi-label datasets are adjusted.
      675
  • Publication
    Stacked-MLkNN: A stacking based improvement to Multi-Label k-Nearest Neighbours
    Multi-label classification deals with problems where each datapoint can be assigned to more than one class, or label, at the same time. The simplest approach for such problems is to train independent binary classification models for each label and use these models to independently predict a set of relevant labels for a datapoint. MLkNN is an instance-based lazy learning algorithm for multi-label classification that takes this approach. MLkNN, and similar algorithms, however, do not exploit associations which may exist between the set of potential labels. These methods also suffer from imbalance in the frequency of labels in a training dataset. This work attempts to improve the predictions of MLkNN by implementing a two-layer stack-like method, Stacked-MLkNN which exploits the label associations. Experiments show that Stacked-MLkNN produces better predictions than MLkNN and several other state-of-the-art instance-based learning algorithms.
      147
  • Publication
    MeetupNet Dublin: Discovering Communities in Dublin's Meetup Network
    (CEUR Workshop Proceedings, 2018-12-07) ; ; ;
    Meetup.com is a global online platform which facilitates the organisation of meetups in different parts of the world. A meetup group typically focuses on one specific topic of interest, such as sports, music, language, or technology. However, many users of this platform attend multiple meetups. On this basis, we can construct a co-membership network for a given location. This network encodes how pairs of meetups are connected to one another via common members. In this work we demonstrate that, by applying techniques from social network analysis to this type of representation, we can reveal the underlying meetup community structure, which is not immediately apparent from the platform's website. Specifically, we map the landscape of Dublin's meetup communities, to explore the interests and activities of meetup.com users in the city.
      137
  • Publication
    A Comparison of Bayesian Deep Learning for Out of Distribution Detection and Uncertainty Estimation
    Deep neural networks have been successful in diverse discriminitive classification tasks. Despite their good prediction performance, they are poorly calibrated– i.e., often assigns high confidence to misclassified predictions. Potential consequences could lead to trustworthiness and accountability of models deployed in real applications, where predictions are evaluated based on their confidence scores. In this work we propose to validate and test the efficacy of likelihood based models in the task of out-of-distribution (OoD) detection. On different datasets and metrics we show that Bayesian deep learning models on certain occasions marginally outperform conventional neural networks and in the event of minimal overlap between in/out distribution classes, even the best models exhibit a reduction in AUC scores. Preliminary investigations indicate the potential inherent role of bias due to choices of initialisation, architecture or activation functions.
      7
  • Publication
    Ramifications of Approximate Posterior Inference for Bayesian Deep Learning in Adversarial and Out-of-Distribution Settings
    Deep neural networks have been successful in diverse discriminative classification tasks, although, they are poorly calibrated often assigning high probability to misclassified predictions. Potential consequences could lead to trustworthiness and accountability of the models when deployed in real applications, where predictions are evaluated based on their confidence scores. Existing solutions suggest the benefits attained by combining deep neural networks and Bayesian inference to quantify uncertainty over the models’ predictions for ambiguous data points. In this work we propose to validate and test the efficacy of likelihood based models in the task of out of distribution detection (OoD). Across different datasets and metrics we show that Bayesian deep learning models indeed outperform conventional neural networks but in the event of minimal overlap between in/out distribution classes, even the best models exhibit a reduction in AUC scores in detecting OoD data. We hypothesise that the sensitivity of neural networks to unseen inputs could be a multi-factor phenomenon arising from the different architectural design choices often amplified by the curse of dimensionality. Preliminary investigations indicate the potential inherent role of bias due to choices of initialisation, architecture or activation functions. Furthermore, we perform an analysis on the effect of adversarial noise resistance methods regarding in and out-of-distribution performance when combined with Bayesian deep learners.
      534
  • Publication
    Kalman Filter-based Heuristic Ensemble (KFHE): A new perspective on multi-class ensemble classification using Kalman filters
    (Elsevier, 2019-06) ;
    This paper introduces a new perspective on multi-class ensemble classification that considers training an ensemble as a state estimation problem. The new perspective considers the final ensemble classifier model as a static state, which can be estimated using a Kalman filter that combines noisy estimates made by individual classifier models. A new algorithm based on this perspective, the Kalman Filter-based Heuristic Ensemble (KFHE), is also presented in this paper which shows the practical applicability of the new perspective. Experiments performed on 30 datasets compare KFHE with state-of-the-art multi-class ensemble classification algorithms and show the potential and effectiveness of the new perspective and algorithm. Existing ensemble approaches trade off classification accuracy against robustness to class label noise, but KFHE is shown to be significantly better or at least as good as the state-of-the-art algorithms for datasets both with and without class label noise.
    Scopus© Citations 13  374
  • Publication
    Improving multi-label classification using inter-label associations and a new Kalman filter based ensemble method
    (University College Dublin. School of Computer Science, 2020)
    In machine learning, classification algorithms are used to train models to recognise the class, or category, that an object belongs to. Most classification problems are multi-class, in which one object can belong to at most one class. However, there are many important real-world problems in which an object can belong to more than one class simultaneously. These are known as multi-label classification problems as an object can be labelled with more than one class. The multi-label classification algorithms in the literature range from very simple approaches, such as binary relevance, in which independent binary classifiers are built for each label, to sophisticated ensemble techniques, such as classifier chains, that build collections of interconnected classifiers. The most effective approaches tend to explicitly exploit relationships between the labels themselves, inter-label associations, and use ensembles. There is an opportunity, however, to more explicitly take advantage of inter-label associations and to use ensembling techniques that are more sophisticated than the bagging-based approaches that dominate the multi-label classification literature. There are several multi-label classification algorithms in the literature. The most basic methods are binary relevance and label powerset. Binary relevance considers each label as independent binary classification task and learns binary classifier models. Label powerset converts unique label assignment combinations to unique classes and then trains a multi-class classifier model. Although there are other methods which can benefit by considering the inter-label associations or through ensemble algorithms. Ensemble methods in multi-class domain generally perform much better than the individual classifier models. Although, except bagging like methods, there are not much work done in multi-label on boosting or boosting-like methods. This thesis investigates new algorithms for training multi-label classification models that exploit inter-label associations, and/or utilise ensemble models (especially boosting-like methods). Three new methods are proposed: Stacked-MLkNN, a stacked-ensemble-based lazy learning algorithm that exploits inter-label associations at the stacked layer; CascadeML, a neural network training algorithm that uses a cascade architecture to exploit inter-label associations and evolves the network architecture during training which minimises the requirement for hyperparameter tuning; and KFHE-HOMER, a multi-label ensemble training algorithm built using a newly proposed perspective on ensemble training that views it as a static state estimation problem that can be solved using the sensor fusion properties of the Kalman filter. This new perspective on ensemble training is also a contribution of this thesis, as are two new multi-class classification algorithms--- Kalman Filter-based Heuristic Ensemble (KFHE) and KalmanTune---that exploit it. Each newly proposed method is extensively evaluated across a set of well-known benchmark multi-label classification datasets, and compared to the performance of current state-of-the art methods. Each newly proposed method is found to be highly effective. Stacked-MLkNN performs better than all other existing instance-based multi-label classification algorithms against which it was compared. CascadeML can create models with comparable performance to the best performing multi-label methods, without requiring extensive hyperparameter tuning. KFHE outperforms leading multi-class ensemble methods, and KalmanTune can improve the performance of ensembles trained using boosting. Finally, KFHE-HOMER was found to perform better than all other multi-label classification methods against which it was compared.
      349
  • Publication
    CascadeML: An Automatic Neural Network Architecture Evolution and Training Algorithm for Multi-label Classification
    In multi-label classification a datapoint can be labelled with more than one class at the same time. A common but trivial approach to multi-label classification is to train individual binary classifiers per label, but the performance can be improved by considering associations between the labels, and algorithms like classifier chains and RAKEL do this effectively. Like most machine learning algorithms, however, these approaches require accurate hyperparameter tuning, a computationally expensive optimisation problem. Tuning is important to train a good multi-label classifier model. There is a scarcity in the literature of effective multi-label classification approaches that do not require extensive hyperparameter tuning. This paper addresses this scarcity by proposing CascadeML, a multi-label classification approach based on cascade neural network that takes label associations into account and requires minimal hyperparameter tuning. The performance of the CasecadeML approach is evaluated using 10 multi-label datasets and compared with other leading multi-label classification algorithms. Results show that CascadeML performs comparatively with the leading approaches but without a need for hyperparameter tuning.
      289Scopus© Citations 4