Now showing 1 - 10 of 47
  • Publication
    A Topic-Based Approach to Multiple Corpus Comparison
    (CEUR Workshop Proceedings, 2019-12-06) ; ;
    Corpus comparison techniques are often used to compare different types of online media, for example social media posts and news articles. Most corpus comparison algorithms operate at a word-level and results are shown as lists of individual discriminating words which makes identifying larger underlying differences between corpora challenging. Most corpus comparison techniques also work on pairs of corpora and do need easily extend to multiple corpora. To counter these issues, we introduce Multi-corpus Topic-based Corpus Comparison (MTCC) a corpus comparison approach that works at a topic level and that can compare multiple corpora at once. Experiments on multiple real-world datasets are carried demonstrate the effectiveness of MTCC and compare the usefulness of different statistical discrimination metrics - the χ2 and Jensen-Shannon Divergence metrics are shown to work well. Finally we demonstrate the usefulness of reporting corpus comparison results via topics rather than individual words. Overall we show that the topic-level MTCC approach can capture the difference between multiple corpora, and show the results in a more meaningful and interpretable way than approaches that operate at a word-level.
      8
  • Publication
    Benchmarking Multi-label Classification Algorithms
    (CEUR Workshop Proceedings, 2016-09-21) ; ;
    Multi-label classification is an approach to classification prob- lems that allows each data point to be assigned to more than one class at the same time. Real life machine learning problems are often multi-label in nature—for example image labelling, topic identification in texts, and gene expression prediction. Many multi-label classification algorithms have been proposed in the literature and, although there have been some benchmarking experiments, many questions still remain about which ap- proaches perform best for certain kinds of multi-label datasets. This pa- per presents a comprehensive benchmark experiment of eleven multi- label classification algorithms on eleven different datasets. Unlike many existing studies, we perform detailed parameter tuning for each algorithm- dataset pair so as to allow a fair comparative analysis of the algorithms. Also, we report on a preliminary experiment which seeks to understand how the performance of different multi-label classification algorithms changes as the characteristics of multi-label datasets are adjusted.
      676
  • Publication
    Overlapping community finding with noisy pairwise constraints
    In many real applications of semi-supervised learning, the guidance provided by a human oracle might be “noisy” or inaccurate. Human annotators will often be imperfect, in the sense that they can make subjective decisions, they might only have partial knowledge of the task at hand, or they may simply complete a labeling task incorrectly due to the burden of annotation. Similarly, in the context of semi-supervised community finding in complex networks, information encoded as pairwise constraints may be unreliable or conflicting due to the human element in the annotation process. This study aims to address the challenge of handling noisy pairwise constraints in overlapping semi-supervised community detection, by framing the task as an outlier detection problem. We propose a general architecture which includes a process to “clean” or filter noisy constraints. Furthermore, we introduce multiple designs for the cleaning process which use different type of outlier detection models, including autoencoders. A comprehensive evaluation is conducted for each proposed methodology, which demonstrates the potential of the proposed architecture for reducing the impact of noisy supervision in the context of overlapping community detection.
      11
  • Publication
    Graphical Perception of Value Distributions: An Evaluation of Non-Expert Viewers Data Literacy
    (Journal of Community Informatics, 2016-06-06) ;
    An ability to understand the outputs of data analysis is a key characteristic of data literacy and the inclusion of data visualisations is ubiquitous in the output of modern data analysis. Several aspects still remain unresolved, however, on the question of choosing data visualisations that lead viewers to an optimal interpretation of data. This is especially true when audiences have differing degrees of data literacy, and when the aim is to make sure that members of a community, who may differ on background and expertise, will make similar interpretations from data visualisations. In this paper we describe two user studies on perception from data visualisations, in which we measured the ability of participants to validate statements about the distributions of data samples visualised using different chart types. In the first user study, we find that histograms are the most suitable chart type for illustrating the distribution of values for a variable. We contrast our findings with previous research in the field, and posit three main issues identified from the study. Most notably, however,we show that viewers struggle to identify scenarios in which a chart simply does not contain enough information to validate a statement about the data that it represents. In the follow-up study, we ask viewers questions about quantification of frequencies, and identification of most frequent values from different types of histograms and density traces showing one or two distributions of values.This study reveals that viewers do better with histograms when they need to quantify the values displayed in a chart. Among the different types of histograms, interspersing the bars of two distributions in a histogram leads to the most accurate perception. Even though interspersing bars makes them thinner, the advantage of having both distributions clearly visible pays off. The findings of these user studies provide insight to assist designers in creating optimal charts that enable comparison of distributions, and emphasise the importance of using an understanding of the limits of viewers data literacy to design charts effectively.
      240
  • Publication
    A Categorisation of Post-hoc Explanations for Predictive Models
    (Association for the Advancement of Artificial Intelligence, 2019-03-27) ;
    The ubiquity of machine learning based predictive models inmodern society naturally leads people to ask how trustworthythose models are? In predictive modeling, it is quite commonto induce a trade-off between accuracy and interpretability.For instance, doctors would like to know how effective sometreatment will be for a patient or why the model suggesteda particular medication for a patient exhibiting those symptoms? We acknowledge that the necessity for interpretabilityis a consequence of an incomplete formalisation of the prob-lem, or more precisely of multiple meanings adhered to a par-ticular concept. For certain problems, it is not enough to getthe answer (what), the model also has to provide an expla-nation of how it came to that conclusion (why), because acorrect prediction, only partially solves the original problem.In this article we extend existing categorisation of techniquesto aid model interpretability and test this categorisation
      155
  • Publication
    Using Icicle Trees to Encode the Hierarchical Structure of Source Code
    (Eurographics: European Association for Computer Graphics, 2016-06-10) ; ;
    This paper presents a study which evaluates the use of a tree visualisation (icicle tree) to encode the hierarchical structure of source code. The tree visualisation was combined with a source code editor in order to function as a compact overview to facilitate the process of comprehending the global structure of a source code document. Results from our study show that providing an overview visualisation led to an increase in accuracy and a decrease in completion time when participants performed counting tasks. However, in locating tasks, the presence of the visualisation led to a decrease in participants' performance.
      226
  • Publication
    Knowing What You Dont Know: Choosing the Right Chart to Show Data Distributions to Non-Expert Users
    An ability to understand the outputs of data analysis is a key characteristic of data literacy and the inclusion of data visualisations is ubiquitous in the output of modern data analysis. Several aspects still remain unresolved, however, on the question of choosing data visualisations that lead viewers to an optimal interpretation of data, especially when audiences have differing degrees of data literacy. In this paper we describe a user study on perception from data visualisations, in which we measured the ability of participants to validate statements about the distributions of data samples visualised using different chart types. We find that histograms are the most suitable chart type for illustrating the distribution of values for a variable. We contrast our findings with previous research in the field, and posit three main issues identified from the study. Most notably, however, we show that viewers struggle to identify scenarios in which a chart simply does not contain enough information to validate a statement about the data that it represents. The results of our study emphasise the importance of using an understanding of the limits of viewers’ data literacy to design charts effectively, and we discuss factors that are crucial to this end.
      245
  • Publication
    Deep learning at the shallow end: Malware classification for non-domain experts
    Current malware detection and classification approaches generally rely on time consuming and knowledge intensive processes to extract patterns (signatures) and behaviors from malware, which are then used for identification. Moreover, these signatures are often limited to local, contiguous sequences within the data whilst ignoring their context in relation to each other and throughout the malware file as a whole. We present a Deep Learning based malware classification approach that requires no expert domain knowledge and is based on a purely data driven approach for complex pattern and feature identification.
    Scopus© Citations 89  40
  • Publication
    Ramifications of Approximate Posterior Inference for Bayesian Deep Learning in Adversarial and Out-of-Distribution Settings
    Deep neural networks have been successful in diverse discriminative classification tasks, although, they are poorly calibrated often assigning high probability to misclassified predictions. Potential consequences could lead to trustworthiness and accountability of the models when deployed in real applications, where predictions are evaluated based on their confidence scores. Existing solutions suggest the benefits attained by combining deep neural networks and Bayesian inference to quantify uncertainty over the models’ predictions for ambiguous data points. In this work we propose to validate and test the efficacy of likelihood based models in the task of out of distribution detection (OoD). Across different datasets and metrics we show that Bayesian deep learning models indeed outperform conventional neural networks but in the event of minimal overlap between in/out distribution classes, even the best models exhibit a reduction in AUC scores in detecting OoD data. We hypothesise that the sensitivity of neural networks to unseen inputs could be a multi-factor phenomenon arising from the different architectural design choices often amplified by the curse of dimensionality. Preliminary investigations indicate the potential inherent role of bias due to choices of initialisation, architecture or activation functions. Furthermore, we perform an analysis on the effect of adversarial noise resistance methods regarding in and out-of-distribution performance when combined with Bayesian deep learners.
      534
  • Publication
    The Elliptical Basis Function Data Descriptor (EBFDD) Network: A One-Class Classification Approach to Anomaly Detection
    This paper introduces the Elliptical Basis Function Data Descriptor (EBFDD) network, a one-class classification approach to anomaly detection based on Radial Basis Function (RBF) neural networks. The EBFDD network uses elliptical basis functions, which allows it to learn sophisticated decision boundaries while retaining the advantages of a shallow network. We have proposed a novel cost function, whose minimisation results in a trained anomaly detector that only requires examples of the normal class at training time. The paper includes a large benchmark experiment that evaluates the performance of EBFDD network and compares it to state of the art one-class classification algorithms including the One-Class Support Vector Machine and the Isolation Forest. The experiments show that, overall, the EBFDD network outperforms the state of the art approaches.
    Scopus© Citations 1  319