Diverging Divergences: Examining Variants of Jensen Shannon Divergence for Corpus Comparison Tasks

Title: Diverging Divergences: Examining Variants of Jensen Shannon Divergence for Corpus Comparison Tasks
Authors: Lu, JinghuiHenchion, MaeveMacNamee, Brian
Permanent link: http://hdl.handle.net/10197/12574
Date: 13-May-2020
Online since: 2021-10-26T11:13:30Z
Abstract: Jensen-Shannon divergence (JSD) is a distribution similarity measurement widely used in natural language processing. In corpus comparison tasks, where keywords are extracted to reveal the divergence between different corpora (for example, social media posts from proponents of different views on a political issue), two variants of JSD have emerged in the literature. One of these uses a weighting based on the relative sizes of the corpora being compared. In this paper we argue that this weighting is unnecessary and, in fact, can lead to misleading results. We recommend that this weighted version is not used. We base this recommendation on an analysis of the JSD variants and experiments showing how they impact corpus comparison results as the relative sizes of the corpora being compared change.
Funding Details: Science Foundation Ireland
Type of material: Conference Publication
Keywords: Corpus comparisonJensen-Shannon divergence
Other versions: https://lrec2020.lrec-conf.org/en/
Language: en
Status of Item: Peer reviewed
Conference Details: The 12th Language Resources and Evaluation Conference (LREC 2020), Marseille, France, 11-16 May 2020 (cancelled due to coronavirus outbreak)
This item is made available under a Creative Commons License: https://creativecommons.org/licenses/by/3.0/ie/
Appears in Collections:Computer Science Research Collection
Insight Research Collection

Show full item record

Page view(s)

Last Week
Last month
checked on May 21, 2022


checked on May 21, 2022

Google ScholarTM


If you are a publisher or author and have copyright concerns for any item, please email research.repository@ucd.ie and the item will be withdrawn immediately. The author or person responsible for depositing the article will be contacted within one business day.