Multi-Layer-Mesh: A Novel Topology and SDN-based Path Switching for Big Data Cluster Networks

Files in This Item:
File Description SizeFormat 
ICC_2019___copy.pdf481.59 kBAdobe PDFDownload
Title: Multi-Layer-Mesh: A Novel Topology and SDN-based Path Switching for Big Data Cluster Networks
Authors: Batista de Almeida, LeandroMagoni, DamienPerry, PhilipCunha de Almeida, EduardoMurphy, JohnVentresque, Anthony
Permanent link: http://hdl.handle.net/10197/10932
Date: 24-May-2019
Online since: 2019-07-22T09:25:35Z
Abstract: Big Data technologies and tools have being used for the past decade to solve several scientific and industry problems, with Hadoop/YARN becoming the ”de facto” standard for these applications, although other technologies run on top of it. As any other distributed application, those big data technologies rely heavily on the network infrastructure to read and move data from hundreds or thousands of cluster nodes. Although these technologies are based on reliable and efficient distributed algorithms, there are scenarios and conditions that can generate bottlenecks and inefficiencies, i.e., when a high number of concurrent users creates data access contention. In this paper, we propose a novel network topology called MultiLayer-Mesh and a path switching algorithm based on SDN, that can increase the performance of a big data cluster while reducing the amount of utilized resources (network equipment), in turn reducing the energy and cooling consumption. A thorough simulation-based evaluation of our algorithms shows an average improvement in performance of 31.77% and an average decrease in resource utilization of 36.03% compared to a traditional SpineLeaf topology, in the selected test scenarios.
Funding Details: Science Foundation Ireland
Type of material: Conference Publication
Publisher: IEEE
Copyright (published version): 2019 IEEE
Keywords: Big dataHadoopNetwork topologySDN
DOI: 10.1109/icc.2019.8761785
Other versions: https://icc2019.ieee-icc.org/
Language: en
Status of Item: Not peer reviewed
Is part of: ICC 2019 - 2019 IEEE International Conference on Communications (ICC)
Conference Details: IEEE ICC 2019: IEEE International Conference on Communications, Shanghai, China, 20-24 May 2019
Appears in Collections:Computer Science Research Collection

Show full item record

Google ScholarTM

Check

Altmetric


This item is available under the Attribution-NonCommercial-NoDerivs 3.0 Ireland. No item may be reproduced for commercial purposes. For other possible restrictions on use please refer to the publisher's URL where this is made available, or to notes contained in the item itself. Other terms may apply.