Multi-Layer-Mesh: A Novel Topology and SDN-based Path Switching for Big Data Cluster Networks
|Title:||Multi-Layer-Mesh: A Novel Topology and SDN-based Path Switching for Big Data Cluster Networks||Authors:||Batista de Almeida, Leandro; Magoni, Damien; Perry, Philip; Cunha de Almeida, Eduardo; Murphy, John; Ventresque, Anthony||Permanent link:||http://hdl.handle.net/10197/10932||Date:||24-May-2019||Online since:||2019-07-22T09:25:35Z||Abstract:||Big Data technologies and tools have being used for the past decade to solve several scientific and industry problems, with Hadoop/YARN becoming the ”de facto” standard for these applications, although other technologies run on top of it. As any other distributed application, those big data technologies rely heavily on the network infrastructure to read and move data from hundreds or thousands of cluster nodes. Although these technologies are based on reliable and efficient distributed algorithms, there are scenarios and conditions that can generate bottlenecks and inefficiencies, i.e., when a high number of concurrent users creates data access contention. In this paper, we propose a novel network topology called MultiLayer-Mesh and a path switching algorithm based on SDN, that can increase the performance of a big data cluster while reducing the amount of utilized resources (network equipment), in turn reducing the energy and cooling consumption. A thorough simulation-based evaluation of our algorithms shows an average improvement in performance of 31.77% and an average decrease in resource utilization of 36.03% compared to a traditional SpineLeaf topology, in the selected test scenarios.||Funding Details:||Science Foundation Ireland||Type of material:||Conference Publication||Publisher:||IEEE||Copyright (published version):||2019 IEEE||Keywords:||Big data; Hadoop; Network topology; SDN||DOI:||10.1109/icc.2019.8761785||Other versions:||https://icc2019.ieee-icc.org/||Language:||en||Status of Item:||Not peer reviewed||Is part of:||ICC 2019 - 2019 IEEE International Conference on Communications (ICC)||Conference Details:||IEEE ICC 2019: IEEE International Conference on Communications, Shanghai, China, 20-24 May 2019|
|Appears in Collections:||Computer Science Research Collection|
Show full item record
This item is available under the Attribution-NonCommercial-NoDerivs 3.0 Ireland. No item may be reproduced for commercial purposes. For other possible restrictions on use please refer to the publisher's URL where this is made available, or to notes contained in the item itself. Other terms may apply.