BigDataNetSim: A Simulator for Data and Process Placement in Large Big Data Platforms

Files in This Item:
File Description SizeFormat 
SimulatorPaper2018.pdf610.9 kBAdobe PDFDownload
Title: BigDataNetSim: A Simulator for Data and Process Placement in Large Big Data Platforms
Authors: Batista de Almeida, Leandro
Cunha de Almeida, Eduardo
Murphy, John
De Grande, Robson E.
Ventresque, Anthony
Permanent link: http://hdl.handle.net/10197/10594
Date: 17-Oct-2018
Online since: 2019-05-22T07:54:46Z
Abstract: Big Data platforms are convoluted distributed systems which commonly comprise skill- and labour-intensive solution development to treat inherent Big Data application challenges. Several tools have been proposed to help developers and engineers to overcome the involved complexities in coordinating the execution of plenty processes/threads on multiple machines. However, no work so far has been able to combine both an accurate representation of Big Data jobs and realistic modeling of the behaviour of Big Data platforms at scale, including networking elements and data and job placement. In this paper, we propose BigDataNetSim, the first simulator which models accurately all the main components of the data movements in Big Data platforms (e.g., HDFS, YARN/MapReduce, network topologies, switching/routing protocols) in a large scale system. BigDataNetSim can serve as a valuable tool for engineering Big Data solutions, which includes set-up of systems, prototyping of jobs, and improvement of components/algorithms for Big Data platforms. We also demonstrate that BigDataNetSim can simulate a real Hadoop cluster with a high degree of accuracy in terms of data and job placements, being able to scale up to very large systems.
Type of material: Conference Publication
Publisher: IEEE
Copyright (published version): 2018 IEEE
Keywords: Big dataHadoopSimulationTask analysisData modelsToolsNetwork topologyProtocolsYARN
DOI: 10.1109/DISTRA.2018.8601018
978-1-5386-5048-6
Other versions: http://ds-rt.com/2018/
Language: en
Status of Item: Not peer reviewed
Is part of: Besada, E., Polo. O.R., De Grande, R., Risco, J.L. (eds.). Proceedings of the 2018 IEEE/ACM 22nd International Symposium on Distributed Simulation and Real Time Applications (DS-RT) October 15-17, 2018, Madrid, Spain
Conference Details: The 2018 IEEE/ACM 22nd International Symposium on Distributed Simulation and Real Time Applications (DS-RT)
Appears in Collections:Computer Science Research Collection

Show full item record

Google ScholarTM

Check

Altmetric


This item is available under the Attribution-NonCommercial-NoDerivs 3.0 Ireland. No item may be reproduced for commercial purposes. For other possible restrictions on use please refer to the publisher's URL where this is made available, or to notes contained in the item itself. Other terms may apply.