BigDataNetSim: A Simulator for Data and Process Placement in Large Big Data Platforms
|Title:||BigDataNetSim: A Simulator for Data and Process Placement in Large Big Data Platforms||Authors:||Batista de Almeida, Leandro; Cunha de Almeida, Eduardo; Murphy, John; De Grande, Robson E.; Ventresque, Anthony||Permanent link:||http://hdl.handle.net/10197/10594||Date:||17-Oct-2018||Online since:||2019-05-22T07:54:46Z||Abstract:||Big Data platforms are convoluted distributed systems which commonly comprise skill- and labour-intensive solution development to treat inherent Big Data application challenges. Several tools have been proposed to help developers and engineers to overcome the involved complexities in coordinating the execution of plenty processes/threads on multiple machines. However, no work so far has been able to combine both an accurate representation of Big Data jobs and realistic modeling of the behaviour of Big Data platforms at scale, including networking elements and data and job placement. In this paper, we propose BigDataNetSim, the first simulator which models accurately all the main components of the data movements in Big Data platforms (e.g., HDFS, YARN/MapReduce, network topologies, switching/routing protocols) in a large scale system. BigDataNetSim can serve as a valuable tool for engineering Big Data solutions, which includes set-up of systems, prototyping of jobs, and improvement of components/algorithms for Big Data platforms. We also demonstrate that BigDataNetSim can simulate a real Hadoop cluster with a high degree of accuracy in terms of data and job placements, being able to scale up to very large systems.||Type of material:||Conference Publication||Publisher:||IEEE||Copyright (published version):||2018 IEEE||Keywords:||Big data; Hadoop; Simulation; Task analysis; Data models; Tools; Network topology; Protocols; YARN||DOI:||10.1109/DISTRA.2018.8601018
|Other versions:||http://ds-rt.com/2018/||Language:||en||Status of Item:||Not peer reviewed||Is part of:||Besada, E., Polo. O.R., De Grande, R., Risco, J.L. (eds.). Proceedings of the 2018 IEEE/ACM 22nd International Symposium on Distributed Simulation and Real Time Applications (DS-RT) October 15-17, 2018, Madrid, Spain||Conference Details:||The 2018 IEEE/ACM 22nd International Symposium on Distributed Simulation and Real Time Applications (DS-RT)|
|Appears in Collections:||Computer Science Research Collection|
Show full item record
This item is available under the Attribution-NonCommercial-NoDerivs 3.0 Ireland. No item may be reproduced for commercial purposes. For other possible restrictions on use please refer to the publisher's URL where this is made available, or to notes contained in the item itself. Other terms may apply.