BigDataNetSim: A Simulator for Data and Process Placement in Large Big Data Platforms

DC FieldValueLanguage
dc.contributor.authorBatista de Almeida, Leandro-
dc.contributor.authorCunha de Almeida, Eduardo-
dc.contributor.authorMurphy, John-
dc.contributor.authorDe Grande, Robson E.-
dc.contributor.authorVentresque, Anthony-
dc.date.accessioned2019-05-22T07:54:46Z-
dc.date.available2019-05-22T07:54:46Z-
dc.date.copyright2018 IEEEen_US
dc.date.issued2018-10-17-
dc.identifier.urihttp://hdl.handle.net/10197/10594-
dc.descriptionThe 2018 IEEE/ACM 22nd International Symposium on Distributed Simulation and Real Time Applications (DS-RT)en_US
dc.description.abstractBig Data platforms are convoluted distributed systems which commonly comprise skill- and labour-intensive solution development to treat inherent Big Data application challenges. Several tools have been proposed to help developers and engineers to overcome the involved complexities in coordinating the execution of plenty processes/threads on multiple machines. However, no work so far has been able to combine both an accurate representation of Big Data jobs and realistic modeling of the behaviour of Big Data platforms at scale, including networking elements and data and job placement. In this paper, we propose BigDataNetSim, the first simulator which models accurately all the main components of the data movements in Big Data platforms (e.g., HDFS, YARN/MapReduce, network topologies, switching/routing protocols) in a large scale system. BigDataNetSim can serve as a valuable tool for engineering Big Data solutions, which includes set-up of systems, prototyping of jobs, and improvement of components/algorithms for Big Data platforms. We also demonstrate that BigDataNetSim can simulate a real Hadoop cluster with a high degree of accuracy in terms of data and job placements, being able to scale up to very large systems.en_US
dc.language.isoenen_US
dc.publisherIEEEen_US
dc.relation.ispartofBesada, E., Polo. O.R., De Grande, R., Risco, J.L. (eds.). Proceedings of the 2018 IEEE/ACM 22nd International Symposium on Distributed Simulation and Real Time Applications (DS-RT) October 15-17, 2018, Madrid, Spainen_US
dc.rights© 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.en_US
dc.subjectBig dataen_US
dc.subjectHadoopen_US
dc.subjectSimulationen_US
dc.subjectTask analysisen_US
dc.subjectData modelsen_US
dc.subjectToolsen_US
dc.subjectNetwork topologyen_US
dc.subjectProtocolsen_US
dc.subjectYARNen_US
dc.titleBigDataNetSim: A Simulator for Data and Process Placement in Large Big Data Platformsen_US
dc.typeConference Publicationen_US
dc.internal.authorcontactotheranthony.ventresque@ucd.ieen_US
dc.internal.webversionshttp://ds-rt.com/2018/-
dc.statusNot peer revieweden_US
dc.identifier.doi10.1109/DISTRA.2018.8601018-
dc.identifier.doi978-1-5386-5048-6-
dc.neeo.contributorBatista de Almeida|Leandro|aut|-
dc.neeo.contributorCunha de Almeida|Eduardo|aut|-
dc.neeo.contributorMurphy|John|aut|-
dc.neeo.contributorDe Grande|Robson E.|aut|-
dc.neeo.contributorVentresque|Anthony|aut|-
dc.date.updated2019-02-09T00:14:09Z-
item.fulltextWith Fulltext-
item.grantfulltextopen-
Appears in Collections:Computer Science Research Collection
Files in This Item:
File Description SizeFormat 
SimulatorPaper2018.pdf610.9 kBAdobe PDFDownload
Show simple item record

SCOPUSTM   
Citations 50

2
Last Week
0
Last month
checked on Sep 11, 2020

Page view(s)

385
Last Week
2
Last month
15
checked on Oct 24, 2020

Download(s) 50

227
checked on Oct 24, 2020

Google ScholarTM

Check

Altmetric


This item is available under the Attribution-NonCommercial-NoDerivs 3.0 Ireland. No item may be reproduced for commercial purposes. For other possible restrictions on use please refer to the publisher's URL where this is made available, or to notes contained in the item itself. Other terms may apply.