Options
Minimizing Network Traffic for Distributed Joins Using Lightweight Locality-Aware Scheduling
Author(s)
Date Issued
2018-08-31
Date Available
2019-04-18T11:48:29Z
Abstract
Large computing systems such as data centers are becoming the mainstream infrastructures for big data processing. As one of the key data operators in such scenarios, distributed joins is still challenging current techniques since it always incurs a significant cost on network communication. Various advanced approaches have been proposed to improve the performance, however, most of them just focus on data skew handling, and algorithms designed specifically for communication reduction have received less attention. Moreover, although the state-of-the-art technique can minimize network traffic, it provides fine-grained optimal schedules for all individual join keys, which could result in obvious overhead. In this paper, we propose a new approach called LAS (Lightweight Locality-Aware Scheduling), which targets reducing network communication for large distributed joins in an efficient and effective manner. We present the detailed design and implementation of LAS, and conduct an experimental evaluation using large data joins. Our results show that LAS can effectively reduce scheduling overhead and achieve comparable performance on network reduction compared to the state-of-the-art.
Sponsorship
European Commission Horizon 2020
Type of Material
Conference Publication
Publisher
Euro-Par
Start Page
293
End Page
305
Copyright (Published Version)
2018 Springer Nature Switzerland AG
Web versions
Language
English
Status of Item
Unspecified
Part of
Aldinucci, M., Padovani, L., Torquati, M. (eds.). Euro-Par 2018: Parallel Processing 24th International Conference on Parallel and Distributed Computing, Turin, Italy, August 27 - 31, 2018, Proceedings
Conference Details
The 24th International European Conference on Parallel and Distributed Computing (EURO-PAR 2018), Turin, Italy, 27-31 2018
ISBN
978-3-319-96983-1
This item is made available under a Creative Commons License
File(s)
Owning collection
Views
931
Last Week
1
1
Last Month
2
2
Acquisition Date
Mar 28, 2024
Mar 28, 2024
Downloads
515
Last Week
3
3
Last Month
7
7
Acquisition Date
Mar 28, 2024
Mar 28, 2024