SupMR: Circumventing Disk and Memory Bandwidth Bottlenecks for Scale-up MapReduce

TitleSupMR: Circumventing Disk and Memory Bandwidth Bottlenecks for Scale-up MapReduce
Publication TypeConference Paper
Year of Publication2014
AuthorsSevilla, M, Nassi, I, Ioannidou, K, Brandt, SA, Maltzahn, C
Conference NameLSPP'14 (in conjunction with IPDPS 2014)
Date Published05/2014
PublisherIEEE
Conference LocationPhoenix, AZ
Abstract

Reading input from primary storage (i.e. the ingest phase) and aggregating results (i.e. the merge phase) are important pre- and post-processing steps in large batch computations. Unfortunately, today’s data sets are so large that the ingest and merge job phases are now performance bottlenecks. In this paper, we mitigate the ingest and merge bottlenecks by leveraging the scale-up MapReduce model. We introduce an ingest chunk pipeline and a merge optimization that increases CPU utilization (50 - 100%) and job phase speedups (1.16× - 3.13×) for the ingest and merge phases. Our techniques are based on well-known algorithms and scale-out MapReduce optimizations, but applying them to a scale-up computation framework to mitigate the ingest and merge bottlenecks is novel.

AttachmentSize
PDF icon sevilla-lspp14.pdf908.97 KB