Efficient shuffle management with SCache for DAG computing frameworks

No abstract is available for this paper.

• Publication year

  2018

• Venue

  ACM SIGPLAN Symposium on Principles & Practice of Parallel Programming

• Publication date

  2018-02-10

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/3178487.3178510
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Apache Hadoop

  2019cited by this paper
• Analysis of TPC-DS: the first standard benchmark for SQL-based big data systems

  2017cited by this paper
• SYNC or ASYNC: time to fuse for distributed graph-parallel computation

  2015cited by this paper
• Efficient Coflow Scheduling Without Prior Knowledge

  2015cited by this paper
• Apache Tez: A Unifying Framework for Modeling and Building Data Processing Applications

  2015cited by this paper
• Making Sense of Performance in Data Analytics Frameworks

  2015cited by this paper
• Efficient coflow scheduling with Varys

  2014cited by this paper
• Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks

  2014cited by this paper
• ShuffleWatcher: Shuffle-aware Scheduling in Multi-tenant MapReduce Clusters

  2014cited by this paper
• Improving MapReduce performance in heterogeneous environments with adaptive task tuning

  2014cited by this paper
• DynMR: dynamic MapReduce with ReduceTask interleaving and MapTask backfilling

  2014cited by this paper
• Preemptive ReduceTask Scheduling for Fair and Fast Job Completion

  2013cited by this paper
• SkewTune: mitigating skew in mapreduce applications

  2012cited by this paper
• Load Balancing in MapReduce Based on Scalable Cardinality Estimates

  2012cited by this paper
• PACMan: Coordinated Memory Caching for Parallel Jobs

  2012cited by this paper
• Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

  2012cited by this paper
• Managing data transfers in computer clusters with orchestra

  2011cited by this paper
• Resource Provisioning Framework for MapReduce Jobs with Performance Goals

  2011cited by this paper
• The Design of Approximation Algorithms

  2011cited by this paper
• Starfish: A Self-tuning System for Big Data Analytics

  2011cited by this paper
• Fast crash recovery in RAMCloud

  2011cited by this paper
• iHadoop: Asynchronous Iterations for MapReduce

  2011cited by this paper
• Reining in the Outliers in Map-Reduce Clusters using Mantri

  2010cited by this paper
• Towards automatic optimization of MapReduce programs

  2010cited by this paper
• ZooKeeper: Wait-free Coordination for Internet-scale Systems

  2010cited by this paper
• Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling

  2010cited by this paper
• On Availability of Intermediate Data in Cloud Computations

  2009cited by this paper
• Quincy: fair scheduling for distributed computing clusters

  2009cited by this paper
• Dryad: distributed data-parallel programs from sequential building blocks

  2007cited by this paper
• Distributed caching with memcached

  2004cited by this paper
• The Google file system

  2003cited by this paper
• Random sampling with a reservoir

  1985cited by this paper
• A Study of Replacement Algorithms for Virtual-Storage Computer

  1966cited by this paper

• FuxiShuffle: An Adaptive and Resilient Shuffle Service for Distributed Data Processing on Alibaba Cloud

  2026influential citation
• Decision Shuffle: Efficient Pre-scheduling System for Push-based Shuffle in DAG Computing Frameworks

  2025influential citation
• Turbo: Efficient Communication Framework for Large-scale Data Processing Cluster

  2024cites this paper
• Using Vectorized Execution to Improve SQL Query Performance on Spark

  2021cites this paper
• Efficient shuffle management for DAG computing frameworks based on the FRQ model

  2021cites this paper
• OPS: Optimized Shuffle Management System for Apache Spark

  2020cites this paper
• Shadow: Exploiting the Power of Choice for Efficient Shuffling in MapReduce

  2019cites this paper
• H-Scheduler: Storage-Aware Task Scheduling for Heterogeneous-Storage Spark Clusters

  2018cites this paper