The Power of Choice in Data-Aware Cluster Scheduling

No abstract is available for this paper.

• Publication year

  2014

• Venue

  USENIX Symposium on Operating Systems Design and Implementation

• Publication date

  2014-10-06

• Fields of study

  Computer Science

• Identifiers

  No identifiers available.

• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• MapReduce

  2020cited by this paper
• Fitness Function in ABC Algorithm for Uncapacitated Facility Location Problem

  2015cited by this paper
• Knowing when you're wrong: building fast and reliable approximate query processing systems

  2014cited by this paper
• Quasar: resource-efficient and QoS-aware cluster management

  2014cited by this paper
• Choosy: max-min fair sharing for datacenter jobs with constraints

  2013cited by this paper
• The Case for Tiny Tasks in Compute Clusters

  2013cited by this paper
• MLbase: A Distributed Machine-learning System

  2013cited by this paper
• Sparrow: distributed, low latency scheduling

  2013cited by this paper
• A The Quantcast File System

  2013cited by this paper
• Effective Straggler Mitigation: Attack of the Clones

  2013cited by this paper
• Leveraging endpoint flexibility in data-intensive clusters

  2013cited by this paper
• Surviving failures in bandwidth-constrained datacenters

  2012cited by this paper
• Large Scale Distributed Deep Networks

  2012cited by this paper
• BlinkDB: queries with bounded errors and bounded response times on very large data

  2012influential reference
• Shark: SQL and rich analytics at scale

  2012cited by this paper
• Coflow: a networking abstraction for cluster applications

  2012cited by this paper
• Early Accurate Results for Advanced Analytics on MapReduce

  2012cited by this paper
• Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads

  2012cited by this paper
• Blink and It's Done: Interactive Queries on Very Large Data

  2012cited by this paper
• PACMan: Coordinated Memory Caching for Parallel Jobs

  2012cited by this paper
• Distributed GraphLab: A Framework for Machine Learning in the Cloud

  2012cited by this paper
• PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs

  2012cited by this paper
• Advances in Neural Information Processing Systems 25

  2012cited by this paper
• Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

  2012cited by this paper
• Scarlett: coping with skewed content popularity in mapreduce clusters

  2011cited by this paper
• Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

  2011cited by this paper
• Dominant Resource Fairness: Fair Allocation of Multiple Resource Types

  2011cited by this paper
• Disk-Locality in Datacenter Computing Considered Irrelevant

  2011cited by this paper
• Online aggregation for large MapReduce jobs

  2011cited by this paper
• UvA-DARE ( Digital Academic Repository ) SciBORQ : scientific data management with bounds on runtime and quality

  2011cited by this paper
• Managing data transfers in computer clusters with orchestra

  2011cited by this paper
• Hedera: Dynamic Flow Scheduling for Data Center Networks

  2010cited by this paper
• Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling

  2010cited by this paper
• Large-Scale Machine Learning with Stochastic Gradient Descent

  2010cited by this paper
• Scale-Out Networking in the Data Center

  2010cited by this paper
• Reining in the Outliers in Map-Reduce Clusters using Mantri

  2010influential reference
• Wave Computing in the Cloud

  2009cited by this paper
• Quincy: fair scheduling for distributed computing clusters

  2009cited by this paper
• VL2: a scalable and flexible data center network

  2009cited by this paper
• Improving MapReduce Performance in Heterogeneous Environments

  2008cited by this paper
• Data management projects at Google

  2008cited by this paper
• A scalable, commodity data center network architecture

  2008cited by this paper
• A Stochastic Quasi-Newton Method for Online Convex Optimization

  2007cited by this paper
• The Tradeoffs of Large Scale Learning

  2007cited by this paper
• Large Language Models in Machine Translation

  2007cited by this paper
• Detecting Spam Blogs: A Machine Learning Approach

  2006cited by this paper
• Effective use of block-level sampling in statistics estimation

  2004cited by this paper
• Query Processing, Approximation, and Resource Management in a Data Stream Management System

  2003cited by this paper
• Query Processing, Resource Management, and Approximation ina Data Stream Management System

  2002cited by this paper
• The Power of Two Choices in Randomized Load Balancing

  2001cited by this paper
• Aqua: A Fast Decision Support Systems Using Approximate Query Answers

  1999influential reference
• Approximation algorithms for facility location problems (extended abstract)

  1997cited by this paper
• Online aggregation

  1997cited by this paper
• Optimal algorithms for approximate clustering

  1988cited by this paper
• Sampling theory of surveys, with applications

  1955cited by this paper
• This Paper Is Included in the Proceedings of the 11th Usenix Symposium on Networked Systems Design and Implementation (nsdi '14). Grass: Trimming Stragglers in Approximation Analytics Grass: Trimming Stragglers in Approximation Analytics

  year unknowncited by this paper

• Liberating the Data Aware Scheduler to Achieve Locality in Layered Scientific Workflow Systems

  2025cites this paper
• Design and Implementation of a Distributed Data Pipeline Framework for Multi-Hub Environments

  2025cites this paper
• A Survey of Fundamental Principles and Analysis for Job Scheduling on Warehouse-Scale Computers

  2025cites this paper
• Memory-aware parallelism setting algorithm in the spark platform

  2024cites this paper
• Distributed Data Analysis Workflow System across Multiple Data Hubs

  2024cites this paper
• Resource Allocation with Service Affinity in Large-Scale Cloud Environments

  2024cites this paper
• Cloud-Native Computing: A Survey From the Perspective of Services

  2023cites this paper
• A Survey on Spark Ecosystem: Big Data Processing Infrastructure, Machine Learning, and Applications (Extended abstract)

  2023cites this paper
• DAG Scheduling in Mobile Edge Computing

  2023cites this paper
• TaskVine: Managing In-Cluster Storage for High-Throughput Data Intensive Workflows

  2023cites this paper
• Performance Impact of Queue Sorting in Container-Based Application Scheduling

  2023cites this paper
• SiloD: A Co-design of Caching and Scheduling for Deep Learning Clusters

  2023cites this paper
• Serverless Computing: State-of-the-Art, Challenges and Opportunities

  2023cites this paper
• Extending and Programming the NVMe I/O Determinism Interface for Flash Arrays

  2022cites this paper
• Graph Deep Factors for Probabilistic Time-series Forecasting

  2022cites this paper
• Graph Deep Factors for Forecasting with Applications to Cloud Resource Allocation

  2021cites this paper
• Network Cost-Aware Geo-Distributed Data Analytics System

  2021cites this paper
• Llama: A Heterogeneous & Serverless Framework for Auto-Tuning Video Analytics Pipelines

  2021cites this paper
• A Data-Aware Scheduling Strategy for Executing Large-Scale Distributed Workflows

  2021cites this paper
• Endpoint-Flexible Coflow Scheduling Across Geo-Distributed Datacenters

  2020cites this paper
• An efficient deadline constrained and data locality aware dynamic scheduling framework for multitenancy clouds

  2020cites this paper
• Enhancing the power of two choices load balancing algorithm using round robin policy

  2020cites this paper
• Deadline Flow Scheduling in Datacenters with Time-Varying Bandwidth Allocations

  2020cites this paper
• Sol: Fast Distributed Computation Over Slow Networks

  2020cites this paper
• A Network Cost-aware Geo-distributed Data Analytics System

  2020cites this paper
• Cost-Aware Partitioning for Efficient Large Graph Processing in Geo-Distributed Datacenters

  2020cites this paper
• Graph Deep Factors for Forecasting

  2020cites this paper
• Mitigating the Performance-Efficiency Tradeoff in Resilient Memory Disaggregation

  2019cites this paper
• ApproxSSD: Data Layout Aware Sampling on an Array of SSDs

  2019cites this paper
• Gapprox: using Gallup approach for approximation in Big Data processing

  2019cites this paper
• Intelligent Scheduling for Parallel Jobs in Big Data Processing Systems

  2019influential citation
• Terra: Scalable Cross-Layer GDA Optimizations

  2019cites this paper
• THE UNIVERSITY OF CHICAGO APPROXIMATION ALGORITHMS FOR CAPACITATED K-MEDIAN AND SCHEDULING WITH RESOURCE AND PRECEDENCE CONSTRAINTS A DISSERTATION SUBMITTED TO THE FACULTY OF THE DIVISION OF THE PHYSICAL SCIENCES IN CANDIDACY FOR THE DEGREE OF DOCTOR OF PHILOSOPHY

  2019cites this paper
• ReLAQS: Reducing Latency for Multi-Tenant Approximate Queries via Scheduling

  2019cites this paper
• Centralized Core-granular Scheduling for Serverless Functions

  2019cites this paper
• Data-Driven Intelligent Scheduling For Long Running Workloads In Large-Scale Datacenters

  2019influential citation
• Shadow: Exploiting the Power of Choice for Efficient Shuffling in MapReduce

  2019influential citation
• MEER: Online Estimation of Optimal Memory Reservations for Long Lived Containers in In-Memory Cluster Computing

  2019cites this paper
• Architecting a Cloud Data Platform: Bridging Storage and Compute for Enhanced Data Processing

  2019cites this paper
• A novel disk I/O scheduling framework of virtualized storage system

  2019cites this paper
• Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads

  2019cites this paper
• Interactive Data Exploration of Distributed Raw Files: A Systematic Mapping Study

  2019cites this paper
• A Generic Construction for All Parameters in Minimum Storage Regenerating Codes

  2019cites this paper
• Flexible and efficient computation in large data centres

  2018cites this paper
• A Survey on Spark Ecosystem: Big Data Processing Infrastructure, Machine Learning, and Applications

  2018cites this paper
• EC-Store: Bridging the Gap between Storage and Latency in Distributed Erasure Coded Systems

  2018cites this paper
• Improving the Effectiveness of Burst Buffers for Big Data Processing in HPC Systems with Eley

  2018cites this paper
• VideoEdge: Processing Camera Streams using Hierarchical Clusters

  2018cites this paper
• Leveraging Endpoint Flexibility when Scheduling Coflows across Geo-distributed Datacenters

  2018cites this paper
• A Divide and Conquer Algorithm for DAG Scheduling under Power Constraints

  2018cites this paper
• A Survey on Spark Ecosystem for Big Data Processing

  2018influential citation
• Joint Scheduling of Tasks and Network Flows in Big Data Clusters

  2018cites this paper
• Operating system support for warehouse-scale computing

  2018cites this paper
• Big Data, Data Streaming, and the Mobile Cloud

  2018cites this paper
• Data and container placement in scalable data analytics platforms

  2018cites this paper
• MeteCloud: Meteorological Cloud Computing Platform for Mobile Weather Forecasts based on Energy-aware Scheduling

  2018cites this paper
• MixHeter: A global scheduler for mixed workloads in heterogeneous environments

  2018cites this paper
• Wide-area analytics with multiple resources

  2018cites this paper
• Riffle: optimized shuffle service for large-scale data analytics

  2018cites this paper
• Multi-tenant GPU Clusters for Deep Learning Workloads: Analysis and Implications

  2018cites this paper
• Bridging the Gap between Application and Solid-State-Drives

  2018cites this paper
• SLAQ: quality-driven scheduling for distributed machine learning

  2017cites this paper
• PBSE: a robust path-based speculative execution for degraded-network tail tolerance in data-parallel frameworks

  2017cites this paper
• Efficient Memory Disaggregation with Infiniswap

  2017cites this paper
• Eley: On the Effectiveness of Burst Buffers for Big Data Processing in HPC Systems

  2017cites this paper
• High-Performance Systems for Crowdsourced Data Analysis

  2017cites this paper
• Cioppino: Multi-Tenant Crowd Management

  2017cites this paper
• System Design for Large Scale Machine Learning

  2017influential citation
• Pandas: Robust Locality-Aware Scheduling With Stochastic Delay Optimality

  2017cites this paper
• Design and analysis of a new distributed scheduling approach

  2017cites this paper
• Live Video Analytics at Scale with Approximation and Delay-Tolerance

  2017cites this paper
• Reducing Randomization in the Power of Two Choices Load Balancing Algorithm

  2017cites this paper
• Meeting deadlines for approximation processing in MapReduce environments

  2017cites this paper
• DeMS: A hybrid scheme of task scheduling and load balancing in computing clusters

  2017cites this paper
• Optimizing the Performance of Data Analytics Frameworks

  2017cites this paper
• Shadow: Exploiting the Power of Choice for Efficient Shuffling in MapReduce

  2017influential citation
• THE BERKELEY DATA ANALYSIS SYSTEM (BDAS): AN OPEN SOURCE PLATFORM FOR BIG DATA ANALYTICS

  2017cites this paper
• Enabling fast failure recovery in shared Hadoop clusters: Towards failure-aware scheduling

  2017cites this paper
• Execution of Recursive Queries in Apache Spark

  2017cites this paper
• Efficient Big Data Processing on Large-Scale Shared Platforms: Managing I/Os and Failures. (Sur l'efficacité des traitements Big Data sur les plateformes partagées à grande échelle: gestion des entrées-sorties et des pannes)

  2017cites this paper
• ApproxSSD : Fast Data Sampling on SSD Arrays

  2016cites this paper
• Unobtrusive Storage Maintenance

  2016cites this paper
• Chronos: Meeting coflow deadlines in data center networks

  2016cites this paper
• Erasure Coding for Big-data Systems: Theory and Practice

  2016cites this paper
• Custody: Towards Data-Aware Resource Sharing in Cloud-Based Big Data Processing

  2016influential citation
• Increasing large-scale data center capacity by statistical power control

  2016cites this paper
• Optimizing Hash-based Distributed Storage Using Client Choices

  2016cites this paper
• An Efficient MapReduce Computing Model for Imprecise Applications

  2016cites this paper
• Towards An Efficient Cloud Computing System: Data Management, Resource Allocation and Job Scheduling

  2016cites this paper
• When Do Redundant Requests Reduce Latency?

  2016cites this paper
• Storage Load Control Through Meta-Scheduler Using Predictive Analytics

  2016cites this paper
• SIS: A scheme for dynamic independent task scheduling in a cloud environment

  2016cites this paper
• Getafix : Workload-aware Data Management in Lookback Processing Systems

  2016cites this paper
• Towards Practical and Near-Optimal Coflow Scheduling for Data Center Networks

  2016cites this paper
• TetriSched: global rescheduling with adaptive plan-ahead in dynamic heterogeneous clusters

  2016cites this paper
• Symbiosis: Network-aware task scheduling in data-parallel frameworks

  2016influential citation
• Speeding up distributed storage and computing systems using codes

  2016cites this paper
• Ernest: Efficient Performance Prediction for Large-Scale Advanced Analytics

  2016cites this paper
• Research Statement Systems for Efficient Execution

  2016cites this paper
• Workload-Aware Scheduling Across Geo-distributed Data Centers

  2016cites this paper