BiDAl: Big Data Analyzer for Cluster Traces

Modern data centers that provide Internet-scale services are stadium-size structures housing tens of thousands of heterogeneous de- vices (server clusters, networking equipment, power and cooling infras- tructures) that must operate continuously and reliably. As part of their operation, these devices produce large amounts of data in the form of event and error logs that are essential not only for identifying problems but also for improving data center eciency and management. These activities employ data analytics and often exploit hidden statistical pat- terns and correlations among dierent factors present in the data. Un- covering these patterns and correlations is challenging due to the sheer volume of data to be analyzed. This paper presents BiDAl, a prototype \log-data analysis framework" that incorporates various Big Data tech- nologies to simplify the analysis of data traces from large clusters. BiDAl is written in Java with a modular and extensible architecture so that dif- ferent storage backends (currently, HDFS and SQLite are supported), as well as dierent analysis languages (current implementation supports SQL, R and Hadoop MapReduce) can be easily selected as appropriate. We present the design of BiDAl and describe our experience using it to analyze several public traces of Google data clusters for building a simulation model capable of reproducing observed behavior.

• Publication year

  2014

• Venue

  GI-Jahrestagung

• Publication date

  2014-10-06

• Fields of study

  Computer Science, Engineering

• Identifiers
  arXiv 1410.1309
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• R: A language and environment for statistical computing.

  2014cited by this paper
• CPI2: CPU performance isolation for shared compute clusters

  2013cited by this paper
• Scuba: Diving into Data at Facebook

  2013cited by this paper
• List of parameters

  2013cited by this paper
• The Failure Trace Archive: Enabling the comparison of failure measurements and models of distributed systems

  2013influential reference
• Characterizing Machines and Workloads on a Google Cluster

  2012cited by this paper
• Characterization and Comparison of Google Cloud Load versus Grids

  2012cited by this paper
• Heterogeneity and dynamicity of clouds at scale: Google trace analysis

  2012cited by this paper
• Design Insights for MapReduce from Diverse Production Workloads

  2012cited by this paper
• Characterization and Comparison of Cloud versus Grid Workloads

  2012cited by this paper
• Towards understanding heterogeneous clouds at scale : Google trace analysis

  2012cited by this paper
• The Case for Evaluating MapReduce Performance Using Workload Suites

  2011cited by this paper
• Characterizing Task Usage Shapes in Google Compute Clusters

  2011cited by this paper
• Modeling and synthesizing task placement constraints in Google compute clusters

  2011cited by this paper
• Towards Synthesizing Realistic Workload Traces for Studying the Hadoop Ecosystem

  2011cited by this paper
• Data warehousing and analytics infrastructure at facebook

  2010cited by this paper
• An Analysis of Traces from a Production MapReduce Cluster

  2010cited by this paper
• MapReduce: a flexible data processing tool

  2010cited by this paper
• Towards characterizing cloud backend workloads: insights from Google compute clusters

  2010cited by this paper
• A survey of online failure prediction methods

  2010cited by this paper
• The Hadoop Distributed File System

  2010cited by this paper
• Autopilot: automatic data center management

  2007cited by this paper

• Customization of virtual machine allocation policy using k-means clustering algorithm to minimize power consumption in data centers

  2017cites this paper
• Towards operator-less data centers through data-driven, predictive, proactive autonomics

  2016cites this paper
• Towards Data-Driven Autonomics in Data Centers

  2015cites this paper
• Analysis and Clustering of Workload in Google Cluster Trace Based on Resource Usage

  2015cites this paper
• Towards a systematic analysis of cluster computing log data: the case of IBM BlueGene/Q

  2014cites this paper
• A Holistic Approach to Log Data Analysis in High-Performance Computing Systems: The Case of IBM Blue Gene/Q

  2014cites this paper