Optimal Component Composition for Scalable Stream Processing

Stream processing has become increasingly important with emergence of stream applications such as audio/video surveillance, stock price tracing, and sensor data analysis. A challenging problem is to provide optimal component composition in a distributed stream processing environment. The goal of optimal component composition is to achieve load balancing subject to multiple function, resource, and quality-of-service (QoS) constraints while composing stream applications. In this paper, we present an adaptive composition probing (ACP) approach to the problem. Different from previous work, ACP provides a new hybrid approach that combines distributed composition probing with coarse-grain global state management. Guided by the coarse-grain global state information, ACP selectively probes a subset of candidate components to discover an approximately optimal component composition. Further, ACP is self-tuning, which can adoptively adjust the number of probes to maintain a specified composition performance target (i.e., composition success rate) in a dynamic stream environment. While the optimal component composition problem is NP-hard, our ACP approach provides an adaptive polynomial approximation solution. We have conducted extensive simulation experiments to show the efficiency, scalability, and adaptability of the ACP approach by comparing with other alternative solutions

• Publication year

  2005

• Venue

  IEEE International Conference on Distributed Computing Systems

• Publication date

  2005-06-06

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1109/ICDCS.2005.59
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Distributed multimedia service composition with statistical QoS assurances

  2006cited by this paper
• SpiderNet: an integrated peer-to-peer service composition framework

  2004cited by this paper
• Contract-Based Load Management in Federated Distributed Systems

  2004cited by this paper
• sFlow: towards resource-efficient and agile service federation in service overlay networks

  2004cited by this paper
• QoS-assured service composition in managed service overlay networks

  2003cited by this paper
• Load balancing and stability issues in algorithms for service composition

  2003cited by this paper
• Load Shedding in a Data Stream Manager

  2003cited by this paper
• Flux: an adaptive partitioning operator for continuous query systems

  2003cited by this paper
• Inet-3.0: Internet Topology Generator

  2002cited by this paper
• Infopipes for composing distributed information flows

  2001cited by this paper
• Knit: component composition for systems software

  2000cited by this paper
• The click modular router

  2000cited by this paper
• The NP-Completeness Column: An Ongoing Guide

  1982cited by this paper
• Computers and In stractability: A Guide to the Theory of NP-Completeness. W. H Freeman, San Fran

  1979influential reference
• Johnson: computers and intractability: a guide to the theory of np- completeness (freeman

  1979cited by this paper

• Parallelizing Stream Compression for IoT Applications on Asymmetric Multicores

  2023cites this paper
• Runtime Adaptation of Data Stream Processing Systems: The State of the Art

  2022cites this paper
• Network-wide complex event processing over geographically distributed data sources

  2020cites this paper
• A Practical Verification of Protocol and Data Format Negotiation Methods in ComSS Platform

  2016cites this paper
• LASEC: A Localized Approach to Service Composition in Pervasive Computing Environments

  2015cites this paper
• A Logic Framework for Flexible and Security-Aware Service Composition

  2013cites this paper
• Planning for composition of eHealth services

  2013cites this paper
• Resource management using pattern-based prediction to address bursty data streams

  2013cites this paper
• Universal Platform for Composite Data Stream Processing Services Management

  2013cites this paper
• Planning-Based Method for Communication Protocol Negotiation in a Composition of Data Stream Processing Services

  2013cites this paper
• Communication Protocol Negotiation in a Composite Data Stream Processing Service

  2013cites this paper
• Concepts and algorithms for efficient distributed processing of data streams

  2013cites this paper
• RADAR: Adaptive Rate Allocation in Distributed Stream Processing Systems under Bursty Workloads

  2012cites this paper
• Fulfilling end-to-end latency constraints in large-scale streaming environments

  2011cites this paper
• Operator placement with QoS constraints for distributed stream processing

  2011cites this paper
• Automated software and hardware evolution analysis for distributed real-time and embedded systems

  2011cites this paper
• Configuration and deployment derivation strategies for distributed real-time and embedded systems

  2011cites this paper
• Placement of replicated tasks for distributed stream processing systems

  2010cites this paper
• Providing QoS Guarantees in Large-Scale Operator Networks

  2010cites this paper
• Adaptive component composition and load balancing for distributed stream processing applications

  2009influential citation
• MDA-based Configuration of Distributed Real-time and Embedded Systems

  2009cites this paper
• QoS-Aware Shared Component Composition for Distributed Stream Processing Systems

  2009cites this paper
• Automating Hardware and Software Evolution Analysis

  2009cites this paper
• Towards secure dataflow processing in open distributed systems

  2009cites this paper
• Accommodating bursts in distributed stream processing systems

  2009cites this paper
• Biologically-Inspired Distributed Middleware Management for Stream Processing Systems

  2008cites this paper
• Hot-spot prediction and alleviation in distributed stream processing applications

  2008cites this paper
• Synergy: Quality of service support for distributed stream processing systems

  2008cites this paper
• Alleviating HotSpots in Peer-to-Peer Stream Processing Environments

  2007cites this paper
• Synergy : A Distributed Stream Processing Middleware

  2007cites this paper
• BridgeNet: An Adaptive Multi-Source Stream Dissemination Service Overlay

  2007cites this paper
• BridgeNet: An Adaptive Multi-Source Stream Dissemination Overlay Network

  2007cites this paper
• RASC: Dynamic Rate Allocation for Distributed Stream Processing Applications

  2007cites this paper
• Self-Configuring Information Management for Large-Scale Service Overlays

  2007cites this paper
• Design and Implementation of a Middleware for Sentient Spaces

  2007cites this paper
• ViCo: an adaptive distributed video correlation system

  2006cites this paper
• Synergy: Sharing-Aware Component Composition for Distributed Stream Processing Systems

  2006influential citation
• On Composing Stream Applications in Peer-to-Peer Environments

  2006cites this paper
• Load balancing techniques for distributed stream processing applications in overlay environments

  2006cites this paper
• Service Clouds : A Distributed Infrastructure for Composing Autonomic Communication Services ∗

  2005cites this paper
• Adaptive Load Diffusion for Stream Joins

  2005cites this paper
• Fulfilling End-to-End Latency Constraints in Large-scale Streaming Environments

  year unknowncites this paper
• ΟΙΚΟΝΟΜΙΚΟ ΠΑΝΕΠΙΣΤΗΜΙΟ ΑΘΗΝΩΝ ΠΡΟΓΡΑΜΜΑ ΜΕΤΑΠΤΥΧΙΑΚΩΝ ΣΠΟΥ∆ΩΝ ΣΤΗΝ ΕΠΙΣΤΗΜΗ ΤΩΝ ΥΠΟΛΟΓΙΣΤΩΝ ∆ιπλωματική Εργασία Μεταπτυχιακού ∆ιπλώματος Ειδίκευσης «Adaptive Rate Allocation in Distributed Stream Processing Systems under Bursty Workloads»

  year unknowncites this paper