Handling Tradeoffs between Performance and Query-Result Quality in Data Stream Processing

Date

Nov 28, 2017

Time

9:00 AM - 10:00 AM

Speaker

M.C.S. Yuanzhen Ji

Affiliation

Institut für Systemarchitektur, Professur für Systems Engineering

Language

en

Main Topic

Informatik

Other Topics

Informatik

Description

Data streams in the form of potentially unbounded sequences of tuples arise naturally in a large variety of domains including finance markets, sensor networks, social media, and network traffic management. The increasing number of applications that require processing data streams with high throughput and low latency have promoted the development of data stream processing systems (DSPS). A DSPS processes data stream with continuous queries, which are issued once and return query results to users continuously as new tuples arrive. For stream-based applications, both the query-execution performance (in terms of, e.g., throughput and end-to-end latency) and the quality of produced query results (in terms of, e.g., accuracy and completeness) are important. However, a DSPS often needs to make tradeoffs between these two requirements, either because of the data imperfection within streams, or because of the limited computation capacity of the DSPS itself. Performance versus result-quality tradeoffs caused by data imperfection are inevitable, because the quality of the incoming data is beyond the control of the system, whereas tradeoffs caused by system limitations can be alleviated, even erased, by enhancing the system itself. This dissertation seeks to advance the state of the art on handling the performance versus result-quality tradeoffs in data stream processing caused by the above two aspects of reasons. For tradeoffs caused by data imperfections, this dissertation focuses on the typical data-imperfection problem of stream-disorder and proposes the concept of quality-driven disorder handling (QDDH). QDDH enables making flexible and user-configurable tradeoffs between the end-to-end latency and the query-result quality in face of stream disorder. Moreover, compare to existing disorder handling approaches, QDDH can significantly reduce the end-to-end latency, and at the same time provide users with desired query-result quality. In this dissertation, a generic buffer-based QDDH framework and three instantiations of the generic framework for distinct query types are presented. For tradeoffs caused by system limitations, this dissertation proposes a system-enhancement approach that combines the row-oriented and the column-oriented data layout and processing techniques in data stream processing to improve the throughput. To fully exploit the potential of such hybrid execution of continuous queries, a static cost-based optimizer is introduced, which works at the operator level and takes the unique property of execution plans of continuous queries, feasibility, into account.

Last modified: Nov 28, 2017, 8:51:33 AM