Distributed dynamic partially stateful dataflow

• Main Author: Behrens, Jonathan (Jonathan Kyle)
• Other Authors: M. Frans Kaashoek and Malte Schwarzkopf.
• Format: Others
• Language: English
• Published: Massachusetts Institute of Technology 2018
• Subjects: Electrical Engineering and Computer Science.
• Online Access: http://hdl.handle.net/1721.1/118054

Thesis: S.M., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2018. === Cataloged from PDF version of thesis. === Includes bibliographical references (pages 43-49). === This thesis present a distributed implementation of Noria, a new streaming dataflow system that simplifies the infrastructure of read-heavy web applications by unifying the database, caching layer, and parts of application logic in a single system. Noria's partially-stateful dataflow allows it to evict and reconstruct state on demand, and avoid prior dataflow systems' restriction to windowed state. Unlike existing dataflow systems, Noria adapts on-line to schema and query changes, and shares state and computation across related queries to eliminate duplicate effort. Noria's distributed design enables it to leverage the compute power of an entire cluster while providing high availability thanks to its fault tolerant design. On a single machine, Noria already outperforms MySQL by up to 7 x, but when running across a cluster of machines, it can scale to tens of millions of reads and millions of writes per second. === by Jonathan Behrens. === S.M.