Efficient querying for analytics on Internet of Things databases and streams

• Main Author: Siow, Eugene
• Other Authors: Tiropanis, Athanassios
• Published: University of Southampton 2018
• Online Access: https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.736776

This thesis is concerned with the development of efficient methods for managing contextualised time-series data and event streams produced by the Internet of Things (IoT) so that both historical and real-time information can be utilised to generate value within analytical applications. From a database systems perspective, two conflicting challenges motivate this research, interoperability and performance. IoT applications integrating streams of time-series data from heterogeneous IoT agents require a level of semantic interoperability. This semantic interoperability can be achieved with a common flexible data model that represents both metadata and data. However, applications might also have time constraints or require processing to be performed on large volumes of historical and streaming time-series data, possibly on resource-constrained platforms, without significant delay. Obtaining good performance is complicated by the complexity of the data model. In the first part of the thesis, a graph data model is shown to support the representation of metadata and data that various research and standard bodies are working towards, while the ‘volume’ of IoT data is shown to exhibit flat, wide and numerical characteristics. A three step abstraction is defined to reconcile queries on the graph model with efficient underlying storage by query translation. This storage is iteratively improved to exploit the character of time-series IoT data, achieving orders of magnitude performance improvement over state-of-the-art commercial, open-source and research databases. The second part of the thesis extends this abstraction to efficiently process real-time IoT streams continuously and proposes an infrastructure for fog computing that shows how resource-constrained platforms close to source IoT agents can co-operatively orchestrate stream processing. The main contributions of this thesis are therefore, i) a novel interoperable and performant abstraction for querying IoT graph representations, ii) high performance historical, streaming and fog computing time-series database implementations and iii) analytical applications and platforms built on this abstraction that act as practical models for the socio-technical development of the IoT.