Distributed Reinforcement Learning for Overlay Networks

• Main Author: Mastour Eshgh, Somayeh Sadat
• Format: Others
• Language: English
• Published: KTH, Skolan för informations- och kommunikationsteknik (ICT) 2011
• Subjects: TECHNOLOGY; TEKNIKVETENSKAP
• Online Access: http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-92131

In this thesis, we study Collaborative Reinforcement Learning (CRL) in the context of Information Retrieval in unstructured distributed systems. Collaborative reinforcement learning is an extension to reinforcement learning to support multiple agents that both share value functions and cooperate to solve tasks. Specifically, we propose and develop an algorithm for searching in peer to peer systems by using collaborative reinforcement learning. We present a search technique that achieve higher performance than currently available techniques, but is straightforward and practical enough to be easily incorporated into existing systems. Theapproach is profitable because reinforcement learning methods search for good behaviors gradually during the lifetime of the learning peer. However, we must overcome the challenges due to the fundamental partial observability inherent in distributed systems which have highly dynamic nature and changes in their configuration are common practice. Also, we undertake a performance study of the effects that some environment parameters, such as the number of peers, network traffic bandwidth, and partial behavioral knowledge from previous experience, have on the speed and reliability of learning. In the process, we show how CRL can be used to establish and maintain autonomic properties of decentralized distributed systems. This thesis is an empirical study of collaborative reinforcement learning. However, our results contribute to the broader understanding of learning strategies and design of different search policies in distributed systems. Our experimental results confirm the performance improvement of CRL in heterogeneous overlay networks over standard techniques such as random walking.