| Summary: | Augmented Reality (AR) and Virtual Reality (VR) provide powerful, natural, and robust ways to interact with digital content, across a number of different domains. AR and VR, collectively known as Extended Reality (XR), can facilitate the execution of surgical procedures, aid in maintenance and repair of mechanical equipment, provide novel visualization paradigms for data analysis, and even empower new ways to experience video games. These experiences are built on rich, complex real-time interactive systems (RISs) that require the integration of numerous components supporting everything from rendering of virtual content to tracking of objects and people in the real world. There are decades of research on the development of robust RISs, utilizing different software engineering modalities, which facilitate the creation of these systems. While in the past, developers would frequently write all of the components and the “logical glue” themselves (often built with graphics suites such as OpenGL and DirectX), with the the rise of popular 3D game creation engines, such as Unity and Unreal, new development modalities have begun to emerge.
While the underlying game engines provide a significantly easier pipeline to integrate different subsystems of AR/VR applications, there are a number of development questions that arise when considering how interaction, visualization, rendering, and application logic should interact, as developers are often left to create the “logical glue” on their own, leading to software components with low reusability. As the needs of users of these systems increase and become more complex, and as the software and hardware technology improves and becomes more sophisticated, the underlying subsystems must also evolve to help meet these needs. In this work, I present a new software design pattern, the Relay & Responder (R&R) pattern, that attempts to address the concerns found with many traditional object-oriented approaches in XR systems. The R&R pattern simplifies the design of these systems by separating logical components from the communication infrastructure that connects them, while minimizing coupling and facilitating the creation of logical hierarchies that can improve XR application design and module reuse.
Additionally, I explore how this pattern can, across a number of different research development efforts, aid in the creation of powerful and rich XR RISs. I first present related work in XR system design and introduce the R&R pattern. Then I discuss how XR development can be eased by utilizing modular building blocks and present the Mercury Messaging framework, which implements the R&R pattern. Next I delve into three new XR systems that explore complex XR RIS designs (including user study management modules) using the pattern and framework. I then address the creation of multi-user, networked XR RISs using R&R and Mercury. Finally I end with a discussion on additional considerations, advantages, and limitations of the pattern and framework, in addition to prospective future work that will help improve both.
|
|---|
