Technological Foundations of Hands-Free Media

Embedding media into the real world allows for new immersive experiences that overlay information in a context- and location-aware way and provide a rich hands-free media experience. While the opportunities are vast, many open technological challenges remain. Only after overcoming these challenges it will become clear which forms of hands-free media consumption are possible.

The experience with augmented reality (AR) applications hinges on the availability of exciting content to consume. At the moment, relatively few compelling AR experiences exist because it is still very challenging to create gripping AR content, especially if the goal is a city-wide experience. Generating an AR experience typically requires good programming and 3D modeling skills, which severely limits who can create AR content.

Goals

The project addresses the specific technical challenges in the space of augmented reality ranging from where and how to visualize media content in this new space, to how to provide new tools for augmented reality content creators.

Outcomes

Content creation made easy
We developed a very streamlined way to generate AR content in the city of Zurich. Using a VR device, the content creator is virtually placed on a square in the city of Zurich. The user can then freely place 2D and 3D content and draw 3D shapes. All creations are automatically transferred to the correct locations within Zurich and can be experienced by other users using their smartphone or compatible Augmented Reality headsets. We believe that this allows artists and content creators to conceive impressive artistic installations within Zurich at ease. Also, it allows for fast prototyping of AR applications and ideas that require no programming knowledge or prior experience with AR technology.

Automatic placement of virtual billboards
We developed a tool that can automatically place virtual 2D billboards within the city of Zurich. The system automatically projects the 2D content at building walls to blend the real world environment with the virtual content seamlessly. Our system automatically sets the virtual billboards in places that pedestrians easily see and optimizes it for other factors, such as distance to the viewer. Our application renders content at three dierent levels of detail. From a distance, it provides just the most essential information and as the user gets closer to the content, it provides a detailed view. For example, the user would only see the headline from an article when far away from a billboard and moving closer to the billboard the user would see a teaser image and a teaser text.

Automatic placement in the city of Zurich
Our framework automatically places any content created with the methods described above within the city of Zurich. Currently, users can test the experience either using a smartphone or the Microsoft HoloLens. The exact user location is automatically computed based on so-called spatial anchors that allow us to identify the user location and put all content at the correct place.