Project Background

IllumiRoom is a proof-of-concept system designed by Microsoft Research in 2013. Using a Kinect for Windows camera and video projector, it augments the surroundings around a television with projected visualisations to create more immersive entertainment experiences.

We are leading the development of a “remastered” IllumiRoom, building on Microsoft’s original work. This project is completed in collaboration with our clients Prof Dean Mohamedally (UCL) and Prof Lee Stott (Microsoft).

Project Objectives

From our initial assessment of the project brief and introductory talks with the clients, we arrived at a set of preliminary project objectives.

  1. Replace the Kinect with a standard webcam or smartphone camera. This calls for a re-design of the depth and projection mapping processes.
  2. Design immersive effects to project around a TV when watching films or playing video games.
  3. Identify commercial and open-source games which can be used with the system.
  4. Create a system that is modular and expandable, with strong use of abstraction. It should be possible for more effects to be added in the future.

Gathering Requirements

We captured our requirements in 3 ways. Discussions with Prof Dean Mohamedally yielded the primary requirements for the system. These included:

  • The key purpose of the system which was to provide expanded experiences for entertainment.
  • The place in which the system would be used, a setting with enough space for a TV and a large projection area. This would be the user's living room.

We also observed the user study conducted by the Microsoft Research team in their paper, IllumiRoom: Peripheral Projected Illusions for Interactive Experiences (2013). These reviews were collected by recruiting 10 participants (aged 20-30, 2 female and 8 male) who were familiar with first-person shooter games. The results provided us with an overview of the most and least popular projected visualisations and the heuristics we should use to design immersive projections. We concluded that:

  • A variation of the Radial Wobble effect would be great to implement as it was described as ‘magical’ and did well to create a feeling of immersion in the game world.
  • We should be careful not to overwhelm the users with effects as some participants in the original study complained of motion sickness.

We extended our understanding of potential users by carrying out comprehensive, semi-structured interviews. We concluded that interviews in this style would allow us to probe the users based on their responses whilst maintaining a baseline set of questions to analyse. Our participants were asked various questions to gauge their interest and understanding the project. Each interviewee was shown an information sheet and signed a consent form before the interview. They were then shown a video demo of Microsoft’s IllumiRoom system to identify potential areas of improvement. Some questions we asked each user type are shown below.

Gamer with mobility impairments, Age 28

Right now, TV-based gaming experiences are limited to the confines of the TV. To be able to go beyond that would be incredibly immersive if the animations could be accurately mapped to the surroundings. VR and AR technologies are very popular right now, and to not require an expensive headset and tracking would be very beneficial.​

Considering my mobility, not very often, at most once every few months. The background should not change too often either way. Any data collected must be stored and used securely as sensitive information such as pictures may be recorded.​

Experienced gamer, Age 20

It would be very helpful in adventurous games such as COD, GTA and Overwatch. These are games in which I would hope to experience a more realistic and immersive experience.

Gestures and in-air joypad would be the most useful as it would allow the system to track the gesture and motion of the user. Being able to control the motions with an in-air joypad can be more comfortable as it is hand-free and will allow the system to be more accessible. However, for most users, a standard game controller will be most convenient​.

Game developer, Age 30

A lower camera quality would result in depth maps that aren’t as informative or accurate about the landscape. Hence, the projection mapping may have some inconsistencies that ruin the illusion for the user. So, projections with a looser connection to the game content could be utilised (e.g. filter effects to change the ambiance of the room).

The touchless component improves accessibility and is appealing as it means that users can interact with the system without the need for a physical control system (remote or the existing console) or by switching away from their game to an app to change the mode they're on.

Personas

We reflected on the interview responses by creating personas and scenarios, categorising the key behaviours of our users and focusing on improving accessibility.

Prototyping

To ensure a seamless user experience, we paid great attention to the system's human-computer interaction (HCI) components. As such, aside from conducting user interviews and developing personas, we also created software and hardware prototypes. These were heuristically evaluated and improved upon, leading to an iterative design process.

Harware Sketches

We created initial sketches of the hardware setup. This included considering the projector placement in the user's living room as well as how they would calibrate and use the system.

Hardware Sketch
Hardware Sketch
Hardware Sketch

UI Sketches

We also made sketches of the UI, using Balsamiq to come up with high fidelity wireframes. One such wireframe from our iterative process is shown below. The UI displayed general projection modes to meet the needs of the Amika Shah persona. The game-specific modes ensured compatibility with a range of games as desired by Harry. We also considered implementing a gesture control setting to enable touchless interactions with the system for Sally.

UI Sketch

Heuristic Evaluation of the Hardware

We determined important heuristics that the system had to follow based on the requirements we had gathered.

Hardware Evaluation

Heuristic Evaluation of the UI

UI Evaluation

Improved UI Prototypes

Following on from the our evaluation, we improved our UI prototype.

Improved UI

MoSCoW Requirements

After discussing the project further with Prof Dean Mohamedally, we came to decision of removing gesture control from the system to reduce its complexity. We also decided to move forward with an MFC-based UI as this is required to deploy to the Microsoft Store. The table below shows our final functional and non-functional requirements (with non-functional in blue).

ID Description Priority
1 Design a calibration system to correct the projection from a fixed user perspective MUST
2 Obtain an image of the projected area to use as a base for some projection modes MUST
3 The calibration process must take less than 3 minutes in total MUST
4 Create a GUI to allow the user to setup and run the system (including settings selection for the projection modes) MUST
5 Allow the user to categorise their display screens to retrieve each display’s details (top left coordinates, width, height) MUST
6 Provide a comprehensive user guide to allow for easy setup MUST
7 System compatibility with most, if not all, commercial games MUST
8 Create 5+ projection modes, including game and non-game dependent modes MUST
9 Capture the TV display to create projection modes which react to the TV content in real-time MUST
10 Capture the TV audio to launch a projection mode when certain sounds are detected e.g., via machine learning (ML) technology MUST
11 Analyse the TV display to perform real-time weather detection on games and films using ML technology MUST
12 Include modes which use the features of the room to create animations MUST
13 Compile the Python project into an executable using Nuitka MUST
14 Deploy to the Microsoft Store MUST
15 Release the project as an open-source framework on GitHub with detailed documentation SHOULD
16 Create an extendable system using OOP principles to ensure that developers can create their own projection mode designs SHOULD
17 Compile projector-camera calibration into an EXE usable from the main project SHOULD
18 Optimise capturing camera images for the calibration using multithreading technology SHOULD
19 Allow switching between the projection modes without restarting the application SHOULD
20 System compatibility with films and other TV-based entertainment COULD
21 Implement calibration from a fixed user perspective to correct the image or video for a wider variety of backgrounds, such as wall corners [to assist Team 34 in developing their Stellar Cartography system] COULD
22 Move calibration implementation to C++ for ease of compilation and improved performance COULD
23 Given a sound effect, identify when the sound is heard whilst playing a game or watching a film COULD
24 Incorporate hands or facial gesture control to navigate the set of projection modes whilst playing a game WON'T