Research

Related Projects

IllumiRoom by Microsoft Research, 2013

The initial IllumiRoom system was created in 2013, and a paper [1] was published explaining the proof-of-concept. The system enabled more immersive PC gaming experiences through the use of a TV, projector and Kinect camera. It showcased various types of projections: ones which made use of game source code to create game-specific projections and others which induced general peripheral motion. We will attempt to reimplement some of the features demonstrated, as well as some of our own extensions.

IllumiRoom's Projection Modes

From IllumiRoom’s user study on the 11 types of projected visualisations, it appears that the most popular were:

1. Focus+Context Selective: this is useful for games where items had to be found as these were highlighted in the peripheral vision. Focus+Context Segmented also gave game information without distracting the user.
2. Peripheral Flow illusions: these are used in the IllumiRoom system when you don’t have access to the game’s source code. Information about the motion of the game’s camera is used to induce motion through peripheral visual flow. Bounce (an element leaves the virtual game and interacts with the physical reality) was favoured by the users and game designers as it also provided game context.
3. Radial Wobble: this effect distorts the room geometry using a ripple effect) was considered to be impactful when used sparsely.

When designing our projection modes, we must consider the following:

1. Focus+Context (F+C): these illusions were more likely to overwhelm the user, inducing sickness. E.g. F+C Full provided too much information and F+C Edges emphasised high contrast edges.
2. Peripheral Flow illusions induced motion successfully but users missed the game context provided by F+C illusions. They also thought illusions like Appearance (which augments the surroundings to match the theme of the game, appear cartoon-like or look black and white) were not interesting by themselves, which prompts us to consider combining different illusions.
3. Illusions like Lighting (which matched the lighting conditions of the surroundings with the theme in the game) required a strict connection with the game so a mismatch could destroy the illusion for the user according to the game developer feedback.

Projection Mapping Central [2] (referenced in the paper) provides many examples of projection designs which can be utilised when generating ideas. Taking the above concerns into consideration, the goals for the illusions in IllumiRoom can be extended for our system so that each illusion supports a combination of the following: extend the user’s field of view (FOV); give additional game content; induce motion for the user; match the theme of the game; make the user feel more immersed/ present in the game; not overwhelm the user.

RoomAlive by Microsoft Research, 2014

The initial IllumiRoom system was created in 2013, and a paper [1] was published explaining the proof-of-concept. The system enabled more immersive PC gaming experiences through the use of a TV, projector and Kinect camera. It showcased various types of projections: ones which made use of game source code to create game-specific projections and others which induced general peripheral motion. We will attempt to reimplement some of the features demonstrated, as well as some of our own extensions.

Projection Mapping

RoomAlive Toolkit

The project requires us to implement 2D projection mapping. This technique will be used to turn the irregular surroundings behind the TV into a display surface for video projection.

The projection mapping used for IllumiRoom is explained in Microsoft’s RoomAlive Toolkit [4], a toolkit used to create dynamic projection mapping. The aim of the projection mapping process is to generate a render of the view to be projected onto the surroundings, from the user’s viewpoint. This requires a projection matrix to be computed for the user’s projector. The projector’s intrinsic parameters (internal parameters such as focal length, field of view, etc.) should be considered when assembling this matrix.

There are several types of projection matrices which can be formulated as detailed in Projections in Context [7], such as parallel, oblique, orthographic and perspective projections. The paper argues that the perspective projection is the most realistic as it closely resembles human vision.

Graphics APIs

Presently, there are various open-source real-time graphics APIs which can be used to assemble a perspective projection matrix. Leading APIs include Microsoft’s Direct3D, OpenGL and Vulkan.

1. DirectX / Direct3D — widely used for Windows and Xbox
2. OpenGL — available for most operating systems
3. Vulkan API — lower-level API which offers higher performance and more efficient CPU and GPU usage. But it presents a steeper learning curve, requiring more boilerplate and state management.

OpenGL is usually considered to have better performance than Direct3D though this differs depending on the game. Since DirectX typically works better on Windows platforms and it is our client’s operating system, we plan to use it for our graphics API.

Libraries

The RoomAlive toolkit uses Direct3D to set up a perspective projection matrix for the projector. Along with the projector intrinsics (mentioned previously), it is also necessary to take into account the that the principle point of the projector may not be the centre of projection (COP). Since the COP is typically placed at the origin [8], this would result in an off-centre/ off-axis perspective projection matrix.

For simple 2D mapping, RoomAlive has used Direct3D via SharpDX, a .NET wrapper of the DirectX API along with Math.NET Numerics packages. SharpDX is relatively stable but hasn’t been maintained since 2019. Furthermore SharpDX is one of the biggest blockers for migrating to newer .NET versions, as it is only compatible with .NET Framework 4.8.

Hence, it is important that we replace any code taken from the toolkit to not use SharpDX if we want to ensure the longevity of the project. Some alternatives to the SharpDX library are: Vortice, TerraFX (low-level), Silk.NET. We have decided to investigate Vortice further as it most closely resembles SharpDX.

Summary of Technical Decisions

We will implement our system – UCL Open-Illumiroom V2 in Python, rather than build on the original C# implementation for 2 reasons:

1. In our team, we are much more experienced with Python and have identified its various libraries we could make use of such as OpenCV2, Librosa and more.
2. We need to create a new implementation of the system as the original relied on the Kinect camera and ours needs to be able to calibrate using only a webcam or smartphone camera.

For creating the projection modes, the system will make use of the MSS library for capturing game data (by recording the game display). OpenCV will be used to analyse and transform the data for generating various visual effects. We will also make use of Librosa to analyse audio data. Since we aim to make our own variation of the Microsoft IllumiRoom’s Radial Wobble, we hope to use the audio detection to trigger the animation when certain sounds are detected.

Our projection effects will be displayed using a PySide2 window (the Python binding of the GUI toolkit Qt) and our system will be compiled for distribution, accessible on the Microsoft Store. Nuitka will most likely be used for compilation to C as it allows for a full build to be generated with all libraries used statically linked.

We will continue to investigate depth and projection mapping, as this will be the most complex part of the system that will be implemented. As discussed in Depth Mapping, structured light algorithms will also likely be an option which we can use. The projection mapping from RoomAlive will almost certainly be helpful.