PEACH Reality HoloLens app

Author: Timur Kuzhagaliyev

HoloLens application is responsible for downloading models and DICOM images ("objects" from here onwards) from the server, displaying them to our users and allowing them to add annotation to any object.

Usage Tutorials

Here are some short tutorial videos meant to teach users on how to use different aspects of our HoloLens application. They can also be considered a demonstration of our system.

Manipulation modes

Data markers

DICOM images

Feature Overview

Features currently implemented in the prototype:

Main menu. Main menu is a tag-along[1] that follows user around and tries to stay in their field of view. The menu can be toggled using voice command Show Menu and Hide Menu. In the menu, users can view what holographic patient cases are available for on HoloLens. After choosing a specific holographic patient case, users can spawn all models and DICOM images associated with it.
Object manager that supports loading of multiple objects into the scene. Loading is completely dynamic and is done during runtime, meaning new models can be downloaded to HoloLens while the app is working and our application will be able to load them into the scene and also style them.
All objects have context menus above them that allow users to interact with them in different ways using different manipulation modes. Every button in the context menu has a tooltip with the description of the action.
All objects have a semi-transparent bounding box attach them. These bounding boxes are a visual guide that show the user which model they are currently interacting with and if their cursor is in the right position to start the interaction.
Objects can be removed, rotated, translated or scaled by using appropriate manipulation modes. Some manipulation modes expose a progress bar that shows the user their current progress in a certain action.
Dynamically loaded models consisting of multiple components.
- Each component is textured and coloured based on the model part name during runtime.
- Models can be expanded to study each component of the model separately. In expanded view, every component of the model can be rotated and its transparency can be adjusted.
- In expanded view, user can attach data markers to different parts of the meshes of the model by using the data marker placement mode.
  - Data markers can hold text notes, inputted using either Cortana dictation or by typing on a virtual keyboard. Additionally, the user can attach and playback voice recordings on these data markers.
Dynamically loaded DICOM images.
- Users can scroll through DICOM images by entering the appropriate manipulation modes.

Future Development Plans

There are several major points that future developers might find worth addressing.

Achieving complete API compatibility. At the moment only the code for basic API authorisation and downloading has been implemented (as seen in the model-viewer-get branch in the peach-relity repo). Getting HoloLens to utilise all API endpoints might be a good starting point for the next iteration of the PEACH Reality project.
Extending the current functionality of data markers. Existing data marker scripts can be extended to support SNOMED CT codes and attachment of various multimedia files, like images or videos. That said, designing the UI to achieve the latter could be very tricky and might not prove to be feasible on HoloLens.

How to Extend the HoloLens Application

First of all, consult the Implementation Overview section below to understand the inner workings behind the code in the Unity project. Then, I'd suggest updating the HoloToolkit version used by our Unity project to the most recent version available to you.

If you want to add a new manipulation mode (that will be represented as a button in the context menu above the object), first create a new controller using existing controllers as an example (e.g. ManipulationController class). You might have to use the IInitialisable interface if your interface requires additional setup before it can be used in the scene. Once this is done, edit the ObjectManager class to bootstrap your newly created controller to a relevant object. At the moment you can attach it in one of the 3 places: to DICOM images, to models or to separate model components (which are seen when the model is expanded).

If you want to change the way models are loaded and processed before they are available to users to study and annotate, you will need to make changes to the ModelLoader class.

If you want to alter the functionality of data markers, you might have to edit one of these classes:

MarkerManager if you want to change the way data marker placement mode works, e.g. how the button is displayed in the context menu or what happens to markers when the model is combined back into its normal state. Additionally, this controls the mechanism behind placing a data marker.
MarkerController if you want to change the behaviour of a single marker, e.g. add a new way of data input or extend the functionality of text notes.

Other smaller components of the application are controlled by the scripts found in Assets/Scripts/Components in the Unity project.

Implementation Overview

This prototype was developed using Unity 5.5.2f1 with C# as the primary language of development. The only external library that was used is the Unity HoloToolkit[1], which provided a lot of useful classes for management of voice commands, user input, various events such as toggling the virtual keyboard or hovering an object with Gaze.

The main scene, ModelViewer3.unity, contains several objects that are crucial for our application. Apart from the trivial ones (directional light, main camera, event system) it contains the following root components:

Managers object. This object holds various of the so-called manager scripts that act like singletons, in a sense that there is only one instance of each manager available globally and all other objects in the scene (including those that were created dynamically) will reference this particular instance. Managers that are currently used:
- Gaze Manager: Uses raycasting to see which object is currently being hovered by Gaze. It must be noted that only objects with colliders attached to them can be considered.
- Hands Manager: Used to detect users' hands when they appear in front of the sensors. Useful to notify users when their gestures are actually being tracked and when their hands have drifted too far away from the sensors for gestures to be picked up.
- Gesture Manager: Used to detect gestures such as Gaze, Air tap, Manipulation or Navigation. This manager can be referenced by other scripts to fetch numerical values associated with the gesture that is currently being performed, e.g. vector representing the displacement of the hand from its initial position at the start of the Manipulation gesture.
- Keyword Manager: Used to globally detect when specific keywords come up in users' speech and then triggering certain events depending on what keyword was mentioned, resulting in voice command functionality.
- Voice Line Manager: Used to play audio clips with voice instructions on how to use different features of the application, such as selecting a manipulation mode or toggling the main menu.
Cursor object. Used to indicate when user's Gaze is hovering over an interactive object. To some extent it is similar to a conventional cursor controlled by a mouse, where an air tap or the HoloLens clicker would represent the left mouse button. Has several scripts attached that ensure that the object used to represent the cursor changes when hovering an actual object as opposed to empty space, and also when user's hand is detected by the sensors.
LoadingAnchor object. This object was initially added purely for cosmetic purposes as it displays the animation of the Peach Relity logo, but it can also be used to indicate that the application has successfully loaded and is ready for use.
MainMenu object. This one is self explanatory, it uses Unity UI to expose the screen that allows users to view holographic patient cases available on HoloLens. After choosing a specific holographic patient case, users can spawn all models and DICOM images associated with it. The main menu can be toggled using Show Menu and Hide Menu voice commands.
ObjectManager object. This object has a script with the same name attached it. This script is responsible for loading models and DICOM images into the scene, attaching all of the necessary helper scripts and controllers to them after loading. As an example, both models and DICOM images get bounding box controllers and manipulation controllers attached to them. The former is used to generate the bounding box that appears in grey on the video above and is used to indicate when the user is hovering an object, the latter adds basic functionality to objects, such as deletion, scaling, translation or rotation.
- All models have explosion controller attached to them. Explosion controller is responsible for expanding the model and making sure that when that happens, no two parts of the model overlap and hence make it hard to interact with each one separately.
  - In exploded view, users can interact with each model part separately. This model parts had data marker, rotation and transparency controllers attached to them during initialisation. Data marker controller is responsible for allowing the user to place markers on model meshes, and said markers can hold various data like text notes or voice recordings. Transparency controller allows user to change the transparency of each component.
- All DICOM images have a scrolling controller attached to them which allows the user to scroll through the slices of a CT scan using gestures.

The bullet points above summarise the abstract idea behind prototype's current architecture, but some points where left out to make the narrative easier to follow. For more information I suggest you consult the source code as described in the section below.

Building HoloLens Application

The source code for the HoloLens prototype is available at the main GitLab repository on the master branch. The Unity project can be found inside prototypes/ModelViewer3 directory. To build the project, follow the instructions below. The main scene is called ModelViewer3.unity and can be found inside the Unity project folder in Assets/Scenes.

One can preview the scene directly in the Unity editor, by pressing Play, then hovering the mouse cursor of over the Game window, pressing shift and moving the mouse round. Doing this will allow you to look around, while WASD controls can be used to move. C key can be used to open the menu and V to close it. To simulate an airtap gesture, press spacebar while still holding shift.

Open the project in Unity 5.5.2f1 or above
Select File -> Build Settings
Add ModelViewer3 scene to the list of scenes to build
Select Windows Store
Set the SDK to Universal 10
Set Target Device to HoloLens
Set UWP Build Type to XAML (needed to support the virtual keyboard)
Set "Build and Run on" to Local Machine
Click Player Settings and then check Virtual Reality Supported and select Windows Holographic
Click Build

This should build the project to the directory you've specified. Now, to run the project in HoloLens emulator or on the actual device, do the following:

Open the .sln file in the new build folder in Visual Studio
Change the solution configuration from Debug to Release
Change the solution platform from ARM to x86
Change the run target from Local Machine to HoloLens Emulator (or to Remote Device, if you want to deploy to HoloLens)
Select Debug -> Start Without Debugging

Testing

At the moment, most scripts that do not directly rely on HoloToolkit are covered by the integration testing (as seen in ModelLoadTest.unity scene found in Assets/Scenes folder of the Unity project). The script starts the process of loading a model which in turn bootstraps all other sub-systems, including but not limited to:

ModelLoader module
ResourceHelper module
Marker, Object, VoiceLine managers
Explosion, Marker, Opacity, MenuButton, Manipulation controllers

If any errors in logic have been introduced to any of the modules above, console would report errors during the loading process. At the moment, due to constantly changing of the scripts there are not unit tests for different components of the system. Additionally, we currently do not simulate user input (gestures, voice) for the tests.

Demonstration of an older prototype

If you're interested, below you can find a demonstration of an earlier prototype of the HoloLens application.