HoloLens testing and evaluation strategies
HoloLens application
The fact that our HoloLens application relies on a 3D engine and non-trivial user input such as gestures and voice commands makes it reasonably hard to test edge cases. Additionally, since Mixed Reality is a relatively new concept it's quite hard to predict user behaviour and extract explicit test cases from activities undertaken by our users.
Unity and integration testing
Unity Technologies have released a free Unity asset called Unity Test Tools[1], which contain a number of useful utilities. It provides an assertion engine and various helper classes for comparison of different states of objects introduced by various Unity libraries, such as colliders, vectors, quaternions and so on. This will make unit testing separate scripts trivial as Unity Test Tools provide a way to generate object and method stubs that can be used to confirm that a certain part of our code functions as intended[2]. Integration testing will be a bit harder, since said test tools are not capable of imitating realistic user input especially when it comes to HoloLens emulation. That said, we can still test complete parts of our application that do not rely on user gesture/voice input and those parts that depend on user input but in some indirect manner, so that we can simulate that input ourselves if necessary.
Manual testing
The applications of Unity Test Tools described above should cover around 90% of the code since we're planning to use Adapter interfaces for pretty all HoloLens specific features, which means that we'll be able to automate the testing of a reasonable part of our HoloLens applicaiton. As for manual testing, we can conduct User Acceptance Testing (UAT) by providing our testers with a list of requirements that our application was meant to conform to and evaluating their feedback. To make sure that our manual testing is exhaustive, we'll use the guidelines provided by Microsoft[3] to make sure we test our application under various conditions and in different environments.
Simulating human input
Finally, to automate the testing of HoloLens specific features, including gestures that are exclusive to HoloLens and voice commands, we'll use the Perception Simulator[4] module developed by Microsoft. It is capable of simulating various 3D spaces and human input and was specifically designed to make automated tasting easier. We can use scenarios we produced as a part of requirements gathering process to generate a set of test cases that will be fed to the perception simulator and later be used to confirm that our application functions as expected.
Evaluation strategies
To evaluate each prototype we create we decided to create a table that we could fill out during the evaluation. We wanted to consult both our requirements and our client during evaluation so a column for each of those is included in the table. The idea would be to consult the requirements that are applicable to the particular prototype and go through each of them to see if the prototype meants them. We would make comments on each requirement, whether it is met or not; but taking care to add as much detail as possible where requirements are not met. The next step would be to present the prototype to our client and ask for their feedback; this would then all be noted in the client feedback column. We would then take the table and read through the comments in each column and write improvements that need to be made when we create the next protoype, this helps to keep the iterative process of creating prototypes progressive rather than not knowing what needs to change in each iteration.
For the specific task of evaluating the user interface; it would also be possible to consult the scenarios we wrote. This would be especially useful if our client was not available to give his own feedback. We can look at the prototype and see if it would work as the persona would want it to work in the context of the scenario; if it doesn't then we can add comments in the evaluation table about changes that may need to be made.