Tend-VR

Research Reports

For our project, our clients asked to complete two research tasks on features which they could add to their app. They wanted us to look into how other apps use gamification techniques to increase user engagement with the app and encourage users to continue to regularly use the app.

They also asked us to research methods which they could potentially use to assess the mood of users when they use the app, as well as the different options to take these assessments such as using the VR controllers or using voice input recognition services like Google Cloud Speech-to-Text.

Speech-to-Text Mood Questionnaire

Upon receiving the task to implement a mood questionnaire that could be completed with speech-to-text, we initially looked into Unity’s KeywordRecognizer library. Within a few hours, we were able to get the application working successfully by setting a string array of commands to be recognized and using the KeywordRecognizer to identify the user’s words in real time.

However, when researching further into this implementation, we found that this would only work with a VR headset that is connected to a PC running Windows as the KeywordRecognizer uses UnityEngine.Windows, which makes it completely dependent on whether the VR application is running on Windows or not. As the speech-to-text functionality needed to work with Android, we had to find an alternative way to complete this aspect of the workload, so we began researching a variety of ways in which we could get the VR application in Unity to communicate with Google Cloud.

After looking into the speech-to-text documentation on the Google Cloud website to figure out how to integrate the speech recognition technology with the Unity application, we were able to determine the method for creating an audio transcription request to be sent over using the command line. Any audio files sent to the API would be returned with a best-match transcript, a corresponding confidence level, as well as an array of alternative transcriptions.

When trying to get this working with the Unity VR application, we ran into issues as there was no documentation on Google Cloud support with Unity or C#. The project partners specified that it would’ve been very hard to get this working and that they were happy for us to use the Unity Asset store as Tend’s USP was nothing to do with voice recognition. With this in mind, we looked into helper tools that we could use and came across a tool that provided the functionality for speech recognition using the Google speech-to-text API, developed by Frostweep Games.

By referring to the provided documentation, we managed to implement a fully functioning implementation of the mood questionnaire that allowed direct interaction with the user interface as well as completing it verbally.

Water Surface Ripple Shader

Initially we looked into using Unity's Shader Graph tool to develop the shader, but the app was switched to using Unity's Built-in Render Pipeline, which did not have Shader Graphs. Instead we decided to write the ripple shader using a standard surface shader in Unity's Shaderlab language, as this was more convenient than using vertex and fragment shaders, as the surface shader would generate the code for the vertex and fragment shader.

Flocking Behaviour

When we were asked to give a virtual ‘animal’ the ability to react and engage with its neighbours in a way that mimicked the real behaviour of flocking, we initially did a great deal of research into what made flocking behaviour possible. We watched a lot of examples of the behaviour both in real scenarios and in already implemented digital ones. We gained perspective from Craig Reynold’s Boids in 1986 to how we could best implement it.

We found that basic models of flocking behaviour are controlled by three simple rules (Separation, Cohesion and Alignment) and we had to write a script that took into account these simple rules. The resulting flock would move in an extremely realistic way that couldn’t be done if they were not implemented.