AR Environment

The app is written in C# on the Unity Android and iOS platform. It uses the popular ARCore and ARKit libraries for each platform respectively to interact with the camera environment on display.

The scene makes use of AR planes, both vertical and horizontal to map out the environment in the AR Viewport. Each plane is then judged on its walkability and enabled for ray-casting

The Pet

The app displays an anthropomorphic pet that can walk around the planes mapped by the AR Plane Manager object.

The Pet game object is composed of an Animator (in charge of controlling skeletal poses), a mesh/avatar (the 3D skinned mesh on display) and a Behaviour Script that allows the pet to walk around the AR viewport when in input is presented.

In addition to its input driven movement, the pet also tracks the user's location by turning to always face the camera.

Voice Command Functionality

Much like on the desktop version, the pet actively listens for voice commands using the phones microphone. Once a sentence is recorded, it is then transcribed into text using the IBM Cloud SDK's Speech-To-Text functionality. The text is scanned for available commands, a reply is selected, and an appropriate audio clip is generated by the SDK's Text-To-Speech module.

The UI

The user-interface is implemented as a collection of canvases and a UI manager. Each of the UI canvases has a main panel and a back button. There are canvases for the pets belonging to the user, settings, user information and accessories if available for the current pet.

Web API (C#)

The API uses a .NET HttpClient to send and receive requests from the web server.

Each request and response are formatted from JSON strings to C# structs and vice versa using Newtonsoft.Json. Requests are sent asynchronously, and responses are received asynchronously.

Each response is serialized into an APIResponse<T> generic structure that contains a T type payload and error code if the request was unsuccessful.

Functionality

The web api has commands for all the functionality available from the Web Server.

It has both regular and Async versions of each request.

The api stores session information into a Session object which is then used to request data from the server..

Documentation

Documentation of the API is available on its GitHub repository README.

Testing tools (Developer CLI)

Together with the API is also packaged a command line tool that support executing commands on the web server directly

It was mainly used for managing pet data during development sessions

Login

The login screen sends and receives data from the web server when the user enters their login credentials. The login fails if the post request does not contain the user login information; otherwise, the user account information is verified and the SceneManager.LoadScene() method is used to load the application/GUI.

Speech-to-text input

The speech-to-text input works by the user’s microphone detecting the user’s speech input. This input is then sent by the STT (Speech to Text) manager to the Watson backend and retrieves the text string in response. The Assistant script then checks if the command is spoken and sends the text string to the chatbot. After receiving a response from the chatbot, the speech bubble shows up next to the pet. Meanwhile, the TTS (Text to Speech) sends the response text to the Watson backend, gets the audio clip in response to the user’s input, then plays the audio.

Chatbot

The chatbot is implemented using an open-source machine learning framework for automated text and voice-based communications software called Rasa. Rasa is a highly customisable, Python-based software that enables our project to interpret the input provided by the user in the form of speech-to-text and respond with the appropriate reply.

Voice Control

We implemented 2 types of voice commands in Unity.

The first are built-in Unity functions, like “play music” and “start timer”.

The second are external commands like “Open BBC”, “Search how to run”, etc. These commands can be customised by editing the configuration file under resources/configs folder. The app reads these commands from the config file and stores the commands in a dictionary.

Both methods use keyword detection. If the keyword is detected (after it is processed by IBM’s Watson Cloud SDK Speech-to-text function), the corresponding command is executed.

Desktop Gadgets

The desktop widgets include a music player with an autoscrolling bar, that is updated every frame by changing the x-y positions of the related GUI elements during each frame. The timer functions by calculating the time between the clicks on the start and stop buttons, while the real-time clock displays the time using Unity's Quaternion.Euler() methods to rotate the hour and minute hands.

Desktop Pet

Our first implementation of the desktop pet was to have it look at the cursor. However, we scrapped this idea as the pet looking at the cursor was too weird and unnatural. We also tried changing the background as time-of-day changes – like stars during night, but we decided against it as its functionality was buggy and could cause some problems for people in different time zones. In our current implementation, the pet AI is essentially a state machine that reacts to different clicks. The OnDrag() method allows for the user to drag the pet across the screen, having the pet follow the mouse’s x-y coordinates on the screen plane to update the location of the virtual pet. The SwitchAnimation() method under the ReactToInteraction.cs script detects the input of the users mouse (left click, right click, scroll click or drag) to check whether the mouse is on the pet when it is clicked; if it is, then it redirectst the code to the appropriate pet interaction method and causes the pet to react accordingly.

Some notes on Design Patterns

We used the Singleton pattern for the WatsonManager and GameManager because we want them to be globally accessible and to be instantiated only once – they will not be destroyed upon entering the AR mode.

We used the Observer pattern for the SpeechBubble and TextToSpeech. This way, the Assistant sends a notification after receiving a response from the chatbot. After observing it, the SpeechBubble shows up and starts a timer.

When the timer reaches 0, the SpeechBubble disappears. TextToSpeech sends the text string to the backend, retrieves an audio clip, and plays it.