When doing initial research for this project, we were not able to find very many similar projects. More and more chatbot avatars are being developed, but these are rarely combined with AR. We have not yet been able to find an AR avatar receptionist project, although it’s very possible that this does already exist, for example in another language that our searches may not have turned up. In addition, those projects that we were able to find are all proprietary, and so it was difficult to find out what technology had been used behind the scenes in order to build these. This is what leads us to believe that we are one of the first teams to be developing an open source AR avatar receptionist.
The two most applicable similar projects that we could find were:
This is a virtual AR avatar language teacher that can guide you
through basic language learning scenarios and conversations
in your home. This is a virtual animated avatar that appears in
world space, in the same way that we hope our avatar to. It
also uses voice interaction to communicate with the user.
London based AR/VR agency, Arcade, developed this AR avatar
chatbot for use in SeaLife’s London aquarium. The app is aimed
at kids, who can interact with it inside the aquarium and take part
in challenges and games set for them by the avatar. Evidently the
setting of this project differs from our own, but the chatbot and
AR capabilities are similar to what we hope to achieve. Another difference from our project is that this avatar interacts with users through text and not through voice interaction, however there are similarities in the types of questions that users may ask “tell me
more about...” and in rendering a character in world space.
Additionally, some of the non-AR receptionist chatbots that are available are:
Deployed by Robots of London, Pepper is a physical robot with
a built-in touch screen, who interacts with visitors using voice recognition and is able to stand in for a human receptionist.
Pepper has already been deployed at offices in London. This
obviously differs significantly from our project given that Pepper
is a physical robot, but the chatbot and voice recognition
capabilities, as well as the kind of scenarios that Pepper is
equipped to deal with, are similar to the aims of our own project.
https://www.robotsoflondon.co.uk/pepper-the-receptionist [3]
Developed by WinTech in the US, ALICE is a virtual receptionist
touch screen program that can welcome visitors to a building.
This is similar in functionality to what we would like to achieve,
but very different in terms of implementation. For one, ALICE is
deployed on a standalone touchscreen device, wheras we would
like to build our project to function on any existing android device. Additionally, Alice uses video recordings of real human beings for
its avatar rather than computer generated animation.
The main backend of our project, including voice recognition, conversion and chatbot technology, was already provided for us by IBM. This meant that we were slightly constrained in making technology choices because we needed to choose frameworks and software that fit well with IBM technology.
Since we were using AR, we had to develop for mobile devices that have this capability, which means Android and Apple mobile devices were our main choices, since both have relatively advanced AR development packages (ARCore and ARKit respectively). Ultimately, we wanted our project to be as widely available as possible, so we wanted to develop for both Android and Apple. We decided to start with Android because firstly, we had access to Android devices from our department, and secondly, we assessed the development pipeline for working with Android would be slightly easier – Apple have stricter controls on certain AR capability such as facial recognition (which we initially thought we would like to use for user sentiment analysis), microphone and camera permissions, and an Apple Developer Account is required to deploy your app to a phone. For this reason, we chose to start with Android with a view to deploying on iOS at a later stage.
Our main choices were between Unreal Game Engine and Unity Game Engine [5]. The decision was made much easier for us because IBM Watson, which we were required to work with, already has an SDK [6],[7] for Unity which made our work much easier. We could have chosen to write our own interface to the Watson cloud, but we felt it would be unrealistic for our skill level and the amount of time we had. For this reason, we chose to work with Unity and the Watson Unity SDKs.
With our ultimate goal of cross-platform development, we wanted an AR package that worked on both Android and Apple, and so using either ARCore or ARKit alone did not suit our needs. That left us with two main choices for an AR package that worked with Unity: Vuforia or Unity’s own AR Foundation [8]. Ultimately, we assessed that Vuforia probably had slightly better AR capability, but AR Foundation was developing rapidly and was also completely free, whereas Vuforia requires a subscription. With our goal of creating a low cost, open source project, we decided to take a chance with AR Foundation because we feel it’s more future proof and likely to significantly improve in capability over time. In fact, AR Foundation released a new version at the beginning of 2020 which we were able to use in our project.
IBM Watson Assistant can call out to a HTTP webhook to request information. We therefore knew we would have to use a webhook and connect to a URL. With an Azure Subscription provided to us for the project it made sense to use an Azure Function as our webhook. This also that we could use C# to write the webhook, which is the same language as we are using for scripting in Unity, making our workload slightly easier in terms of learning new technologies. We needed to connect our webhook to a simple database. We decided to use a Relational Database (as opposed to a Non-Relational Database) because we knew the kind of data we would want to store and query (staff names and contact information, book names and authors etc.) would be highly structured, and in addition we might want to put together some more complicated queries using joins,( for example if a patient provided their name to check into an appointment, we might want to look in another table to pull information about which doctor they were attached to). We chose to use the SQL database solution provided by Azure, which conveniently allowed us to include both database and webhook in the same resource group, which in a real deployment scenario would allow us to price these and manage their usage together. [9],[10]
