Team 12 AR Receptionist Project

We are team 12, our project is to create an augmented reality avatar receptionist. This would appear on a mobile device in the lobby of an office building, library or doctor's surgery, for example, and would be able to offer basic help to visitors and answer questions about the organisation. We are working for IBM and using IBM Watson to underly the main logic of our avatar.

Problem

In busy office, healthcare or educational settings, much time is spent by administration staff in manning the reception, waiting for visitors to arrive and being the ‘face’ of their organisation. Reception staff also usually double as administrative staff, and constantly breaking up their day by greeting visitors or accepting deliveries, means their time is often mismanaged, and this is particularly a problem in busy and understaffed workplaces, such as healthcare settings, schools or startups.

Our Solution

The technology now exists for some of this reception functionality to be delivered by a virtual assistant, freeing up administration staff to focus on other important tasks, to spend more time with the most important visitors, and preventing queues from building up in reception areas. Our solution is to use a simulated humanoid character in an augmented reality setting, which provides a more “human”-like welcome than a simple touch screen, and allows a company more control over the first impression that a visitor has when coming to a building. Our project set out to achieve this using IBM Watson technology.

Achievement and Impact

  • There are several pieces of technology that come into the creation of a virtual AR avatar: Voice recognition is required to understand what the user is saying; the output of this then needs to be sent to a chatbot backend, underpinned with machine learning algorithms, to decide on an appropriate response; and this response will most likely come back in text form, and therefore needs to be converted back into speech. Finally, the avatar itself needs to be designed and animated, and placed in world space on a mobile device using an Augmented Reality capable graphics engine.
  • In 6 months we have made significant progress towards a rugged, deployable avatar receptionist: Using IBM tools on the backend we were able to join up multiple pieces of technology: IBM Watson Speech-To-Text, Assistant and Text-To-Speech, Unity Game engine and the AR Foundation package, as well as link this to a webhook and SQL database hosted in the Azure cloud. This pipeline allows a user to ask a wide range of questions in three different example settings: the IBM headquarters reception; A library; and a GP surgery. The avatar can ping the backend database to get particular information, for example about available books, prescriptions, appointments and information about the organization and it can also send a text message to a staff member to inform them that someone is here to see them.

      • Key Features

    • Our avatar uses voice recognition to respond to a visitor’s spoken request.
    • Our avatar gives spoken answers to basic requests and questions that might be asked in a reception setting, such as: 
    • “What events are happening here today?”
    • “I’m here for an appointment with Brenda.”
    • “Can you tell me more about your organisation?”
    • “Where’s the toilet?”
    • And many others!
    • Our avatar has the ability to summon a human receptionist by text message, and to send text messages to particular staff members to inform them that someone is there to see them.
    • Our avatar comes equipped with basic animation, user prompts and menu buttons allowing for ease of use even for users who haven’t encountered augmented reality before.
    • We’ve demonstrated the ability for our avatar to be adapted for use in different settings, by providing three different demonstration avatars: one for an office reception at IBM HQ; one for a library; and one for a GP surgery.
    • We’ve provided documentation and an editable database backend so that anyone can insert their own information and adapt our avatar for use in their own organisation.

    Here you can see a video showing our demonstration avatar in action:

    Team

    Lilly Sinek - Team Leader

    AR design, Android AR programming, Unity programming, Chatbot design, UI design, Website content, Video editing

    Dillon Lim

    Website design and programming, UI design, reporting

    dillon.lim.18@ucl.ac.uk

    Oliver Vickers

    Backend programmer, Database design and programming, Azure webhook programming

    oliver.vickers.18@ucl.ac.uk