Evaluation

Achievements


Known Bugs



Contributions

Critical evaluation of the project


User Experience

Pros: From our user testing we can see that most users understand the concept of the app and that the UI is simple and useable. The UI looks aesthetic and the experience of an AR avatar is still novel and impressive for many users. Our avatar responds well to questions that it knows how to answer, and, in these cases, the spoken request and response is smooth, leaving users feeling like they are having a real conversation.

Cons: Our avatar is not very aesthetic – to put it in the words of our TA, it “doesn’t spark joy”! We have not yet managed to find an appropriate lightweight avatar that has the performance of the current basic one (see Design Page, Experiments Section). This definitely has a negative impact on the user experience. Furthermore, users could probably do with even more on screen instructions and error messages to help them navigate the app, for example, we are still encountering occasional issues with the Watson connection freezing and in this case the user is left hanging without knowing what to do. The avatar conversation is still quite inflexible, because writing a realistic receptionist chatbot that can handle all possible reception conversations is a huge task.


Functionality

Pros: We have achieved about 95% of our must and should have functionality. About 75% of this works very reliably, and the remaining 20% works most of the time. The project is very functional if users are given a conversation map that shows them what they can and cannot say to the avatar.

Cons: We have the following notes on certain functionalities in our MoSCoW list (see our achievements table for details of the requirements we are referring to)

No.8 This requirement was deliberately vague because we were not sure when we started the project how much we would be able to adapt our chatbot to different organisations. We have certainly met the first part “ability to adapt the avatar chatbot engine to different organisations” because we have provided three different types of chatbot (IBM office, GP and library), and this has been a significant part of our work this term. This demonstrates to our client and users that our avatar is adaptable to different settings. There is “API-Like functionality” to edit “the chatbots recognised answers” through the Watson Assistant GUI and the Azure SQL database editor, but we did not build any new API. We have marked this requirement as complete because we believe we achieved the overall objective of demonstrating that the chatbot is adaptable to different organisations, and we put in significant work to do this.

No.12 “Basic Directions”. Our avatar offers some direction in the form of a) telling a user vocally which office someone has and b) showing them a map. It does not give directions such as “go straight ahead and turn left.” We realised this was much more complicated than expected to implement and we did not have time to do so. Therefore, we have marked this requirement as 50% complete.

No. 14 In our final avatar, very little text is displayed to the user in world space because people found this confusing and difficult to see. The text that is displayed adjusts rotation and position but not size. This is because we realised it was unnecessary to change the text size, because a user would just move closer to see the text bigger.

No. 14 In our final avatar, very little text is displayed to the user in world space because people found this confusing and difficult to see. The text that is displayed adjusts rotation and position but not size. This is because we realised it was unnecessary to change the text size, because a user would just move closer to see the text bigger.


Stability

Pros: Our app and its connection to Watson Speech to Text, Text to Speech and Assistant is relatively stable in the majority of cases. Even if there are issues with the Watson backend, the AR frontend very rarely crashes.

Cons: We have some stability issues when errors or blank responses are returned from Watson Assistant, which causes the Watson logic of our avatar to freeze up (see Known Bugs list). We also have some issues with image tracking when the trigger image goes out of frame (see known bugs list).


Efficiency

Pros: Our app is quite lightweight – we haven’t added any unnecessary functionality, and our avatar is relatively basic, so it renders efficiently on the mobile device even whilst the camera is moving and tracking the trigger image at the same time.

Cons: We went over our Azure budget for our SQL database and webhook in the last months which suggests that there may be some inefficiencies in our backend.


Compatibility

Cons: We haven’t managed to do a lot of compatibility testing with different Android devices, partly due to not having access to these because of Covid-19, and we also haven’t achieved our goal of building for iOS.


Maintainability

Our project is based on numerous different frameworks, which means it relies on many different packages to be functional. This could present some challenges in maintaining it over the long term, if some of these packages or frameworks are updated.


Project Management

Pros: We worked consistently throughout the year on this project and made incremental progress each week. We managed to find a good workload split between the front and backend of the project, so that one team member could work on each at one time without interfering in each other’s work.

Cons: We were too ambitious and didn’t give ourselves enough time at the end to finish all our documentation and testing. We should have tried to document and test as we went along in a more thorough way.


Future Work

This was a highly ambitious project - as far as we are aware, there is no similar project that has been brought to market, and existing AR avatars are in the early stages, with large companies and development teams supporting the launch of just one avatar. Therefore, for a student team working part time, although our product is not nearly as shiny or error-proof as a professional project, at the same time we have achieved something significant in joining up existing technologies. Our solution is extremely low cost - for the price of an azure database, anyone could make their own AR avatar.

There is significantly more work to do and with another 6 months we would be able to take more steps towards a product that would be useable by an average user without a supervising human. The steps would be:

  • The tracking of our avatar in the world is still buggy - the avatar occasionally displays floating around or disappears. We believe this is due to the implementation of tracked images in Unity’s AR Foundation and Is likely to improve with further releases of this package. We could also look into writing lower level implementations of this functionality.

  • We would look into implementing sentiment analysis of the user to improve user experience and allow the avatar to react to user behavior.

  • The pipeline between text to speech, Watson assistant and speech to text is relatively quick given that it involves six separate http messages. However even this is more laggy than ideal in a real-life setting. We could work with IBM to find a way to pass information between Watson and text to speech directly, rather than having to return information to the device. This could reduce this latency and make the product more useable.

  • The voice recognition provided by IBM Watson has some flaws. It is not yet able to recognize heavy accents or operate in a noisy environment. This would be expected to improve as time goes on and subsequent upgrades of the framework are released.

  • We would work on providing a GUI for non-technical users to create and launch their chatbot.

  • We would introduce compatibility with iOS.

  • We would make updates to the aesthetics of our product. Our focus was on implementing the core features. With more time, our UI and user messages could be further improved with animations and adding more user prompts for noisy environments or strange surfaces etc.

  • Additionally, we could make significant improvements to the appearance of our avatar. We could find someone who would build a custom avatar for us. We tried to implement one that was more good looking, but it was too processor heavy to render. We could also spend a lot more time refining our animations.

  • Security: Our pipeline is currently insecure in numerous places, primarily because we didn’t have time to develop adequate security and because we are using fake demonstration data. However, for our app to be deployed in a real setting we would want to set up authentication to our webhook and work on re-enabling SSL connections to Watson. This is currently not working when building to Android which seems to be an error with the Watson SDK which we weren’t able to get around.

  • Bug fixes and more comprehensive testing. Unfortunately, our testing strategy was one of the weakest elements of this project, because we didn’t think about it early enough. We should have conducted user testing throughout development, which would have made it easy to uncover critical bugs and generally would have incentivized a quicker development pipeline. In future we would make testing an integral part of our workflow.