Back to Top

Evaluation

Home → Evaluation

MoSCoW List Achievement

ID Description Priority State
1 Chatbot provides resources about the PRIDAR requirements MUST
2 User can have their own questions answered about the PRIDAR standard MUST
3 Adherence to the EU GDPR MUST
4 Explanations for each PRIDAR topic SHOULD
5 A general test on each PRIDAR topic to understand NHS deployment preparedness SHOULD
6 Scoring system for the user to keep track of progress SHOULD
7 A visualization of user scores SHOULD
8 Voice based interaction COULD
9 Deployment to Amazon Alexa or Google Assistant COULD
10 Text based deployment to Microsoft Teams or Telegram COULD
11 Capability to scale database and userbase COULD
12 Efficient algorithms for scaling COULD
13 Allow users to choose topics to view resources or get tested on COULD

Individual Distribution

Work Packages Adel Linta Fraser
Client Liason 40% 30% 30%
Requirement Analysis 33% 33% 33%
Research and Experiments 50% 20% 30%
Implementation 40% 30% 30%
Testing 60% 0% 40%
Report Website 20% 70% 10%
Blog 55% 45% 0%
Videos 40% 20% 40%
Presentations 50% 20% 30%
Overall contribution 40% 30% 30%

Critical Evaluation


We decided to draw up personas based on information we got from people we interviewed. This would help us to visualize and keep in mind our potential types of users as we started developing our project.

User Experience

With learnings from our HCI component, we tried our best to deliver a usable chatbot experience that gave the user options, set up their expectations, and directed them well. We achieved this by giving the user choices where possible, such as when to display resources after completing a test, or without taking the test at all. The PRIDAR chatbot also re-prompts users in the case that their input is unexpected, and never leads them to a dead-end response, as can be seen in the login/signup segment of the bot. Additionally, the feature of only asking users topics they have answers for gives them power in choosing how much time to spend on the test and allowing them to skip topics that aren’t helpful to them.

With that being said, our project still has a ways to go in terms of HCI, as we often implemented these “quality of life” additions after a features base functionality was complete. For instance, the PRIDAR chatbot still only accepts a limited input from the user, not accepting synonyms of words such as “yes”. Also, feedback on the test feature indicates that it can feel repetitive typing the exact, case sensitive answer to potentially over 100 questions.

Functionality

We have worked hard to implement all the key and additional requirements:

  • Test user knowledge on the PRIDAR standard
  • Illustrate how prepared user’s projects are for NHS deployment
  • Help users understand the PRIDAR requirements
  • EU GDPR compliance
  • Allow for easy management by clinical professionals
  • Lightweight and easily distributable software
  • Provide visualisations of data
  • Establish a national average for scores

While we are proud of this accomplishment, some features could be improved upon and deepened further with more time.

For instance, the test mode could be adjusted to allow users to test for a single topic or custom range of topics. Similarly, the view resources feature could allow the same kind of customization instead of listing all the resources.

Stability & Efficiency

The PRIDAR chatbot functions consistently well, much in part thanks to Voiceflow hosting the project, and the chosen backend integrations being reliable and giving fast API responses. The same is true of the bot’s Telegram deployment, which we are very proud of.

Those strengths however could rarely turn into weaknesses, for instance if one of our service providers is down for maintenance, features cannot work, and if this happens to be Voiceflow then the bot will be completely down. That being said, this is a rare occurrence and can be kept track of easily through the appropriate provider’s website.

Also, the internal algorithms used to sort content and search users are linear. While this works just fine for now, in the case of the bot becoming congested with content and users, this could slow it down and even time requests to the bot. The reason for these linear algorithms is a lack of time and difficulty implementing data structures in Voiceflow’s coding environment, which lags with complicated code, provides no syntax highlighting, and is restricted to a very small view with no scroll bar, making it hard to use.

Availability

We believe that the PRIDAR chatbot is extremely accessible to users, and was one of the key goals we kept in mind during development. This was achieved by using reliable services which are rarely ever down or not functioning as intended.

Our text-based approach and lightweight deployment played an arguably bigger part in this however, as users could interact with the bot regardless of their hardware, through either Telegram or Voiceflow’s web app, requiring just a url to visit.

Maintainability

A huge consideration going into this project was how clinicians would both update the bot’s content, as well as potentially extend it’s functionality. This drove our decisions in technology, and we believe it paid off.

The bot is entirely manageable through the databases spreadsheet interface, which is intuitive, and can be shared easily. This data can also be transported elsewhere, managed, and integrated with extra functionality (such as automatically emailing or running scripts on).

The Voiceflow project is very accessible due to it’s easily comprehensible block design, and the attention we paid to commenting on blocks and code thoroughly. In addition to this, we have created a step by step walkthrough of the components in the Voiceflow project, with highlighted examples and notes explaining the most intricate of details in order to help clinicians who aren’t familiar with the project or Voiceflow edit it down the line without issue.

Project Management

Our team allocated roles efficiently, based on both availability and ability. We always worked using version history (both for the website and Voiceflow project) in a collaborative manner.

As much as possible, we followed an AGILE approach to development, meeting our client Dr Connor every Thursday, before discussing feedback and creating a plan for the coming week together in a separate meeting. We would then have meetings at the start of the week to check our progress and evaluate our direction.

Our main mode of communication was through Whatsapp, which was extremely convenient and made us more accessible to one another. We conducted team meetings through Zoom at least once a week, and presented through MS Teams. We also used MS office to share documents and collaborate, particularly towards assessed deadlines.

Bugs

Thanks to our comprehensive testing approach, our bot has no bugs present.

However, certain limitations could cause issues in the future, such as database GET requests only returning a certain number of records, databases only storing a finite number of records, or linear algorithms taking too long causing the bot to time out. Additionally, the bot currently does not scale very well with topics. This means that the resources table and averages table must be updated accordingly by hand when editing the number of topics in the bot.

Other issues with the chatbot, such as lack of ability to comprehend different inputs, are attributed to it’s limited scope rather than unintended bugs in behaviour.

Future Work


Due to the time constraints in this project, there are several changes, both substantial and minor which can be made to improve the bot:

  • Accepting alternate and case insensitive inputs, or implementing intents based on user utterances
  • Allowing for customization of test topics and resource topics
  • Tracking score changes over time
  • Giving resources and option to take test on previously incorrect topics
  • Voice based input and deployment
  • Tweaking how buttons are displayed in Telegram deployment
  • Limiting the number of questions asked per topic, choosing a random subset to ask


The content of the bot can easily be modified and expanded through Jotform’s table interface. As for expanding upon the features of the bot, using the documentation provided, this should be possible without spending extra time figuring out how the system works. Switching to an NLP based chatbot experience would require a considerable rework of the bot, but would provide a much more natural and flexible experience.