Evaluation

MoSCoW Achievement List

ID Description Priority State Contributors
1 Low-cost chatbot generation service MUST ✅ Rifqi, Zhiyu, Yunsheng
2 The tool is adaptable to any NHS trusts' website MUST ✅ Rifqi, Zhiyu, Yunsheng
3 Chatbot can answer predefined questions (based on web scraping tool) MUST ✅ Rifqi & Yunsheng
4 Fast chatbot generation service website MUST ✅ Rifqi, Zhiyu, Yunsheng
5 Retrieval-based Chatbot (using intent classification, entity recognition, and response selection) MUST ✅ Zhiyu & Yunsheng
6 IT admins can download a dialog JSON file and upload it to IBM Watson Assistant MUST ✅ Zhiyu
7 User-friendly chatbot that can answer the question in a useful and friendly way MUST ✅ Rifqi & Zhiyu
8 IT admins can track the history of previously generated chatbots on the generation service website SHOULD ✅ Zhiyu
9 IT admins can directly preview the chatbot on the generation service website SHOULD ✅ Zhiyu
9 Use IBM NLU API to filter the data from our web scraping tool SHOULD ✅ Yunsheng
10 Provide a "Reference Code" for IT admins to get the dialog JSON file later SHOULD ✅ Zhiyu
11 Settings menu for changing IBM API Key and Watson Assistance Instance ID SHOULD ✅ Zhiyu
11 Help menu to provide deployment manual SHOULD ✅ Rifqi, Yunsheng, Zhiyu
12 IT admins can modify the chatbot dialogue via IBM Watson website (e.g., when covid responses were needed) COULD ✅ Rifqi, Yunsheng, Zhiyu
13 Use Azure Cognitive Service for Language to evaluate the answers and extract the most relevant information from Bing API COULD ✅ Zhiyu
14 About menu for linking to our project website COULD ✅ Rifqi, Yunsheng, Zhiyu
15 Display estimated waiting time for generation service COULD ✅ Zhiyu
16 Clean UI Design for the websites COULD ✅ Rifqi
17 Deploy the chatbot to Microsoft Azure HealthBot COULD Left for future work (est. time: 2 Weeks ⏲️) N/A
18 Use the latest GPT API in order to enhance the chatbot's answers COULD Left for future work (est. time: 4 Weeks ⏲️) N/A
19 Use Amazon Web Services (AWS) - no permissions WON'T HAVE ❌ N/A
20 Integrate chatbot into NHS website - their IT admins will do it WON'T HAVE ❌ N/A

Key Functionalities: 100%
Optional Functionalities: 70%

Contributions

Tasks / Work Packages Rifqi Zhiyu Yunsheng
Partners Liaison 50% 30% 20%
Requirement Analysis 34% 33% 33%
HCI 33% 33% 33%
Research and Experiments 30% 35% 35%
Frontend 20% 60% 20%
Backend Web Scraping Tool 30% 20% 50%
Backend API 20% 60% 20%
Azure Services 5% 90% 5%
Testing 33% 33% 33%
Development Blog 40% 30% 30%
Report Website 90% 5% 5%
Video Editing 30% 35% 35%
Overall 33% 33% 33%

Note: Contribution percentages are just a approximation, all technical and non-technical tasks were distributed equally between all members

Critical Evaluation

User Interface & Experience (UI/UX)

NHS Auto-chatbot offers an interactive and simple-to-use user interface to its users. The chatbot uses a style that is typical of messaging apps. The messages field is well-organized and intuitive. There are also "About" and "Help" menus that users can click on, to get further guidance on using the interface, if needed.

Functionality

NHS Auto-chatbot provides functionality that can answer all questions based on the specific NHS website. This functionality saves users (patients) a lot of time and effort to get medical questions. The chatbot can also answer predefined and important questions, such as an address, phone number, and opening hours.

Stability-Efficiency

To ensure that a high coverage (98%) was achieved and that all code segments and functions behaved as expected, NHS Auto-chatbot underwent a rigorous testing process that included numerous unit and systems tests. The chatbot's robustness was also tested, and it maintained proper operation even in the face of complex or invalid queries. A huge advantage of NHS Auto-chatbot is that it doesn't rely on any external software, which makes it self-sustainable and more stable.

NHS Auto-chatbot generates an answer quite quickly for predefined questions and can take up to a few seconds to answer other general and specific questions. The answer is based on the specific website, which the data retrieved from the web scraping process and Bing API. This makes the chatbot generation service runs very efficiently.

Compatibility

Because the primary chatbot logic is separated from the chatbot framework being used, NHS Auto-chatbot is very compatible (Django). The chatbot code's many auxiliary features make it simple to incorporate it for a variety of uses into any web application. The chatbot generation service can be deployed using both Windows and Linux/macOS.

Maintainability

Given that its chatbot dialogue can be updated whenever necessary, NHS Auto-chatbot is simple to maintain. The system is adaptable in that it can be made to serve a variety of needs. Because it uses the separation of concerns principle, the chatbot is scalable and future-proof and the code is kept clean.

Project Management

The team worked well together from the beginning, divided tasks fairly, and met once a week to discuss progress and make the necessary technical decisions to keep the development process on track. As a result, project management was simple. We used Discord and Kanban Board to communicate and track our project, respectively.

Bug List

ID Bug Description Priority
1 Our chatbot sometimes gets random answers, which though rarely happens. Low

Note: The probability that the chatbot will give a random answer is very low. For example, a really specific question might get a random answer. But throughout our testing, it only happened once.

Future Work

1. Try deploying the chatbot to Microsoft Azure HealthBot and integrating it into any website.

2. Use the latest GPT API in order to enhance the chatbot's answers. We used Bing API because, during our development phase, the latest GPT API hasn't been released yet.

3. Improve the privacy and security of our tool. For example, by making the patients' data more secure and private.

4. Process more unstructured data such as PDF & PowerPoint files and images.