Because NHS Auto-chatbot is a client-side application, it is crucial to run a wide range of tests on it to make sure it is stable, usable, and functional. This guarantees that it will live up to the requirements of the client and any other users of the application.
Testing Scope
1. We must take into account the unpredictable behaviours of people since our chatbot will
be used by people (IT admin, teenage patients, and elderly patients). For instance, typos.
Therefore, we thoroughly tested NHS Auto-chatbot to make sure it can handle and respond to
any
uncertain actions correctly and, where possible, provide clear guidance and responses.
2. Wide-ranging features offered by NHS Auto-chatbot include answering predefined & general
questions
and obtaining information from the specific NHS trust website. Therefore, it is essential to
thoroughly test each of these features to make sure the programme is reliable and that users
can easily interact with it.
3. The goal of NHS Auto-chatbot is to assist NHS IT admins in creating their own chatbot so
that NHS
patients can use it. As a result, we tested the application to make sure it satisfies their
needs and requirements.
Testing Methodology
1. Unit Testing was used to test NHS Auto-chatbot's built-in features.
2. Integration Testing was used to test the effectiveness and reliability of chatbot's
dialogues.
3. User Acceptance Testing was used to evaluate the NHS Auto-chatbot's UI and design.
4. API testing was used to test the interactions between different components.
Testing Principle
We carried out various tests as the NHS Auto-chatbot application was being developed. In
order to make
sure the system behaved correctly and as expected, we continuously wrote and tested it using
various inputs. Any tests that failed were looked into, and any bugs discovered were fixed
right away. We created various unit, integration, API, and system tests after completing the
functionality to test as much of the code as we could.
For our functions, we created unit tests. Python's unittest was used to accomplish this. Our web scraping method, filtering content method, duplication on filtering, and tokenization method were all covered by our unit tests. We tested various potential inputs in order to cover the majority of the code.
We generated system and integration tests for the entire generation service system and chatbot by creating tests for all potential inputs, such as greetings, predefined and general questions that needed to be retrieved from our web scraping tool and Bing API, questions with no answers, etc. System, integration, and unit tests taken together cover 98% of our functionality.
To ensure that the interactions between different components are functioning correctly and
producing accurate results, we did API testing using Insomnia for Azure
Functions and Django. For Azure Functions, we tested the API of Bing Access, Sending
messages to Watson Assistant, etc. For Django, we tested the URL & JSON interactions.
Azure Functions API Test
Django API Test
We asked four people to test NHS Auto-chatbot for us because it is crucial that it fulfils user needs and requirements and to get helpful feedback. This included our client as well as the pseudo-users who assisted us in gathering requirements.
Testers
Testers include:
These users were selected because they are likely to use the NHS Auto-chatbot, such as the NHS patient and a mother that frequently ask about medicine or prescription and researchers that ask about specific data regarding the NHS's website. These users come from various professional backgrounds, and none of them is an expert in technology or software.
Test Cases
We divided the test into 4 cases, and the testers were responsible for reviewing each case
and providing feedback. The feedback was based on the acceptance requirements that were
provided to the testers, who were asked to rate each requirement on a Likert scale and add
their own comments.
Feedback from Testers
| Acceptance Requirement | Totally Disagree | Disagree | Neutral | Agree | Totally Agree | Comments |
|---|---|---|---|---|---|---|
| NHS Auto-chatbot runs to end of the job | 0 | 0 | 0 | 0 | 4 | + Everyone agree NHS Auto-chatbot can perform the job from start to end |
| Clear response to every action of users | 0 | 1 | 0 | 2 | 1 |
+ The answers provided by the website are very clear
- Sometimes error answers appear, but rarely happen |
| Accurate answers in question answering | 0 | 0 | 0 | 1 | 3 |
+ Very accurate answers for some specific questions
- NHS Auto-chatbot did not manage to answer all very specific questions |
| Accessible on any browser | 0 | 0 | 0 | 0 | 4 | + Everyone agree NHS Auto-chatbot works well on their web browsers |
| Fast response | 0 | 0 | 0 | 0 | 4 | + Overall response is fast |
| Friendly error messages | 0 | 0 | 1 | 2 | 1 |
+ If the answer is not on the web, error messages are friendly
+ If the answer is on the web, but with low confidence score, error messages will give a suitable link - Standard error messages |
| Easy to use/navigate | 0 | 0 | 0 | 0 | 4 |
+ Clear icons and buttons
+ Easy to use interface like a social media chat |
| Answers received | 0 | 0 | 0 | 1 | 3 |
+ Chatbot answers all questions, regardless the confidence score
+ Question always answered |
Conclusion
We are happy to see that the testers were enthusiastic and had positive views towards NHS
Auto-chatbot as they interacted with it and provided feedback. Their opinions—both positive
and negative—meant a lot to us. These comments have guided us and will continue to guide us
as we improve NHS Auto-chatbot to make it more sophisticated and comprehensive so that it
can assist an increasing number of people.