Python Unit Testing
Python ships with its own unit testing framework which we have used to test the internal logic of the implemented APIs.
To measure coverage of our tests, we have used django-nose. Installation and use is covered here. Our total coverage is 73%, including the database models classes which have significantly lower coverage.
API testing
We additionally performed end-to-end testing by setting up a commandline client that queries the chat API. As this article rightly puts it, it is very important that chatbots end up saying the right thing. Specifically, edge cases such as the end of the conversation with the chatbot have been covered this way.
Postman
In order to automate those end-to-end API tests, and to ensure a certain conformity in our API design, we have additionally written tests with the tool Postman. In Postman, one can write arbitrary assertion code, and support is given for entering URI parameters etc. Finally, one can export test collections - which we have used for continuous integration
Newman is a tool to run Postman test collection in the commandline. We use Newman to automatically run the Postman tests for continuous deployment.
Automation
During the project, we have taken great care to ensure that the commands run to set up the system locally and on the server are the same. We have realised this by creating a variety of bash scripts. These scripts not only set up the development environment, but they also give an automated way to run Django unit tests and Postman tests.
By having the local development environment, the continuous integration server and the production environment execute the same code, we ensure high consistency - and prevent nightmares and nasty surprises.
Database Population Script
To improve the testability of our application, we created a script that populates our database with representative dialogue models. These are the basis of our automated unit tests.
User acceptance testing
Finally, after doing integration testing, we have carried out some acceptance testing. Concretely, we have asked a medical student and multiple non-technical people to simply use the chatbot. While this is a limited testing environment, we have identified some issues with the current state of the art and have taken actions to address those.