COMP0016 Team 15

To test the backend, we wrote a series of system and unit tests with pytest which use a temporary sqllite mock database. The backend is complex, and testing functionality manually after making changes to make sure the system still worked quickly became extremely time consuming. We wrote these tests to automate this process and allow the developer to run the test suit before each PR to verify the system is working as it should.

We set up our database connection in src/api/database.py. If we are running tests, we create an sqllite database file in ‘src/tests/data’ and use that as our database URL, otherwise we use the production database URL defined by the DB_URL environment variable, which is located in environments.env file in the deployment directory, and set when the backend container is run.

The data directory contains a dummy model and a mock dataset, as well as the test database when the tests are run.

We use these mock models and datasets for testing the model and dataset upload endpoints in our system tests.

We use FastAPI’s test client to run system tests. To set everything up, we define three users, two with admin permissions, and one with read only permissions. Since our endpoints depend on the current user, we need these to test functionalities such as not being able to delete someone else’s model or not being able to upload a model with the wrong permissions.

Here is the admin user for example:

Before running our tests, we populate our sqllite test database with the schema defined in ‘src/models’:

We then override the authentication dependency of our endpoints, that way we don’t have to deal with tokens and it allows the tests to be extendable if we switch to using an external authentication provider which may charge per request or where requesting a token may take time.

Now we are ready to test! We can see below we define our test client with our FastAPI app defined in main, and I’ve copied in our first system tests, which tests creating a user as an example:

We have 34 system tests in total, which test each code-branch of each endpoint.

For our unit testing, we have the directory test_crud, and in that directory there are a set of files which test each of the CRUD utilities for each table in our database.

We use setUp and tearDown methods for setting up mock database data. Here is an example from test_datasets:

Here we set up a mock database with records of a user and a dataset. After the test is complete, we delete the database.

Below is an example test for testing crud.datasets.get_dataset_by_name:

For each pull request we would run this test suit by running pytest in the src directory. If all 55 tests pass, we go ahead with the pull request. Obviously, this is not an optimum solution. If we had more time, I would have loved to have set up automated CI with something like GitHub Actions, Jenkins, or Travis. But since our backend image requires docker in docker, it makes it not so trivial to set up, and in the time frame we settled for this solution.