Our testing strategy has been clear from the beginning, and was outlined by the philosophy of agile development, specifically continuous testing. To put it simply, this is a testing strategy of testing early, often and everywhere.
In the next couple of sections, we will explain the types of tests that we have conducted. Chosen supporting data on these tests can then be found at Test Cases & Feedback from testers and project partners.
To a certain extent, we have also completed integration testing of our components. This is because we have tested the individual components, such as the Apple Watch, the Kinect and the server individually and after thoroughly testing these components, we have tested these systems together, integrated. For example, the merging functionw as tested when we send data from the Kinect and Apple Watch to the server.
We have tested the Apple Watch code on all of the Apple Watch Series in the XCode Simulator, since we did not have them on hand physically. The Kinect had to be designed for the Kinect 2 specifically, so compatibility testing did not apply.
Responsiveness design testing was only useful for the Apple Watch code by using the Apple Watch simulator which the users tested on. They tried to use different version/options of the UI and provided spoken feedback for each. This type of testing led to the final design of the Apple Watch UI. For example, the users wanted to see live data on the display as well as being to view the data after their session. Therefore, the UI was modified to be able to display live user health statistics throughout the session.
We were unable to perform effective stress testing due to hardware limitations as we only disposed with one Apple Watch and two Kinects.
Nonetheless we have tried to type in session data manually as well as send it through the devices at the same time, along with creating many sessions in a very short period. Even though this was not ideal, it was the best we could do in this situation.
Our project partners provided us with contant feedback on our bi-weekly calls. Here are a couple examples: They have often directed us on the right track when we thought of features that were outside the scope of the project such as determining the type of dance by analyzing the music. Furthermore, we had the Kinect stop the dance session after the hands of a user were raised for 5 seconds. The clients thought that this could lead to ending the session accidently as some dance moves require the users hands to be raised. They therefore suggested that we could end the dance session when the user steps out of the frame for a certain time period which is how the final implementation works. They also gave us ideas about more health metrics that we could record such as maximum limb height.
Overall, the project partners were very satisfied with what we managed to implement.
We user-tested our whole system on 5 users, fitting the Jakob Nielsen [1] suggestion that 3 to 5 evaluators is optimal. Despite them being our friends, we told them ot be as harsh as they wanted and not to hold back. Here's what they had to say:
| Timestamp | Name | Apple Watch UI: | Kinect UI: | Server UI: | |||
|---|---|---|---|---|---|---|---|
| Like? | Comments: | Like? | Comments: | Like? | Comments: | ||
| 3/19/2020 19:31:48 | Rajesh Goyal | ✔ | I think the initial screen is done quite well and does not allow for user error which is great. It is really cool that it measures so much interesting data and utilises the apple watch accelerometer to measure distance and spins and everything, really cool. | ✔ | WOW! THATS SO COOL! I hope there's some sort of portal to view the videos but its fine if there isn't, that's crazy cool. The UI was pretty easy to read, simple instructions, good disclaimer from the Kinect message for transparency with the user. | 🔧 | The time stamp field still allows for erroneous input, maybe calendar selection if the format is dd-mm-yyyy. The way it stops you from entering letters, it doesnt stop the letter "e". This may have broken the server... |
| 3/19/2020 19:47:38 | Rikaz Rameez | ✔ | It is very staightforward and easy to use | ✔ | The skeleton is very useful in seeing how the Kinect tracks your limbs. Maybe some prompts on if you are too close/too far from the kinect would be useful. I really like the setup process - very descriptive and straightforward. | 🔧 | The server doesn't have units for the headings for each metric which makes it unclear exactly what each number means. More detailed time stamps would be better, and if the column headings in the table scrolled with the headings it would help in seeing the data representations better. |
| 3/19/2020 19:55:24 | Joel Morgan | ✔ | N/A | ✔ | I enjoyed testing this! | ✔ | It would be good if the delete button would refresh the page, but it's great otherwise. |
| 3/19/2020 20:04:32 | Eesha Irfan | 🔧 | I think it should be clear that when you stop measuring on the apple watch, it does not stop the video on the Kinect. | ✔ | N/A | 🔧 | I think it was hard to remember all the different column titles. |
| 3/19/2020 20:53:24 | Govind Balla | ✔ | Make the metrics screen more colourful, ie differentiate between calories and bpm etc. Other than that, it is good | ✔ | The only thing I would change it to split it into a different colour scheme and font. | ✔ | Looked good |
Here's a short compilation of the users playing with the system:
This type of testing happened during the development process: for each new method we created, we tested its functionality in a variety of situations and made sure it produced repeatable and reliable results.
Test method: having user performing actions at different position to the camera.
Test result:
| X Co-ordinate | Y Co-ordinate | Z Co-ordinate | |
|---|---|---|---|
| Position 1 | -0.195 | 0.796 | 2.012 |
| Position 2 | 0.451 | 0.779 | 2.124 |
Comment: the co-ordiantes of each joint are obtained successfully. As you can see from the table from position 1 to 2, when users move from the left of camera to the right, the X co-ordinate (horizontal distance) changes but Y (height) and Z (vertical distance) remains about the same. However, if the joint is behind the body or other object so that it can not be seen by the camera, the co-ordinate of that joint will be random value and keeps changing.
Activity level is a method that calculate the average displacement of each joint moved between per frame; we can use this result to determine how active the person in front the camera is.
Test method: having user performing actions at different active level.
Test result:
| Stand Still | Gentle Movement | Vigorous Movement | |
|---|---|---|---|
| Test 1 | 0.001 | 0.016 | 0.116 |
| Test 2 | 0.002 | 0.014 | 0.132 |
Comment: the value is almost zero when the user stands still and the faster and the further user moves, the higher the activity level value.
Video:
The angle between different joints can be calculated when the joint name is provided, however, some joint angles are set to be the angle between the two joints specified (which forms a bone) and the person's body (which is obtained by the top, middle, and bottom spine joints).
Test method: having user performing actions at different angle.
Test result:
| Right angle | Parallel | |
|---|---|---|
| Right Shoulder Joint | 81°-97° | 7°-9° |
| Right Elbow Joint | 87°-93° | 177°-179° |
Comment: The angle result can successfully represent the actual angle between joints. As the table shows, the results are not stable and can have a small range of error, this is due to the fact that we observed joint angles by using the bones, but the Kinect calculates it using joint positions, so its result angle may not be fully realistic.
Video:
We have a method using the gyroscope and accelerometer inside the apple watch to identify the number of turns the person performed.
Test method: having a person perform by turning at different angles and speeds, and comparing the results with the amount of spins that actually happened.
Test result:
| Slow Spins | Normal Spins | Fast Spins | Spins at different angles | |
|---|---|---|---|---|
| Actual Spins | 50 | 50 | 50 | 50 |
| Recorded Spins | 39 | 54 | 65 | 46 |
Comment: In most condition the turns are recognised properly, however when you are turning too fast, the method result appears to be higher than the actual result.
Our system saves a video each time: we checked the video saved (and watched the live feed) to identify and fix bugs. Here's an example:
We set up the UCL South Cloisters (where various University Dance groups often practice and choreograph performances) as if it were a studio, using the software we created.
This was a full Kinect system test, with a typical user: our friend Rachel (who is a dancer) got to test our system and give us feedback.
When we decided to implement a user-entered Session ID, we tested how feasible this was on the Apple Watch: