Testing Strategy
To ensure the effectiveness, efficiency, and user-friendliness of all functionalities, our development process included comprehensive testing and feedback mechanisms. Each functionality underwent rigorous testing by the team members responsible for its development, supplemented by evaluations from other teams and a diverse group of potential users. This approach provided valuable insights throughout the development phase. We conducted performance tests to evaluate the system's efficiency, alongside user acceptance tests designed for various game modes. Overall, the iterative testing and review process helped to ensure that each aspect of the system met our high standards of operation and user experience.
Performance Testing
For performance testing, we measured the time taken by our MFC to launch MotionInput after the user selects his/her desired configuration and clicks launch.
We conducted our performance tests on 3 different Windows PCs of our team whose specifications are as follows:
PC 1: Intel Core i5-1135G7 @ 2.4 GHz, 4 core(s), 8GB RAM (3200Mhz), Integrated Intel Iris Xe Graphics, Built-in HD Webcam
PC 2: AMD Ryzen 4500u CPU @ 2375 Mhz, 6 core(s), 8GB RAM (3200Mhz), Integrated AMD Radeon Graphics, integrated 720p camera
PC 3: AMD Ryzen 4600H CPU @ 3001MHz, 6 core(s), 16GB RAM (2400Mhz), NVIDIA GeForce RTX 3050 Graphics, Integrated 720p camera
Rocket League MFC
Test Number | PC 1 (in s) | PC 2 (in s) | PC 3 (in s) |
---|---|---|---|
1 | 17.58 | 14.2 | 9.54 |
2 | 8.13 | 9.83 | 7.22 |
3 | 7.23 | 9.56 | 8.64 |
4 | 7.29 | 10.19 | 6.10 |
Rocket League - Multiple Bodypoints MFC
Test Number | PC 1 (in s) | PC 2 (in s) | PC 3 (in s) |
---|---|---|---|
1 | 16.96 | 13.22 | 10.38 |
2 | 9.37 | 9.56 | 8.62 |
3 | 10.08 | 9.39 | 8.64 |
4 | 9.22 | 10.04 | 7.74 |
General Driving MFC
Test Number | PC 1 (in s) | PC 2 (in s) | PC 3 (in s) |
---|---|---|---|
1 | 15.80 | 14.58 | 9.48 |
2 | 10.45 | 9.27 | 6.33 |
3 | 10.13 | 9.69 | 6.89 |
4 | 9.82 | 9.34 | 7.05 |
General 2D Side Scroller Games MFC
Test Number | PC 1 (in s) | PC 2 (in s) | PC 3 (in s) |
---|---|---|---|
1 | 15.68 | 13.36 | 10.02 |
2 | 8.89 | 9.73 | 6.82 |
3 | 9.51 | 9.15 | 6.89 |
4 | 9.21 | 9.37 | 5.93 |
Lego Avengers MFC
Test Number | PC 1 (in s) | PC 2 (in s) | PC 3 (in s) |
---|---|---|---|
1 | 15.46 | 14.47 | 11.21 |
2 | 8.74 | 8.58 | 7.21 |
3 | 8.97 | 8.37 | 7.52 |
4 | 8.82 | 8.17 | 6.89 |
Batman Arkham Asylum MFC
Test Number | PC 1 (in s) | PC 2 (in s) | PC 3 (in s) |
---|---|---|---|
1 | 16.37 | 14.72 | 11.01 |
2 | 9.47 | 8.93 | 6.45 |
3 | 9.13 | 8.35 | 6.11 |
4 | 9.86 | 8.18 | 6.89 |
From the data in the tables above, a general trend can be observed that the time taken to launch MotionInput is very high for the first time. This may be due to the fact that MotionInput loads up for the first time. However after the first launch, the time becomes relatively lower. This may be due to the fact that the system caching stores some important information and speeds up the process for subsequent launches. The time taken to launch MotionInput from the MFCs is fairly good but it should be brought down as much as possible as in some cases, 10 to 15 seconds may be perceived as lag or delay by users.
User Acceptance Testing
We asked 4 testers to play games using our MFCs and asked them to rate on a scale of 1-5 (5 being the highest and 1 being the lowest) their level of engagement and usability. Some of them even provided feedback which has been collated in the following table. Please note that our testers chose to opt-out from being named in this process and therefore, shall remain anonymous.
MFC | Mode | Objective | Engagement rating | Usability rating | Comments |
---|---|---|---|---|---|
Rocket League | Steering (Speech OFF) |
Drive, boost, jump and brake | 4 | 3 | Moving hands/elbows may click buttons not intented to be clicked. |
Rocket League | Steering (Speech ON) |
Drive, boost, jump and brake | 4 | 4 | Turns are hard to make; Speech leads to less hovering hands/elbows. |
Rocket League | Joystick (Speech OFF) |
Drive, boost, jump and brake | 4 | 3 | NA |
Rocket League | Joystick (Speech ON) |
Drive, boost, jump and brake | 4 | 4 | NA |
Rocket League - Multiple Bodypoints | Steering | Drive, boost, jump and brake | 4 | 4 | Lots of options to choose from. |
Rocket League - Multiple Bodypoints | Joystick | Drive, boost, jump and brake | 4 | 3 | NA |
Forza | Steering (Speech OFF) |
Drive, boost, and brake | 4 | 2 | Turns are hard to make; Speech leads to less hovering hands/elbows. |
Forza | Steering (Speech ON) |
Drive, boost, and brake | 4 | 4 | Turns are hard to make; Speech leads to less hovering hands/elbows. |
Forza | Joystick (Speech ON) |
Drive, boost, and brake | 4 | 4 | NA |
Rocket League - General Driving | Joystick | Drive, boost, jump, and brake | 4 | 4 | Easy to choose game; GIFs helped me better understand how to play; Joystick placement allowed me to play as I am a lefty. |
Sonic - 2D Side Scroller | Speech OFF | Run forward, backward and jump | 4 | 4 | Controls easily respond. |
Sonic - 2D Side Scroller | Speech ON | Jump on enemies to kill them | 4 | 4 | Speech command give me flexibility in hand/elbow movement. |
Lego Avengers | Speech OFF | Move around | 3 | 4 | Took some time to get used to. |
Lego Avengers | Speech ON | Try to hit enemies and get combo moves | 3 | 4 | Buttons placed closely that help me build combo. |
Batman | Speech ON | Move around | 3 | 4 | Nose navigation takes some time but once I got used to, it was great. |
Batman | Speech ON | Move around | 3 | 4 | Nose navigation takes some time but once I got used to, it was great. |
Batman | Speech ON | Change between Player and Detective game mode | 4 | 4 | Simple facial gestures help do the trick. |
Batman | Speech ON | Hit enemies and throw batarang | 4 | 4 | Easy to click buttons; Grapple and batarang can be accessed easily by speech. |
From this iterative test and feedback mechanism, we came to certain conclusions:
For driving games, speech commands helped de-clutter the amount of on-screen buttons and testers felt that apart from turning , rest of the movements were smooth and fluid. The General Driving MFC was convenient for the testers by letting them launch any game they want from a simple dropdown. Provision of left and right joystick positions helped users who wanted a different layout.
The multiple bodypoints gave testers a wide range of options of bodypoints to choose from, but most stuck with either hands or elbows.
Sonic provided testers with a quick-paced game that had simple controls but tricky levels at the same time. Enabling speech commands with on-screen buttons hepled them effectively time their movements.
Lego Avengers and Batman are big and complex games and hence had a bit of learning curve. However, after some time the testers thoroughly enjoyed playing these games.
Independent from all the games, we found that some testers did not like the skeleton silhoutte when using MotionInput, so we provided them with two visual modes: Ghost mode and Bodypoints Visualiser mode. Ghost mode allows users to see themselves while playing, with the option to manage the opacity of this camera video. Bodypoints Visualiser maps the chosen bodypoints (i.e. hands or elbows) as dots on the screen.