Testing Strategy
In order to make sure that all the functionalities implemented are properly working, each of the team members tested their module as it was developed. During development the system was also tested by other teams as well as a wide range of other potential users through micro builds - this gave us extensive feedback throughout the development process, Moreover, we run performance testing to prove the performance achieved in terms of time and memory efficiency. To test the user acceptance, tests were designed for each module, also for multiple modes combined. For the 27 tests, we got reviews from 4 testers.
Performance Testing
In order to demonstrate the running time of the code in different situations, we used the
cProfile library in Python. As well as this, to find the memory usage over
time, we utilised the Memory Profiler library. For each measurement, besides time, there are
also mentioned the number of function calls (request made to perform a predetermined function)
and primitive calls (the calls were not made through recursion).
The machine used for performance testing had the following specifications: Intel Core i7-9750H CPU @ 2.60Ghz, 2592 Mhz, 6 Core(s), 16 GB RAM, Graphics card NVIDIA GeForce GTX 1650.
Initialization
All the initializations for MotionInput v3 are completed in MotionInputAPI.start()
. Using the cProfile library, the following time was achieved: 1.462 seconds
Frame rate for each module
In order to find the frame rate for each module on its own, the running time was recorded for 100 frames. The results are shown in the table below
.
Module(mode) | Running time (for 100 frames) | FPS | Comments |
---|---|---|---|
Hand (basic_hand mode) | 100909 function calls (99407 primitive calls) in 3.465 seconds | 28.86 | This time is registered when the user is running the mode and moves the mouse constantly. |
Hand (basic_hand mode) | 19852 function calls (18650 primitive calls) in 1.531 seconds | 65.31 | This time is registered when the user is running the mode, but he is inactive. Because the Hand module has the idle mode, this time is significantly lower (more than 2 times less). |
Body (extremity_triggers) | 19848 function calls in 3.819 seconds | 26.18 | This time is registered when the user is running the mode and triggers events constantly. |
Body (extremity_triggers) | 18953 function calls in 3.769 seconds | 26.53 | This time is registered when the user is running the mode without performing any action. |
Head (head_nose_position) | 19555 function calls in 4.208 seconds | 23.76 | This time is registered when the user is running the mode and moves the mouse constantly. Since for the Head module there is no idle state, the time would not differ because the program always tracks the user's face movement. |
Head (head_nose_position) | 16686 function calls in 4.093 seconds | 24.43 | This time is registered when the user is running the mode, but no face is detected. |
Eye (eye_tracking) | 46411 function calls (42811 primitive calls) in 7.581 seconds | 13.19 | This time is registered when the user is running the mode and moves the mouse constantly. |
Eye (eye_tracking) | 14696 function calls (14496 primitive calls) in 3.422 seconds | 29.22 | This time is registered when the user is running the mode, but no face is detected. |
The modes used are composed of the following sets of events:
basic_hand mode: Zoom, Open keyboard, Scroll, Idle, AOI resize, Double click, Monitor change, Mouse movement, Left press, Right press
extremity_triggers: Up, Arm right, Arm left, Kick left, Kick right, Punch left, Punch right
head_nose_position: Raise eye brow, Nose tracking, Smiling, Fish face, Open mouth
eye_tracking: Eye tracking
Here we can see that the hand module has the highest performance, and the eyes module by
far the worst, due to the complexity of gaze inference. The Head module, due to its large number
of landmarks (468), follows that it is slower than the Hand and Body modules that also use
MediaPipe solutions.
Both hands and eye module
have an idle state where the efficiency more than doubles, while the Head and Body modules are
not impacted as much by the presence of the user in the frame.
Frame rate for each module in combination with hand module
At the moment, for switching between modes we predominantly use the "rocknroll_next_mode" event.
This event uses the hand module, which means that if the mode in use is from a different module, the
hand module will need to be proccessed in each frame in addition to the main functionality module.
Hence we ran testing on each module, using it together with the "rocknroll_next_mode" event and compared
the FPS to that, achieved in the previous table.
Module(mode) | FPS without the Hand Modules event | FPS with Hand Modules event | Comments |
---|---|---|---|
Hand (basic_hand mode) | 28.86 | 28.79 | This time is registered when the user is running the mode and moves the mouse constantly. |
Hand (basic_hand mode) | 65.31 | 65.18 | This time is registered when the user is running the mode, but he is inactive. Because the Hand module has the idle mode, this time is significantly lower (more than 2 times less). |
Body (extremity_triggers) | 26.18 | 24.77 | This time is registered when the user is running the mode and triggers events constantly. |
Body (extremity_triggers) | 26.53 | 25.74 | This time is registered when the user is running the mode without performing any action. |
Head (head_nose_position) | 23.76 | 23.52 | This time is registered when the user is running the mode and moves the mouse constantly. Since for the Head module there is no idle state, the time would not differ because the program always tracks the user's face movement. |
Head (head_nose_position) | 24.43 | 24.24 | This time is registered when the user is running the mode, but no face is detected. |
Eye (eye_tracking) | 13.19 | 13.9 | This time is registered when the user is running the mode and moves the mouse constantly. |
Eye (eye_tracking) | 29.22 | 28.97 | This time is registered when the user is running the mode, but no face is detected. |
As shown, adding the hand modules event does not significantly damage the frame rate. As we were initially running the processing of multiple modules in sequence, the performance was closer to the sum of the time each of the modules took separately. However, as we implemented multithreading we were able to achieve the results above.
Frame rate for combined modes
Similarly, we tested the performance when combining the hand, head and eyes modules. As the eyes module currently only allows for mouse navigation, it is in practice commonly used in combination with the head module and sometimes also hand module.
Module(mode) | Running time (for 100 frames) | FPS |
---|---|---|
Hand module (basic_hand mode): | 100909 function calls (99407 primitive calls) in 3.473 seconds | 28.79 |
Head module (head_nose_position) | 19555 function calls in 4.250 seconds | 23.53 |
Eye module (eye_tracking) | 46411 function calls (42811 primitive calls) in 7.581 seconds | 13.19 |
Head and Eye | 59171 function calls (55607 primitive calls) in 8.715 seconds | 11.48 |
Hand, Head and Eye modules combined: | 55064 function calls (51464 primitive calls) in 9.834 seconds | 10.17 |
Once again we can see that combining modules dues not result in the summation of their separate running times.
Performance over a longer period
For these tests, the Hand module is used (basic_hand mode). The time is measured when running the system for 50, 100, 250, 500, 750, 1000, 1500 and 2000 frames respectively.
Nr of frames | Running time using the basic_hand mode | FPS using the basic_hand mode | Running time in idle_hand mode | FPS in idle_hand mode |
---|---|---|---|---|
50 | 50820 function calls (50066 primitive calls) in 1.891 seconds | 26.44 | 10402 function calls (9800 primitive calls) in 0.898 seconds | 55.67 |
100 | 100909 function calls (99407 primitive calls) in 3.473 seconds | 28.79 | 19852 function calls (18650 primitive calls) in 1.534 seconds | 65.18 |
250 | 254097 function calls (250323 primitive calls) in 8.258 seconds | 30.27 | 48202 function calls (45200 primitive calls) in 3.320 seconds | 75.3 |
500 | 377008 function calls (365621 primitive calls) in 13.949 seconds | 35.84 | 95452 function calls (89450 primitive calls) in 6.782 seconds | 73.72 |
750 | 763303 function calls (751989 primitive calls) in 24.375 seconds | 30.76 | 142702 function calls (133700 primitive calls) in 9.858 seconds | 76.08 |
1000 | 808066 function calls (786847 primitive calls) in 29.893 seconds | 33.45 | 189952 function calls (177950 primitive calls) in 13.404 seconds | 74.6 |
1500 | 1479151 function calls (1457269 primitive calls) in 50.001 seconds | 29.99 | 275002 function calls (257600 primitive calls) in 18.952 seconds | 79.14 |
2000 | 1984172 function calls (1954764 primitive calls) in 67.519 seconds | 29.62 | 369502 function calls (346100 primitive calls) in 26.835 seconds | 74.52 |
As we can see the frame rate remains relatively constant as the system runs, thus the MotionInput v3 runtime is not affected by the length of the running period. This suggests that the system remains stable, and will allow for long durations of up-time.
Memory usage
We monitored the memory usage of MotionInput while the user constantly switched between multiple modes (basic_hand, head_nose_position, eye_grid_facial). As we can see there is an initial increase of memory reflecting the modules being first initialized by the system (most of them are maintained in memory after being used once as the initialization is time-consuming and it is assumed to be used again during the given runtime). Afterwards, however, as the modes are constantly iterated over there is no increase in memory usage. This again shows that MotionInput is sustainable for multiple modes of use without having to restart the application.
User Acceptance Testing
As the GUI was not part of our project testing was done on a compile that immediately started the camera view and the MI backend, the functionality was adjusted through the config files.
For testing we used 9 different modes listed below:
-
Modes
-
Mode 1
Mode config
{ "default": "basic_hand", "iteration_order": { "idle_hand": "idle_hand", "basic_hand": "basic_hand" }, "modes": { "basic_hand": [ "rocknroll_next_mode", "palm_height_change_aoi_resize_left_hand", "double_index_pinch_zoom", "palm_center_move_mouse_left_hand", "hand_to_idle_mode_left_hand", "middle_pinch_right_press_left_hand", "Fist_doubleclick_left_hand", "double_pinch_double_click_left_hand", "index_pinch_left_press_left_hand", "index_middle_scroll_left_hand" ], "idle_hand": [ "hand_to_active_mode_left_hand" ] } }
Instructions on how to use the mode
NB! The basic hand mode is currently configured for left-handedness.
To move the mouse, move your left hand while facing the camera - the mouse moves reflect the position of your palm in respect to the blue rectangle on the screen.
To left-click pinch only your index finger on only the left hand
To right-click pinch only your middle finger
To double click clench your fist or pinch index and middle fingers at once
To zoom pinch the index finger on both your hands and move the hands closer together or further apart
To scroll fold your pinky and ring finger, stretch the thumb, hold the middle and index finger together and move them up and down.
-
Mode 2
Mode config
{ "default": "extremity_triggers", "iteration_order": { "extremity_triggers": "extremity_triggers" }, "modes": { "extremity_triggers": [ "rocknroll_next_mode", "punch_left_extremity", "arm_left_extremity", "kick_left_extremity", "up_extremity", "kick_right_extremity", "arm_right_extremity", "punch_right_extremity" ] } }
Instructions on how to use the mode
This mode has 7 extremity triggers activated, which are displayed as circles on the view. Each trigger has a part of the body that activates it (nose, left or right wrist, and left or right ankle), an action (key press, key down, left or right click), and a keybind if required.
The extremity triggers in left to right order are:
punch_left (left wrist, key press, “h”)
arm_left (left wrist, key press, “a”)
kick_left (left ankle, left click)
up (nose, key down, “w”)
kick_right (right ankle, right click)
arm_right (right arm, key press, “d”)
punch_right (right arm, key press, “s”)
To activate a trigger and its action, move the corresponding landmark/body part to the circle, which will turn green if activated, and will increment the count once deactivated.
-
Mode 3
Mode config
{ "default": "exercise_no_equipment", "iteration_order": { "exercise_equipment": "exercise_equipment", "exercise_no_equipment": "exercise_no_equipment" }, "modes": { "exercise_no_equipment": [ "squating_exercise", "walking_exercise" ], "exercise_equipment": [ "rocknroll_next_mode", "cycling_exercise" ] } }
Instructions on how to use the mode
The exercise modes consist of a no equipment mode, and an equipment mode. Each exercise in the modes have an action (key press, key down, left or right click), and a keybind if required.
In the exercise_no_equipment mode, the activated exercises are:
walking (key down, “w”)
squatting (key press, “s”)
In the exercise_equipment mode, there is one activated exercise:
cycling (key down, “w”)
Performing the motion of an exercise (e.g. walking on the spot) will activate the exercise, causing its action to be performed. For walking and cycling, there is an interval (0.5 seconds by default) after an exercise stops being detected where the key (“w” in both cases) will continue to be held down. This delay is intended to help simulate constant holding down in between movements (e.g. switching between legs in walking).
-
Mode 4
Mode config
{ "default": "extremity_walking", "iteration_order": { "extremity_walking": "extremity_walking" }, "modes": { "extremity_walking": [ "extremity_walking_event" ] } }
Instructions on how to use the mode
This mode allows for both walking on the spot and some extremity triggers to function with each other. There are 3 extremity triggers activated in this mode, each with a keybind:
arm_left (left wrist, “a”)
up (nose, “w”)
arm_right (right wrist, “d”)
This mode allows for a current key to be set, which is done by activating one of the extremity triggers, which changes the current key to its keybind. Walking on the spot will cause the current key to be held down.
For walking, there is an interval (0.5 seconds by default) after walking is stopped being detected, where the current key will continue to be held down. This delay is intended to help simulate constant holding down in between walking movements (e.g. switching between legs).
-
Mode 5
Mode config
{ "default": "gaming_extremity_walking", "iteration_order": { "gaming_extremity_walking": "gaming_extremity_walking" }, "modes": { "gaming_extremity_walking": [ "gaming_extremity_walking_event" ] } }
Instructions on how to use the mode
This mode is a special gamepad mode, which allows for walking on the spot to be used in conjunction with extremity triggers.
In the default configuration, you will see two pads of 4 triggers on the screen.
The right pad is the button pad. Each of these triggers has a keybind, and activation causes the key to be held down. The extremity triggers for this pad are:
left_button (right wrist, “j”)
right_button (right wrist, “b”)
up_button (right wrist, “k”)
down_button (right wrist, “l”)
The left pad is the walking pad, and these triggers are used in conjunction with walking on the spot. When an extremity trigger from this pad is triggered while walking is detected, then its keybind is held down. Walking without having any extremity trigger from this pad will not cause an action, and vice versa. The extremity triggers for this pad are:
left_walking (left wrist, “a”)
right_walking (left wrist, “d”)
up_walking (left wrist, “w”)
down_walking (left wrist, “s”)
Additionally, there are two triggers at the top of the screen, which are button extremity triggers and are meant to act as start and quit buttons. From left to right, they are:
start_key (left wrist, “space”)
quit_key (right wrist, “esc”)
For walking, there is an interval (0.5 seconds by default) after walking is stopped being detected, where whatever key of any extremity trigger on the left pad that is activated will continue to be held down. This delay is intended to help simulate constant holding down in between walking movements (e.g. switching between legs).
-
Mode 6
Mode config
{ "default": "head_nose_position", "iteration_order": { "head_nose_position": "head_nose_position" }, "modes": { "head_nose_position": [ "smiling_event", "open_mouth_event", "raise_eye_brow_event", "nose_tracking_event" ] } }
Instructions on how to use the mode
This mode is the first of the two head modes, using nose position.
To move the mouse, move your left hand while facing the camera - the mouse moves reflect the position of your palm in respect to the blue rectangle on the screen.
To left-click, open your mouth.
To right-click, smile.
To double click, raise your eyebrows.
-
Mode 7
Mode config
{ "default": "head_nose_direction", "iteration_order": { "head_nose_direction": "head_nose_direction" }, "modes": { "head_nose_direction": [ "smiling_event", "open_mouth_event", "raise_eye_brow_event", "nose_scroll_event", "nose_zoom_event", "nose_direction_tracking_event_nose_box" ] } }
Instructions on how to use the mode
This mode is also made for the head module, using nose direction.
To move the mouse, move your nose while facing the camera - the mouse moves in the same direction as your nose tip in respect to the small blue rectangle on the centre of the screen.
To left-click, right-click and double click, use the exact same gestures as in Mode 6.
To scroll, smile while moving your nose tip up(to scroll up) and down(to scroll down).
To zoom, perform a fish face gesture while moving your nose tip to the left(to zoom in) and to the right(to zoom out).
-
Mode 8
Mode config
{ "default": "hand_mouth", "iteration_order": { "hand_mouth": "hand_mouth" }, "modes": { "eye": [ "palm_height_change_aoi_resize_left_hand", "palm_center_move_mouse_left_hand", "smiling_event", "open_mouth_event", "raise_eye_brow_event" ] } }
Instructions on how to use the mode
This mode allows you to move the cursor using your eyes. Also, it is combined with head gestures for clicking events.
To move the mouse, move your eyes while facing the camera - the mouse moves to the position on the screen you are looking at.
To left-click, open your mouth.
To right-click, smile.
To double click, raise your eyebrows.
-
Mode 9
Mode config
{ "default": "extremity_triggers", "iteration_order": { "extremity_triggers": "exercise_no_equipment", "exercise_no_equipment": "extremity_triggers" }, "modes": { "extremity_triggers": [ "rocknroll_next_mode", "punch_right_extremity", "up_extremity", "arm_right_extremity", "arm_left_extremity", "punch_left_extremity", "kick_left_extremity", "kick_right_extremity" ], "exercise_no_equipment": [ "rocknroll_next_mode", "squating_exercise", "walking_exercise" ] } }
Instructions on how to use the mode
This mode combines the extremity trigger mode (mode 2) and the no equipment exercise mode (mode 3).
Since there are two modes in the iteration, they can be switched between by making the “rock-n-roll” gesture with both hands.
-
Mode 1
And then with each of the modes created one or multiple tests with the following instructions
-
Tests
Test ID Mode ID Test preparation Test description 1 1 Create a test folder on the desktop with one .txt file that contains a very long text. Open the test folder using the fist double click 2 1 Open the txt file in the folder using the index and middle finger double click 3 1 Verify that the handlers/scroll/speed in config.json is appropriate Attempt to scroll through the txt file 4 1 Verify that the handlers/zoom/speen in config.json is appropriate (mine was 20) Attempt to zoom in and out of the txt file 5 1 Go back to the folder and make a copy of the document (right click -> copy and then right click -> paste) 6 1 drag the document out of the folder and onto the desktop. 7 2 Verify that the extremity triggers with keybinds set actually causes the key to be pressed or held down Open a writable text document (on notepad or word), and attempt to activate any of the extremity triggers (except left_kick and right_kick), and check that each extremity trigger causes its circle to turn green, and its corresponding keybind to be pressed 8 2 Verify that an extremity trigger with an action set to "key_down" indeed holds down the key On a text document again, attempt to trigger and hold the up extremity, and verify that for the duration of its activation, "w" is typed to screen 9 2 Verify that an extremity trigger with a clicking action set ("left_click" or "right_click") indeed causes a mouse click Manually move the mouse cursor to the "close" button of any window, and then activate the left_kick extremity to attempt to close the window by causing a left click action. Then, moving the cursor to an empty part of the desktop, attempt to activate the right_kick extremity to attempt to open up the context menu by causing a right click action 10 3 Verify that the exercise with a keybind set actually causes the key to be pressed or held down On a text document, attempt to activate an exercise, and check that the keybind assigned to the exercise appears on screen 11 3 Verify that an exercise with an action set to "key_down" indeed holds down the key On a text document again, either perform the walking or cycling exercise, and check that multiple keys are typed as you perform the exercise 12 3 Verify that an exercise with an action set to "key_down" has an interval after the exercise is deactivated where the key continues to be held down, and that this may facilitate a more fluid action Firstly, on a text document, check that there is a short duration after you stop walking on the spot or cycling, that a few extra keys are typed. Then, on any game that allows holding the "w" key for walking, attempt to walk on the spot at a non-exaggerated pace, and check that the game character seems to be continuously walking without any weird break when you switch legs 13 3 Verify that if the equipment mode is selected, then attempting to perform a no equipment exercise will not cause it to be detected/an action to occur (and vice versa) With the equipment mode selected, attempt to walk on the spot or squat, and verify nothing happens. If feasible, also attempt to perform cycling while no equipment mode is selected. 14 4 Verify that activating one of the extremity triggers causes the current key to change In a text document, walk on the spot to cause a key to be typed. If no extremity trigger has been activated before this point, this should be the "w" key. Then, attempt to change the current key by activating either the left_arm or right_arm extremity, and walk on the spot again. This should change the key being typed. Then, in a game which uses "WASD" for walking, perform the same steps as before, and verify that your game character changes direction when the current key is changed by activating a different extremity trigger. 15 5 Verify that walking on the spot without activating any of the triggers in the left pad doesn't cause any action to happen. In a text document, walk on the spot and check that nothing is being typed or moved. 16 5 Verify that walking on the spot while activating one of the left pad triggers (walking triggers) causes a key to be held down In a text document, walk on the spot and check that the keybind of the activated trigger is being typed while walking occurs. In a game with "WASD" for walking, perform the same steps as before and check that the game character is walking in the direction of the activated trigger's keybind. 17 5 Verify that activating any of the button triggers causes a key to be held down, and that they are independent of walking on the spot In a text document, activate one of the button triggers and check that the keybind of the activated trigger is being typed. Then also walk on the spot while activating the trigger, and verify nothing happens. 18 5 Verify that walking on the spot + a left pad trigger (walking trigger) happening with a button trigger being activated causes both actions to simultaneously occur In a text document, activate one of the walking triggers while walking on the spot, and also one of the button triggers, and verify that both keybinds of the activated triggers are being typed together.Then in a game with "WASD" support, and the ability to map commands to the keybinds of the extremity triggers, repeat the method and check if the game character can walk in the direction of the activated walking trigger keybind, and while also performing the command mapped to the activated button trigger (e.g. shooting) 19 6 Create a folder on the desktop with an empty Word file inside. Hold the left click and try to move the folder somewhere else on the desktop (drag and drop). 20 6 Perform a right click on the folder and then click "Open" in order to open the test folder. 21 6 Open the Word file inside the test folder with double click. 22 7 Use the same folder created for tests 19/20/21. With the Word file opened, try to scroll up and down. 23 7 Similarly to test 22, try to zoom in and out the Word file. 24 7 Using nose direction mode, go to the top right of the screen and press "X" to close the file. 25 8 Similarly to the previous test, using the hands tracking, close the test folder by pressing "X" on the top right corner. 26 9 Attempt to switch between the extremity trigger mode and the exercise mode using the "rock-n-roll" hand position with both hands.
We asked 4 people to test the system and recorderd their feedback
-
Testers
Tester ID Tester Name Tester details (age, occupation) 1 Azamat 24, structural engineer 2 Märten 21, mechanical engineering student 3 Leonardo 22, electrical engineering student 4 Emanuel 20, mechanical Engineering student
-
Test results overview
Test ID Mode ID Test description Success rate out of 4 Average nr of attempts needed Average usability/comfort rating 1 1 Open the test folder using the fist double click 4 1.75 3.75 2 1 Open the txt file in the folder using the index and middle finger double click 4 1 5 3 1 Attempt to scroll through the txt file 4 1 3.25 4 1 Attempt to zoom in and out of the txt file 4 1.5 5 5 1 Go back to the folder and make a copy of the document (right click -> copy and then right click -> paste) 4 1.75 4.25 6 1 drag the document out of the folder and onto the desktop. 4 1.25 4.75 7 2 Open a writable text document (on notepad or word), and attempt to activate any of the extremity triggers (except left_kick and right_kick), and check that each extremity trigger causes its circle to turn green, and its corresponding keybind to be pressed 4 1 5 8 2 On a text document again, attempt to trigger and hold the up extremity, and verify that for the duration of its activation, "w" is typed to screen 4 1 4.25 9 2 Manually move the mouse cursor to the "close" button of any window, and then activate the left_kick extremity to attempt to close the window by causing a left click action. Then, moving the cursor to an empty part of the desktop, attempt to activate the right_kick extremity to attempt to open up the context menu by causing a right click action 4 1 5 10 3 On a text document, attempt to activate an exercise, and check that the keybind assigned to the exercise appears on screen 4 2.5 3.5 11 3 On a text document again, either perform the walking or cycling exercise, and check that multiple keys are typed as you perform the exercise 4 1.25 4 12 3 Firstly, on a text document, check that there is a short duration after you stop walking on the spot or cycling, that a few extra keys are typed. Then, on any game that allows holding the "w" key for walking, attempt to walk on the spot at a non-exaggerated pace, and check that the game character seems to be continuously walking without any weird break when you switch legs 4 1.25 4.75 13 3 With the equipment mode selected, attempt to walk on the spot or squat, and verify nothing happens. If feasible, also attempt to perform cycling while no equipment mode is selected. 4 1.75 3.75 14 4 In a text document, walk on the spot to cause a key to be typed. If no extremity trigger has been activated before this point, this should be the "w" key. Then, attempt to change the current key by activating either the left_arm or right_arm extremity, and walk on the spot again. This should change the key being typed. Then, in a game which uses "WASD" for walking, perform the same steps as before, and verify that your game character changes direction when the current key is changed by activating a different extremity trigger. 4 1 4.75 15 5 In a text document, walk on the spot and check that nothing is being typed or moved. 4 1 4.75 16 5 In a text document, walk on the spot and check that the keybind of the activated trigger is being typed while walking occurs. In a game with "WASD" for walking, perform the same steps as before and check that the game character is walking in the direction of the activated trigger's keybind. 4 1.5 3.25 17 5 In a text document, activate one of the button triggers and check that the keybind of the activated trigger is being typed. Then also walk on the spot while activating the trigger, and verify nothing happens. 4 1.25 3.5 18 5 In a text document, activate one of the walking triggers while walking on the spot, and also one of the button triggers, and verify that both keybinds of the activated triggers are being typed together.Then in a game with "WASD" support, and the ability to map commands to the keybinds of the extremity triggers, repeat the method and check if the game character can walk in the direction of the activated walking trigger keybind, and while also performing the command mapped to the activated button trigger (e.g. shooting) 4 1.25 3.5 19 6 Hold the left click and try to move the folder somewhere else on the desktop (drag and drop). 4 3.5 2.25 20 6 Perform a right click on the folder and then click "Open" in order to open the test folder. 3 2.333333333 3 21 6 Open the Word file inside the test folder with double click. 3 1.333333333 4.333333333 22 7 With the Word file opened, try to scroll up and down. 4 1.5 2.5 23 7 Similarly to test 22, try to zoom in and out the Word file. 0 24 7 Using nose direction mode, go to the top right of the screen and press "X" to close the file. 4 2.5 3 25 8 Similarly to the previous test, using the hands tracking, close the test folder by pressing "X" on the top right corner. 3 1 4.333333333 26 9 Attempt to switch between the extremity trigger mode and the exercise mode using the "rock-n-roll" hand position with both hands. 4 1 5 -
Individual test feedback
Tester ID Test ID Fail/Success Usability/Comfort rating 1-5 Feedback 1 1 Success (2nd try) 3 Hard to hold the hand still, requires calibration - the pinch is falsely activated 1 2 Success (1st try) 5 Better than the fist 1 3 Success (1st try) 4 Gets easier once you get a hang of it is intuitive. at first bit weird 1 4 Success (1st try) 5 The gesture is intuitive but sometimes the text gets selected 1 5 Success (1st try) 5 1 6 Success (1st try) 5 1 7 Success (1st try) 5 1 8 Success (1st try) 4 Wearing a colour that blends with the background causes more flickering than usual 1 9 Success (1st try) 5 1 10 Success (2nd try) 4 squatting sometimes takes a lot of effort to trigger (aka squatting very deep) 1 11 Success (1st try) 4 The tester noted that when he is at the distance that is far enough for the walking to registr, it is hard to see the counters of actions on the screen 1 12 Success (1st try) 5 Consistent holding between walking motions 1 13 Success (2nd try) 4 No exercise bike to test with, so only tried squatting and walking in equipment mode. On occasion it would detect a cycling exercise because walking has similar movement 1 14 Success (1st try) 4 Extremity trigger circles are somewhat bunched up, so there was a point where one was accidentally triggered because of arm placement 1 15 Success (1st try) 5 1 16 Success (1st try) 3 while walking it is somethimes hard to keep the arms in one place and thus consistently on the trigger - feels awkward 1 17 Success (1st try) 4 Sometimes wrong triggers are unintentionally hit as they are in the way 1 18 Success (1st try) 3 Same issue as Test 16, hard to keep arms in place while walking 1 19 Success (3rd try) 3 shakier than the hand module, overall fine 1 20 Success (3rd try) 2 The nose cursor moves when the mouth is opened- thus hard to hit the target. sometimes the mouth opening is not registered (when the head is tilted forward) 1 21 Success (2nd try) 4 eyebrow lift easy to use but still shaky 1 22 Success (1st try) 2 The scroll itself is a 5 but misfires a lot 1 23 Fail Does not recognize fish face 1 24 Success (1st try) 3 moth open was not registered immediately - needed to open mouth until the jaw clicked (end this man has a big mouth). the previous nagigation mode was better 1 25 Success (1st try) 4 when the gestures are split between different body part, there is less likely to misfire! - As a suggestion, to avoid accidental clicks - move the mouse with one hand and clicking with the other! 1 26 Success (1st try) 5 2 1 Success(1st try) 4 2 2 Success(1st try) 5 2 3 Success(1st try) 1 Due to low mobility of the fingers the gesture is almost unusable (cant flod the ring finger while the middle is straight) 2 4 Success (3rd try) 5 Good gesture but only after you get a hang of it 2 5 Success (4rd try) 3 The gesture itself was comfortable but there were a lot of misfires 2 6 Success (2nd try) 5 The gesture itself was comfortable but there were a lot of misfires 2 7 Success (1st try) 5 Triggers all worked perfectly 2 8 Success (1st try) 4 Head trigger worked best 2 9 Success (1st try) 5 Same as key pressing, works perfectly 2 10 Success (3rd try) 3 Squatting somewhat annoying to detect. Walking is fine but it tends to detect walking when I don't mean it to e.g. turning around 2 11 Success (2nd try) 3 Harder to hold when walking on the spot fast, you need to cheat by holding the leg up longer 2 12 Success (1st try) 5 Clear delay 2 13 Success (3rd try) 3 Sometimes thinks walking is cycling 2 14 Success (1st try) 5 Just like normal triggers 2 15 Success (1st try) 5 No action 2 16 Success (2nd try) 3 Takes some time to get used to keeping arm level 2 17 Success (1st try) 3 Trigger positions can cause misinputs 2 18 Success (2nd try) 4 Again, takes some time to get used to gamepad mode 2 19 Success (4rd try) 1 2 20 Fail smile, and open mouth not distinguishable - lots of misterfires 2 21 Fail the simle open is always triggered and thus the menu bar is always in the way 2 22 Success (2nd try) 3 the scroll itself works fine bit a lot of misfired clicks 2 23 Fail fish face not recognized 2 24 Success (4rd try) 2 moth open was not registered immediately - needed to open mouth until the jaw clicked. the previous nagigation mode was better 2 25 Fail mouth click not registered. 2 26 Success (1st try) 5 3 1 Success (3nd try) 3 The cursor moves as the fist is clenched so the doubleclick is sometimes performed off target 3 2 Success (1st try) 5 3 3 Success (1st try) 4 Tester wishes there was more control of the scroll speed 3 4 Success (1st try) 5 3 5 Success (1st try) 4 Found that the hand gets a bit tired when attempting to do percise motions in the air for long 3 6 Success (1st try) 4 On rare ocations the mouseclick was falsly released 3 7 Success(1st try) 5 3 8 Success(1st try) 4 rare occasioning of flickering of the activated circle - appears that momentarily the detection stops but generally good 3 9 Success(1st try) 5 3 10 Success(2nd try) 4 Tester was asked to perform a squatting action first time around, but it detected walking and then squatting. 2nd attempt it detected squatting on its own fine. 3 11 Success(1st try) 4 Found that lifting on of the legs still holds the walking pose 3 12 Success(1st try) 5 Tester suggested it may feel laggy to have too long of a hold after the exercise is finished 3 13 Success(1st try) 3 When the Tester squatted in equipment mode, no exercise was detected. However, when the Tester tried to walk on the spot, cycling was detected. This was down to the cycling pose being similar to the walking pose, which is why the equipment and no equipment modes are separate to begin with 3 14 Success(1st try) 5 3 15 Success(1st try) 4 Positioning of the triggers by default and the normal walking motion of putting your hands up sometimes caused a trigger to inadvertantly activate 3 16 Success(2nd try) 3 Tester commented that it can be inconvenient to hold a trigger for a long time while walking, as the arms change position when walking, and that can cause it to go outside of the trigger area 3 17 Success(1st try) 4 Close positioning of the buttons by default can cause inadvertant activations, particularly when trying to move towards and away from the webcam 3 18 Success(1st try) 4 3 19 Success (4rd try) 2 Found the mouse uncomfortable to drag, as either it required a lot of torso movement in order to keep the head facing in the same direction or if the nose position was moved by tilting the neck then holding the mouth open was not recognised when head was tilted 3 20 Success (3rd try) 2 3 21 Success (1nd try) 4 3 22 Success (1st try) 2 A lot of unintentional events performed 3 23 Fail Zoomed in once by accident but could not be replicated 3 24 Success (3st try) 4 Better than the nose box navigation as openig the mouth does not unintentionally move the mouse 3 25 Success (1st try) 5 3 26 Success (1st try) 5 4 1 Success(1nd try) 5 4 2 Success(1st try) 5 4 3 Success(1st try) 4 Scrolling moving a bit to fast 4 4 Success (1st try) 5 4 5 Success(1st try) 5 4 6 Success(1st try) 5 4 7 Success(1st try) 5 Works well 4 8 Success(1st try) 5 Works well, same as pressing 4 9 Success(1st try) 5 Same as last 2 tests 4 10 Success(3rd try) 3 Prone to mixing up squatting and walking on the spot (took 3 goes initially to get squatting detected), but took getting used to 4 11 Success(1st try) 5 Key is clearly held 4 12 Success(2nd try) 4 Allows for enough time to switch legs to keep holding 4 13 Success(1st try) 5 Worked well with walking while in cycling mode 4 14 Success(1st try) 5 Works well like the other trigger tests 4 15 Success(1st try) 5 Again, simply a trigger test 4 16 Success(1st try) 4 Very interesting function, works well for what it is 4 17 Success(2nd try) 3 Buttons are very close in this mode, so reaching arm out triggered more than the intended one sometimes 4 18 Success(1st try) 3 Takes some time to get used to 4 19 Success(3nd try) 3 Open mouth was not recognise at first 4 20 Success(1st try) 5 4 21 Success(1st try) 5 4 22 Success(2nd try) 3 Menu opening while scrolling 4 23 Fail Only zoom in worked, for zoom out fish face not recognise 4 24 Success(2nd try) 3 Mouse is a bit to sensitive, missed the button 4 25 Success(1st try) 4 4 26 Success (1st try) 5
Conclusions
Generally from our User-Acceptance Testing, we can see that the Hands are the best received modules. The Hands module
is best suited for high-accuracy general control of the computer. The gestures are accurate and easy to use.
The body module, with the extremity triggers in particular, is well
implemented and accurate for the most part. The gamepad mode in particular has good potential for allowing the user to be active while they
play games with this mode. The exercise detection is generally good, particularly walking on the spot. However, its detection accuracy, e.g.
distinguishing between walking and squatting has room for improvement.
The Head module is a bit trickier to use, with the detector of some gestures, like ones for the mouth, not
being very accurate. The eyebrow gestures on the other hand are fine. The modes to control the cursor with the nose
are a bit unorthodox, due to the unusual nature of moving the nose.