Testing

In order to evaluate the quality of MotionInput V3's functionality, we conducted both performance and user-acceptance testing.

Testing Strategy

In order to make sure that all the functionalities implemented are properly working, each of the team members tested their module as it was developed. During development the system was also tested by other teams as well as a wide range of other potential users through micro builds - this gave us extensive feedback throughout the development process, Moreover, we run performance testing to prove the performance achieved in terms of time and memory efficiency. To test the user acceptance, tests were designed for each module, also for multiple modes combined. For the 27 tests, we got reviews from 4 testers.

Performance Testing

In order to demonstrate the running time of the code in different situations, we used the cProfile library in Python. As well as this, to find the memory usage over time, we utilised the Memory Profiler library. For each measurement, besides time, there are also mentioned the number of function calls (request made to perform a predetermined function) and primitive calls (the calls were not made through recursion).

The machine used for performance testing had the following specifications: Intel Core i7-9750H CPU @ 2.60Ghz, 2592 Mhz, 6 Core(s), 16 GB RAM, Graphics card NVIDIA GeForce GTX 1650.

Initialization

All the initializations for MotionInput v3 are completed in MotionInputAPI.start(). Using the cProfile library, the following time was achieved: 1.462 seconds

Frame rate for each module

In order to find the frame rate for each module on its own, the running time was recorded for 100 frames. The results are shown in the table below
.

Module(mode)Running time (for 100 frames)FPSComments
Hand (basic_hand mode)100909 function calls (99407 primitive calls) in 3.465 seconds28.86This time is registered when the user is running the mode and moves the mouse constantly.
Hand (basic_hand mode)19852 function calls (18650 primitive calls) in 1.531 seconds65.31This time is registered when the user is running the mode, but he is inactive. Because the Hand module has the idle mode, this time is significantly lower (more than 2 times less).
Body (extremity_triggers)19848 function calls in 3.819 seconds26.18This time is registered when the user is running the mode and triggers events constantly.
Body (extremity_triggers)18953 function calls in 3.769 seconds26.53This time is registered when the user is running the mode without performing any action.
Head (head_nose_position)19555 function calls in 4.208 seconds23.76This time is registered when the user is running the mode and moves the mouse constantly. Since for the Head module there is no idle state, the time would not differ because the program always tracks the user's face movement.
Head (head_nose_position)16686 function calls in 4.093 seconds24.43This time is registered when the user is running the mode, but no face is detected.
Eye (eye_tracking)46411 function calls (42811 primitive calls) in 7.581 seconds13.19This time is registered when the user is running the mode and moves the mouse constantly.
Eye (eye_tracking)14696 function calls (14496 primitive calls) in 3.422 seconds29.22This time is registered when the user is running the mode, but no face is detected.

The modes used are composed of the following sets of events:
basic_hand mode: Zoom, Open keyboard, Scroll, Idle, AOI resize, Double click, Monitor change, Mouse movement, Left press, Right press
extremity_triggers: Up, Arm right, Arm left, Kick left, Kick right, Punch left, Punch right
head_nose_position: Raise eye brow, Nose tracking, Smiling, Fish face, Open mouth
eye_tracking: Eye tracking

Here we can see that the hand module has the highest performance, and the eyes module by far the worst, due to the complexity of gaze inference. The Head module, due to its large number of landmarks (468), follows that it is slower than the Hand and Body modules that also use MediaPipe solutions.

Both hands and eye module have an idle state where the efficiency more than doubles, while the Head and Body modules are not impacted as much by the presence of the user in the frame.

Frame rate for each module in combination with hand module

At the moment, for switching between modes we predominantly use the "rocknroll_next_mode" event. This event uses the hand module, which means that if the mode in use is from a different module, the hand module will need to be proccessed in each frame in addition to the main functionality module. Hence we ran testing on each module, using it together with the "rocknroll_next_mode" event and compared the FPS to that, achieved in the previous table.

Module(mode)FPS without the Hand Modules eventFPS with Hand Modules eventComments
Hand (basic_hand mode)28.8628.79This time is registered when the user is running the mode and moves the mouse constantly.
Hand (basic_hand mode)65.3165.18This time is registered when the user is running the mode, but he is inactive. Because the Hand module has the idle mode, this time is significantly lower (more than 2 times less).
Body (extremity_triggers)26.1824.77This time is registered when the user is running the mode and triggers events constantly.
Body (extremity_triggers)26.5325.74This time is registered when the user is running the mode without performing any action.
Head (head_nose_position)23.7623.52This time is registered when the user is running the mode and moves the mouse constantly. Since for the Head module there is no idle state, the time would not differ because the program always tracks the user's face movement.
Head (head_nose_position)24.4324.24This time is registered when the user is running the mode, but no face is detected.
Eye (eye_tracking)13.1913.9This time is registered when the user is running the mode and moves the mouse constantly.
Eye (eye_tracking)29.2228.97This time is registered when the user is running the mode, but no face is detected.

As shown, adding the hand modules event does not significantly damage the frame rate. As we were initially running the processing of multiple modules in sequence, the performance was closer to the sum of the time each of the modules took separately. However, as we implemented multithreading we were able to achieve the results above.

Frame rate for combined modes

Similarly, we tested the performance when combining the hand, head and eyes modules. As the eyes module currently only allows for mouse navigation, it is in practice commonly used in combination with the head module and sometimes also hand module.

Module(mode)Running time (for 100 frames)FPS
Hand module (basic_hand mode):100909 function calls (99407 primitive calls) in 3.473 seconds28.79
Head module (head_nose_position)19555 function calls in 4.250 seconds23.53
Eye module (eye_tracking)46411 function calls (42811 primitive calls) in 7.581 seconds13.19
Head and Eye59171 function calls (55607 primitive calls) in 8.715 seconds11.48
Hand, Head and Eye modules combined:55064 function calls (51464 primitive calls) in 9.834 seconds10.17

Once again we can see that combining modules dues not result in the summation of their separate running times.

Performance over a longer period

For these tests, the Hand module is used (basic_hand mode). The time is measured when running the system for 50, 100, 250, 500, 750, 1000, 1500 and 2000 frames respectively.

Nr of framesRunning time using the basic_hand modeFPS using the basic_hand modeRunning time in idle_hand modeFPS in idle_hand mode
5050820 function calls (50066 primitive calls) in 1.891 seconds26.4410402 function calls (9800 primitive calls) in 0.898 seconds55.67
100100909 function calls (99407 primitive calls) in 3.473 seconds28.7919852 function calls (18650 primitive calls) in 1.534 seconds65.18
250254097 function calls (250323 primitive calls) in 8.258 seconds30.2748202 function calls (45200 primitive calls) in 3.320 seconds75.3
500377008 function calls (365621 primitive calls) in 13.949 seconds35.8495452 function calls (89450 primitive calls) in 6.782 seconds73.72
750763303 function calls (751989 primitive calls) in 24.375 seconds30.76142702 function calls (133700 primitive calls) in 9.858 seconds76.08
1000808066 function calls (786847 primitive calls) in 29.893 seconds33.45189952 function calls (177950 primitive calls) in 13.404 seconds74.6
15001479151 function calls (1457269 primitive calls) in 50.001 seconds29.99275002 function calls (257600 primitive calls) in 18.952 seconds79.14
20001984172 function calls (1954764 primitive calls) in 67.519 seconds29.62369502 function calls (346100 primitive calls) in 26.835 seconds74.52

As we can see the frame rate remains relatively constant as the system runs, thus the MotionInput v3 runtime is not affected by the length of the running period. This suggests that the system remains stable, and will allow for long durations of up-time.

Memory usage

We monitored the memory usage of MotionInput while the user constantly switched between multiple modes (basic_hand, head_nose_position, eye_grid_facial). As we can see there is an initial increase of memory reflecting the modules being first initialized by the system (most of them are maintained in memory after being used once as the initialization is time-consuming and it is assumed to be used again during the given runtime). Afterwards, however, as the modes are constantly iterated over there is no increase in memory usage. This again shows that MotionInput is sustainable for multiple modes of use without having to restart the application.

User Acceptance Testing

As the GUI was not part of our project testing was done on a compile that immediately started the camera view and the MI backend, the functionality was adjusted through the config files.

For testing we used 9 different modes listed below:

    • Mode config
      { "default": "basic_hand", "iteration_order": { "idle_hand": "idle_hand", "basic_hand": "basic_hand" }, "modes": { "basic_hand": [ "rocknroll_next_mode", "palm_height_change_aoi_resize_left_hand", "double_index_pinch_zoom", "palm_center_move_mouse_left_hand", "hand_to_idle_mode_left_hand", "middle_pinch_right_press_left_hand", "Fist_doubleclick_left_hand", "double_pinch_double_click_left_hand", "index_pinch_left_press_left_hand", "index_middle_scroll_left_hand" ], "idle_hand": [ "hand_to_active_mode_left_hand" ] } }

      Instructions on how to use the mode
      NB! The basic hand mode is currently configured for left-handedness.
      To move the mouse, move your left hand while facing the camera - the mouse moves reflect the position of your palm in respect to the blue rectangle on the screen.
      To left-click pinch only your index finger on only the left hand
      To right-click pinch only your middle finger
      To double click clench your fist or pinch index and middle fingers at once
      To zoom pinch the index finger on both your hands and move the hands closer together or further apart
      To scroll fold your pinky and ring finger, stretch the thumb, hold the middle and index finger together and move them up and down.
    • Mode config
      { "default": "extremity_triggers", "iteration_order": { "extremity_triggers": "extremity_triggers" }, "modes": { "extremity_triggers": [ "rocknroll_next_mode", "punch_left_extremity", "arm_left_extremity", "kick_left_extremity", "up_extremity", "kick_right_extremity", "arm_right_extremity", "punch_right_extremity" ] } }

      Instructions on how to use the mode
      This mode has 7 extremity triggers activated, which are displayed as circles on the view. Each trigger has a part of the body that activates it (nose, left or right wrist, and left or right ankle), an action (key press, key down, left or right click), and a keybind if required.

      The extremity triggers in left to right order are:
      punch_left (left wrist, key press, “h”)
      arm_left (left wrist, key press, “a”)
      kick_left (left ankle, left click)
      up (nose, key down, “w”)
      kick_right (right ankle, right click)
      arm_right (right arm, key press, “d”)
      punch_right (right arm, key press, “s”)
      To activate a trigger and its action, move the corresponding landmark/body part to the circle, which will turn green if activated, and will increment the count once deactivated.
    • Mode config
      { "default": "exercise_no_equipment", "iteration_order": { "exercise_equipment": "exercise_equipment", "exercise_no_equipment": "exercise_no_equipment" }, "modes": { "exercise_no_equipment": [ "squating_exercise", "walking_exercise" ], "exercise_equipment": [ "rocknroll_next_mode", "cycling_exercise" ] } }

      Instructions on how to use the mode
      The exercise modes consist of a no equipment mode, and an equipment mode. Each exercise in the modes have an action (key press, key down, left or right click), and a keybind if required.
      In the exercise_no_equipment mode, the activated exercises are:
      walking (key down, “w”)
      squatting (key press, “s”)

      In the exercise_equipment mode, there is one activated exercise:
      cycling (key down, “w”)

      Performing the motion of an exercise (e.g. walking on the spot) will activate the exercise, causing its action to be performed. For walking and cycling, there is an interval (0.5 seconds by default) after an exercise stops being detected where the key (“w” in both cases) will continue to be held down. This delay is intended to help simulate constant holding down in between movements (e.g. switching between legs in walking).
    • Mode config
      { "default": "extremity_walking", "iteration_order": { "extremity_walking": "extremity_walking" }, "modes": { "extremity_walking": [ "extremity_walking_event" ] } }

      Instructions on how to use the mode
      This mode allows for both walking on the spot and some extremity triggers to function with each other. There are 3 extremity triggers activated in this mode, each with a keybind:
      arm_left (left wrist, “a”)
      up (nose, “w”)
      arm_right (right wrist, “d”)
      This mode allows for a current key to be set, which is done by activating one of the extremity triggers, which changes the current key to its keybind. Walking on the spot will cause the current key to be held down.
      For walking, there is an interval (0.5 seconds by default) after walking is stopped being detected, where the current key will continue to be held down. This delay is intended to help simulate constant holding down in between walking movements (e.g. switching between legs).
    • Mode config
      { "default": "gaming_extremity_walking", "iteration_order": { "gaming_extremity_walking": "gaming_extremity_walking" }, "modes": { "gaming_extremity_walking": [ "gaming_extremity_walking_event" ] } }

      Instructions on how to use the mode
      This mode is a special gamepad mode, which allows for walking on the spot to be used in conjunction with extremity triggers.
      In the default configuration, you will see two pads of 4 triggers on the screen.

      The right pad is the button pad. Each of these triggers has a keybind, and activation causes the key to be held down. The extremity triggers for this pad are:
      left_button (right wrist, “j”)
      right_button (right wrist, “b”)
      up_button (right wrist, “k”)
      down_button (right wrist, “l”)

      The left pad is the walking pad, and these triggers are used in conjunction with walking on the spot. When an extremity trigger from this pad is triggered while walking is detected, then its keybind is held down. Walking without having any extremity trigger from this pad will not cause an action, and vice versa. The extremity triggers for this pad are:
      left_walking (left wrist, “a”)
      right_walking (left wrist, “d”)
      up_walking (left wrist, “w”)
      down_walking (left wrist, “s”)

      Additionally, there are two triggers at the top of the screen, which are button extremity triggers and are meant to act as start and quit buttons. From left to right, they are:
      start_key (left wrist, “space”)
      quit_key (right wrist, “esc”)
      For walking, there is an interval (0.5 seconds by default) after walking is stopped being detected, where whatever key of any extremity trigger on the left pad that is activated will continue to be held down. This delay is intended to help simulate constant holding down in between walking movements (e.g. switching between legs).
    • Mode config
      { "default": "head_nose_position", "iteration_order": { "head_nose_position": "head_nose_position" }, "modes": { "head_nose_position": [ "smiling_event", "open_mouth_event", "raise_eye_brow_event", "nose_tracking_event" ] } }

      Instructions on how to use the mode
      This mode is the first of the two head modes, using nose position.
      To move the mouse, move your left hand while facing the camera - the mouse moves reflect the position of your palm in respect to the blue rectangle on the screen.
      To left-click, open your mouth.
      To right-click, smile.
      To double click, raise your eyebrows.
    • Mode config
      { "default": "head_nose_direction", "iteration_order": { "head_nose_direction": "head_nose_direction" }, "modes": { "head_nose_direction": [ "smiling_event", "open_mouth_event", "raise_eye_brow_event", "nose_scroll_event", "nose_zoom_event", "nose_direction_tracking_event_nose_box" ] } }

      Instructions on how to use the mode
      This mode is also made for the head module, using nose direction.
      To move the mouse, move your nose while facing the camera - the mouse moves in the same direction as your nose tip in respect to the small blue rectangle on the centre of the screen.
      To left-click, right-click and double click, use the exact same gestures as in Mode 6.
      To scroll, smile while moving your nose tip up(to scroll up) and down(to scroll down).
      To zoom, perform a fish face gesture while moving your nose tip to the left(to zoom in) and to the right(to zoom out).
    • Mode config
      { "default": "hand_mouth", "iteration_order": { "hand_mouth": "hand_mouth" }, "modes": { "eye": [ "palm_height_change_aoi_resize_left_hand", "palm_center_move_mouse_left_hand", "smiling_event", "open_mouth_event", "raise_eye_brow_event" ] } }

      Instructions on how to use the mode
      This mode allows you to move the cursor using your eyes. Also, it is combined with head gestures for clicking events.
      To move the mouse, move your eyes while facing the camera - the mouse moves to the position on the screen you are looking at.
      To left-click, open your mouth.
      To right-click, smile.
      To double click, raise your eyebrows.
    • Mode config
      { "default": "extremity_triggers", "iteration_order": { "extremity_triggers": "exercise_no_equipment", "exercise_no_equipment": "extremity_triggers" }, "modes": { "extremity_triggers": [ "rocknroll_next_mode", "punch_right_extremity", "up_extremity", "arm_right_extremity", "arm_left_extremity", "punch_left_extremity", "kick_left_extremity", "kick_right_extremity" ], "exercise_no_equipment": [ "rocknroll_next_mode", "squating_exercise", "walking_exercise" ] } }

      Instructions on how to use the mode
      This mode combines the extremity trigger mode (mode 2) and the no equipment exercise mode (mode 3).
      Since there are two modes in the iteration, they can be switched between by making the “rock-n-roll” gesture with both hands.

And then with each of the modes created one or multiple tests with the following instructions

  • Test IDMode IDTest preparationTest description
    11Create a test folder on the desktop with one .txt file that contains a very long text.Open the test folder using the fist double click
    21 Open the txt file in the folder using the index and middle finger double click
    31Verify that the handlers/scroll/speed in config.json is appropriateAttempt to scroll through the txt file
    41Verify that the handlers/zoom/speen in config.json is appropriate (mine was 20)Attempt to zoom in and out of the txt file
    51 Go back to the folder and make a copy of the document (right click -> copy and then right click -> paste)
    61 drag the document out of the folder and onto the desktop.
    72Verify that the extremity triggers with keybinds set actually causes the key to be pressed or held downOpen a writable text document (on notepad or word), and attempt to activate any of the extremity triggers (except left_kick and right_kick), and check that each extremity trigger causes its circle to turn green, and its corresponding keybind to be pressed
    82Verify that an extremity trigger with an action set to "key_down" indeed holds down the keyOn a text document again, attempt to trigger and hold the up extremity, and verify that for the duration of its activation, "w" is typed to screen
    92Verify that an extremity trigger with a clicking action set ("left_click" or "right_click") indeed causes a mouse clickManually move the mouse cursor to the "close" button of any window, and then activate the left_kick extremity to attempt to close the window by causing a left click action. Then, moving the cursor to an empty part of the desktop, attempt to activate the right_kick extremity to attempt to open up the context menu by causing a right click action
    103Verify that the exercise with a keybind set actually causes the key to be pressed or held downOn a text document, attempt to activate an exercise, and check that the keybind assigned to the exercise appears on screen
    113Verify that an exercise with an action set to "key_down" indeed holds down the keyOn a text document again, either perform the walking or cycling exercise, and check that multiple keys are typed as you perform the exercise
    123Verify that an exercise with an action set to "key_down" has an interval after the exercise is deactivated where the key continues to be held down, and that this may facilitate a more fluid actionFirstly, on a text document, check that there is a short duration after you stop walking on the spot or cycling, that a few extra keys are typed. Then, on any game that allows holding the "w" key for walking, attempt to walk on the spot at a non-exaggerated pace, and check that the game character seems to be continuously walking without any weird break when you switch legs
    133Verify that if the equipment mode is selected, then attempting to perform a no equipment exercise will not cause it to be detected/an action to occur (and vice versa)With the equipment mode selected, attempt to walk on the spot or squat, and verify nothing happens. If feasible, also attempt to perform cycling while no equipment mode is selected.
    144Verify that activating one of the extremity triggers causes the current key to changeIn a text document, walk on the spot to cause a key to be typed. If no extremity trigger has been activated before this point, this should be the "w" key. Then, attempt to change the current key by activating either the left_arm or right_arm extremity, and walk on the spot again. This should change the key being typed. Then, in a game which uses "WASD" for walking, perform the same steps as before, and verify that your game character changes direction when the current key is changed by activating a different extremity trigger.
    155Verify that walking on the spot without activating any of the triggers in the left pad doesn't cause any action to happen.In a text document, walk on the spot and check that nothing is being typed or moved.
    165Verify that walking on the spot while activating one of the left pad triggers (walking triggers) causes a key to be held downIn a text document, walk on the spot and check that the keybind of the activated trigger is being typed while walking occurs. In a game with "WASD" for walking, perform the same steps as before and check that the game character is walking in the direction of the activated trigger's keybind.
    175Verify that activating any of the button triggers causes a key to be held down, and that they are independent of walking on the spotIn a text document, activate one of the button triggers and check that the keybind of the activated trigger is being typed. Then also walk on the spot while activating the trigger, and verify nothing happens.
    185Verify that walking on the spot + a left pad trigger (walking trigger) happening with a button trigger being activated causes both actions to simultaneously occurIn a text document, activate one of the walking triggers while walking on the spot, and also one of the button triggers, and verify that both keybinds of the activated triggers are being typed together.Then in a game with "WASD" support, and the ability to map commands to the keybinds of the extremity triggers, repeat the method and check if the game character can walk in the direction of the activated walking trigger keybind, and while also performing the command mapped to the activated button trigger (e.g. shooting)
    196Create a folder on the desktop with an empty Word file inside.Hold the left click and try to move the folder somewhere else on the desktop (drag and drop).
    206 Perform a right click on the folder and then click "Open" in order to open the test folder.
    216 Open the Word file inside the test folder with double click.
    227Use the same folder created for tests 19/20/21.With the Word file opened, try to scroll up and down.
    237 Similarly to test 22, try to zoom in and out the Word file.
    247 Using nose direction mode, go to the top right of the screen and press "X" to close the file.
    258 Similarly to the previous test, using the hands tracking, close the test folder by pressing "X" on the top right corner.
    269 Attempt to switch between the extremity trigger mode and the exercise mode using the "rock-n-roll" hand position with both hands.

We asked 4 people to test the system and recorderd their feedback

  • Tester IDTester NameTester details (age, occupation)
    1Azamat24, structural engineer
    2Märten21, mechanical engineering student
    3Leonardo22, electrical engineering student
    4Emanuel20, mechanical Engineering student
  • Test IDMode IDTest descriptionSuccess rate out of 4Average nr of attempts neededAverage usability/comfort rating
    11Open the test folder using the fist double click41.753.75
    21Open the txt file in the folder using the index and middle finger double click415
    31Attempt to scroll through the txt file413.25
    41Attempt to zoom in and out of the txt file41.55
    51Go back to the folder and make a copy of the document (right click -> copy and then right click -> paste)41.754.25
    61drag the document out of the folder and onto the desktop.41.254.75
    72Open a writable text document (on notepad or word), and attempt to activate any of the extremity triggers (except left_kick and right_kick), and check that each extremity trigger causes its circle to turn green, and its corresponding keybind to be pressed415
    82On a text document again, attempt to trigger and hold the up extremity, and verify that for the duration of its activation, "w" is typed to screen414.25
    92Manually move the mouse cursor to the "close" button of any window, and then activate the left_kick extremity to attempt to close the window by causing a left click action. Then, moving the cursor to an empty part of the desktop, attempt to activate the right_kick extremity to attempt to open up the context menu by causing a right click action415
    103On a text document, attempt to activate an exercise, and check that the keybind assigned to the exercise appears on screen42.53.5
    113On a text document again, either perform the walking or cycling exercise, and check that multiple keys are typed as you perform the exercise41.254
    123Firstly, on a text document, check that there is a short duration after you stop walking on the spot or cycling, that a few extra keys are typed. Then, on any game that allows holding the "w" key for walking, attempt to walk on the spot at a non-exaggerated pace, and check that the game character seems to be continuously walking without any weird break when you switch legs41.254.75
    133With the equipment mode selected, attempt to walk on the spot or squat, and verify nothing happens. If feasible, also attempt to perform cycling while no equipment mode is selected.41.753.75
    144In a text document, walk on the spot to cause a key to be typed. If no extremity trigger has been activated before this point, this should be the "w" key. Then, attempt to change the current key by activating either the left_arm or right_arm extremity, and walk on the spot again. This should change the key being typed. Then, in a game which uses "WASD" for walking, perform the same steps as before, and verify that your game character changes direction when the current key is changed by activating a different extremity trigger.414.75
    155In a text document, walk on the spot and check that nothing is being typed or moved.414.75
    165In a text document, walk on the spot and check that the keybind of the activated trigger is being typed while walking occurs. In a game with "WASD" for walking, perform the same steps as before and check that the game character is walking in the direction of the activated trigger's keybind.41.53.25
    175In a text document, activate one of the button triggers and check that the keybind of the activated trigger is being typed. Then also walk on the spot while activating the trigger, and verify nothing happens.41.253.5
    185In a text document, activate one of the walking triggers while walking on the spot, and also one of the button triggers, and verify that both keybinds of the activated triggers are being typed together.Then in a game with "WASD" support, and the ability to map commands to the keybinds of the extremity triggers, repeat the method and check if the game character can walk in the direction of the activated walking trigger keybind, and while also performing the command mapped to the activated button trigger (e.g. shooting)41.253.5
    196Hold the left click and try to move the folder somewhere else on the desktop (drag and drop).43.52.25
    206Perform a right click on the folder and then click "Open" in order to open the test folder.32.3333333333
    216Open the Word file inside the test folder with double click.31.3333333334.333333333
    227With the Word file opened, try to scroll up and down.41.52.5
    237Similarly to test 22, try to zoom in and out the Word file.0  
    247Using nose direction mode, go to the top right of the screen and press "X" to close the file.42.53
    258Similarly to the previous test, using the hands tracking, close the test folder by pressing "X" on the top right corner.314.333333333
    269Attempt to switch between the extremity trigger mode and the exercise mode using the "rock-n-roll" hand position with both hands.415
    • Tester IDTest IDFail/SuccessUsability/Comfort rating 1-5Feedback
      11Success (2nd try)3Hard to hold the hand still, requires calibration - the pinch is falsely activated
      12Success (1st try)5Better than the fist
      13Success (1st try)4Gets easier once you get a hang of it is intuitive. at first bit weird
      14Success (1st try)5The gesture is intuitive but sometimes the text gets selected
      15Success (1st try)5 
      16Success (1st try)5 
      17Success (1st try)5 
      18Success (1st try)4Wearing a colour that blends with the background causes more flickering than usual
      19Success (1st try)5 
      110Success (2nd try)4squatting sometimes takes a lot of effort to trigger (aka squatting very deep)
      111Success (1st try)4The tester noted that when he is at the distance that is far enough for the walking to registr, it is hard to see the counters of actions on the screen
      112Success (1st try)5Consistent holding between walking motions
      113Success (2nd try)4No exercise bike to test with, so only tried squatting and walking in equipment mode. On occasion it would detect a cycling exercise because walking has similar movement
      114Success (1st try)4Extremity trigger circles are somewhat bunched up, so there was a point where one was accidentally triggered because of arm placement
      115Success (1st try)5 
      116Success (1st try)3while walking it is somethimes hard to keep the arms in one place and thus consistently on the trigger - feels awkward
      117Success (1st try)4Sometimes wrong triggers are unintentionally hit as they are in the way
      118Success (1st try)3Same issue as Test 16, hard to keep arms in place while walking
      119Success (3rd try)3shakier than the hand module, overall fine
      120Success (3rd try)2The nose cursor moves when the mouth is opened- thus hard to hit the target. sometimes the mouth opening is not registered (when the head is tilted forward)
      121Success (2nd try)4eyebrow lift easy to use but still shaky
      122Success (1st try)2The scroll itself is a 5 but misfires a lot
      123Fail Does not recognize fish face
      124Success (1st try)3moth open was not registered immediately - needed to open mouth until the jaw clicked (end this man has a big mouth). the previous nagigation mode was better
      125Success (1st try)4when the gestures are split between different body part, there is less likely to misfire! - As a suggestion, to avoid accidental clicks - move the mouse with one hand and clicking with the other!
      126Success (1st try)5 
           
      21Success(1st try)4 
      22Success(1st try)5 
      23Success(1st try)1Due to low mobility of the fingers the gesture is almost unusable (cant flod the ring finger while the middle is straight)
      24Success (3rd try)5Good gesture but only after you get a hang of it
      25Success (4rd try)3The gesture itself was comfortable but there were a lot of misfires
      26Success (2nd try)5The gesture itself was comfortable but there were a lot of misfires
      27Success (1st try)5Triggers all worked perfectly
      28Success (1st try)4Head trigger worked best
      29Success (1st try)5Same as key pressing, works perfectly
      210Success (3rd try)3Squatting somewhat annoying to detect. Walking is fine but it tends to detect walking when I don't mean it to e.g. turning around
      211Success (2nd try)3Harder to hold when walking on the spot fast, you need to cheat by holding the leg up longer
      212Success (1st try)5Clear delay
      213Success (3rd try)3Sometimes thinks walking is cycling
      214Success (1st try)5Just like normal triggers
      215Success (1st try)5No action
      216Success (2nd try)3Takes some time to get used to keeping arm level
      217Success (1st try)3Trigger positions can cause misinputs
      218Success (2nd try)4Again, takes some time to get used to gamepad mode
      219Success (4rd try)1 
      220Fail smile, and open mouth not distinguishable - lots of misterfires
      221Fail the simle open is always triggered and thus the menu bar is always in the way
      222Success (2nd try)3the scroll itself works fine bit a lot of misfired clicks
      223Fail fish face not recognized
      224Success (4rd try)2moth open was not registered immediately - needed to open mouth until the jaw clicked. the previous nagigation mode was better
      225Fail mouth click not registered.
      226Success (1st try)5 
           
      31Success (3nd try)3The cursor moves as the fist is clenched so the doubleclick is sometimes performed off target
      32Success (1st try)5 
      33Success (1st try)4Tester wishes there was more control of the scroll speed
      34Success (1st try)5 
      35Success (1st try)4Found that the hand gets a bit tired when attempting to do percise motions in the air for long
      36Success (1st try)4On rare ocations the mouseclick was falsly released
      37Success(1st try)5 
      38Success(1st try)4rare occasioning of flickering of the activated circle - appears that momentarily the detection stops but generally good
      39Success(1st try)5 
      310Success(2nd try)4Tester was asked to perform a squatting action first time around, but it detected walking and then squatting. 2nd attempt it detected squatting on its own fine.
      311Success(1st try)4Found that lifting on of the legs still holds the walking pose
      312Success(1st try)5Tester suggested it may feel laggy to have too long of a hold after the exercise is finished
      313Success(1st try)3When the Tester squatted in equipment mode, no exercise was detected. However, when the Tester tried to walk on the spot, cycling was detected. This was down to the cycling pose being similar to the walking pose, which is why the equipment and no equipment modes are separate to begin with
      314Success(1st try)5 
      315Success(1st try)4Positioning of the triggers by default and the normal walking motion of putting your hands up sometimes caused a trigger to inadvertantly activate
      316Success(2nd try)3Tester commented that it can be inconvenient to hold a trigger for a long time while walking, as the arms change position when walking, and that can cause it to go outside of the trigger area
      317Success(1st try)4Close positioning of the buttons by default can cause inadvertant activations, particularly when trying to move towards and away from the webcam
      318Success(1st try)4 
      319Success (4rd try)2Found the mouse uncomfortable to drag, as either it required a lot of torso movement in order to keep the head facing in the same direction or if the nose position was moved by tilting the neck then holding the mouth open was not recognised when head was tilted
      320Success (3rd try)2 
      321Success (1nd try)4 
      322Success (1st try)2A lot of unintentional events performed
      323Fail Zoomed in once by accident but could not be replicated
      324Success (3st try)4Better than the nose box navigation as openig the mouth does not unintentionally move the mouse
      325Success (1st try)5 
      326Success (1st try)5 
           
      41Success(1nd try)5 
      42Success(1st try)5 
      43Success(1st try)4Scrolling moving a bit to fast
      44Success (1st try)5 
      45Success(1st try)5 
      46Success(1st try)5 
      47Success(1st try)5Works well
      48Success(1st try)5Works well, same as pressing
      49Success(1st try)5Same as last 2 tests
      410Success(3rd try)3Prone to mixing up squatting and walking on the spot (took 3 goes initially to get squatting detected), but took getting used to
      411Success(1st try)5Key is clearly held
      412Success(2nd try)4Allows for enough time to switch legs to keep holding
      413Success(1st try)5Worked well with walking while in cycling mode
      414Success(1st try)5Works well like the other trigger tests
      415Success(1st try)5Again, simply a trigger test
      416Success(1st try)4Very interesting function, works well for what it is
      417Success(2nd try)3Buttons are very close in this mode, so reaching arm out triggered more than the intended one sometimes
      418Success(1st try)3Takes some time to get used to
      419Success(3nd try)3Open mouth was not recognise at first
      420Success(1st try)5 
      421Success(1st try)5 
      422Success(2nd try)3Menu opening while scrolling
      423Fail Only zoom in worked, for zoom out fish face not recognise
      424Success(2nd try)3Mouse is a bit to sensitive, missed the button
      425Success(1st try)4 
      426Success (1st try)5



    Conclusions

    Generally from our User-Acceptance Testing, we can see that the Hands are the best received modules. The Hands module is best suited for high-accuracy general control of the computer. The gestures are accurate and easy to use.

    The body module, with the extremity triggers in particular, is well implemented and accurate for the most part. The gamepad mode in particular has good potential for allowing the user to be active while they play games with this mode. The exercise detection is generally good, particularly walking on the spot. However, its detection accuracy, e.g. distinguishing between walking and squatting has room for improvement.

    The Head module is a bit trickier to use, with the detector of some gestures, like ones for the mouth, not being very accurate. The eyebrow gestures on the other hand are fine. The modes to control the cursor with the nose are a bit unorthodox, due to the unusual nature of moving the nose.