MotionInput.

Project background

UCL (University College London) MotionInput is one of the world’s leading software packages for touchless computing, first launched in September 2020. From its inception it provided an intuitive way to control the mouse of a computer purely by the detection of hand movement and the activation of different actions through broad movements such as walking. These two functionalities are still at the core of the software but since then has expanded massively over the last few years.

Modules such as MediaPipe, Dlib, OpenVino and Vosk have been used extensively to grow the functionality of MotionInput and give rise to individual Hand, full Body, Eyes and Head modules that can work in tangent with each other. PsuedoVR is the name given to the combination of different MotionInput modules that are used in conjunction with each other including other functionalities such as on-screen buttons and voice commands.

This is currently the fourth year of MotionInput with Version 3.2, a more incremental upgrade of the software rather than a massive overhaul over previous years. However, the work done this year will justify a completely updated version of MotionInput for the next year given the extra functionalities developed by this team as well as others on various aspects. Our team’s stakeholders are Prof Dean Mohamedally, Dr Atia Rafiq, Prof Phillippa Chick (Intel), Prof Costas Stylianou (Intel), Lee Stott (Microsoft), John McNamara (IBM).

Project Ananlysis

Since our project solely involved using the MotionInput API, we needed to have a close look at the codebase to see what modules would be the most appropriate for us to use. In that process we found several obstacles that would need to be tackled to effectively implement our solutions

Obstacles found regarding gesture recording/detection:

• The current gesture recorder implemented in the current MotionInput repository is highly flawed. It was implemented on the basis that gestures would be recorded over five seconds which requires a large amount of data to be processed causing the feature to be extremely slow.
• Another issue is that the gesture detector (using the data from the gesture recorder) was designed to be extremely rigid so that for a user to perform the action their movements would have to be the same as the movements that were recorded. This causes successful detection percentage to be exceptionally low.
• As a result of the previous point, gesture detection when done correctly would only work for the user that recorded the gesture as the data points would be mapped specific to them and would not work for any other user due to differing body types.
• The current gesture recorder is full body only which requires a large amount of data but also limits the utility for users.

Obstacles found regarding other issues:

• Making modes (for specific games) is extremely difficult, new features must be added to many different modules in many separate places in both the code and in data files for functionality to work properly. This makes the system unextendible and will be difficult to design any front end to make new modes if that functionality is required.
• In 3D games where hand detection is used to move the cursor and hence the aim in the game, there is a limit to how far the hand can be moved which hence limits the range of aim in the game. It is unlikely for this method of changing the view in games will aid in making gameplay as seamless as possible.
• There is a severe lack of documentation for the entire MotionInput codebase and therefore makes it extremely difficult for us to extend it and utilise the modules that exist in the API.