UCL MotionInput 3 FAQ - www.touchlesscomputing.org
Current version 3.02
------------------

Can I see some videos on what exactly this is?
- our videos playlist is here: https://www.youtube.com/playlist?list=PLItOc9xhnCnidFTWh95oh2fTpbjz64Kub

What hardware do I need to run this?
- A Windows 10 based PC or laptop, with a webcam! Ideally an Intel-based PC, 7th Gen and above, and 4GB RAM. An SSD is highly recommended. The more CPU cores the merrier! Running parallel ML and CV is highly compute intensive, even with our extensive optimisations - so for example in Pseudo-VR mode for an FPS game, you may be doing hands recognition of the mouse, walking on the spot recognition, and hit targets with body landmarks, with speech, simultaneously while rendering advanced game visuals. At its simplest end, doing simple mouse clicks in a web browser, should be much less intensive.


What platforms does this run on?
- Windows 10 for the full software, and Windows 10 and 11 for the Microbuilds of the stand-alone features. We are also developing the software for Linux, Raspberry Pi, Android and Mac. If you would like to help to test this with us, please contact us: https://forms.office.com/r/P3a3dsH8Gx or by email on our touchlesscomputing.org website.

How do I run the software?
- Double click the installer for the microbuilds and it will be installed to your Windows Start Menu. 
Alternatively, download the zip file version and unzip to your PC, and run the MotionInput executable file. If you want to run the full software with the GUI front-end for settings, you will need Windows 10, and both Vigembus (for In-Air Joypad) and Dotnet 3.1 for Desktops (X64) installed. Vigembus currently (as of June 2022) only runs on Windows 10. Follow the instructions on the download link.


What are the hand gestures in the Multitouch microbuild application?
- Raising your right hand, you will be able to do the following:
Index Pinch (Index Finger and Thumb)- native Windows Touchpoints Press (should appear as a round translucent circle)
Middle Pinch (Middle Finger and Thumb) - Left Mouse Click (or just say "click")
Ring Pinch (Ring Finger and Thumb) - Double Left Mouse Click (or just say "double click")
Pinky Pinch (Pinky Little Finger and Thumb) - Right Mouse Click (or just say "right click")
Both right and left hands doing Index Pinch, moving in and out, starting with right hand - Touchpoints Drag (zooming in a browser)
You can also select and drag in-air by saying "hold left" (meaning, hold the left mouse button), and "release left" (to release the left mouse button).
The dominant hand is set to right by default. By default hand gestures are set to right handed but this can be switched.
In the full GUI build this is easily set as an option.
However, in the Multitouch microbuild, replace the data\mode_controller.json file with mode_controller_left_(rename).json (rename it to mode_controller.json and backup that file).
To change the dominant hand from right to left, open in data\events.json and rename the default mode from "touchpoint_right_hand" to "touchpoint_left_hand" . Save the files and restart the program.
We will make this easier in an upcoming build of the Multitouch microbuild. 


What is Ask-KITA and what speech functions are there?
-Ask-KITA is our speech engine, much like Alexa and Siri, and is built on open source VOSK engine. It is intended for teaching, office work and clinical/industrial specific speech recognition. KITA stands for Know-IT-All, and itself is a standalone product as well as embedded in our touchless computing applications. It is a combinatorial speech engine that will mix well with motion gesture technologies and gives three key levels of speech processing - commands (such as turning phrases into keyboard shortcuts and mouse events), localised and offline live caption dictation without user training, delivering recognised words into text-based programs, and gesture combined exploration of spoken phrases.

What interface commands can Ask-KITA currently do?
- The following speech commands as phrases are integrated already.
Mouse Events: click, double click, right click, hold left (for selecting a region), release left 
Windows-level events: minimize, maximize, maximize left, maximize right, files (file explorer), start menu, windows key, windows run, screenshot, volume up, volume down 
Keyboard shortcuts: up, down, left, right (these are the arrow keys), page up, page down, help, switch (alt+tab), save, print, cut, copy, paste, undo, space
PowerPoint specific keywords: show full screen, next, previous, escape 
A commands app is due this summer (2022) to let users customise words for any apps and games that they currently have.

How do I turn on live transcription in any text editor/browser/application/game?
- Say "Transcribe" and it should appear in the camera view as a message. 
Say "Stop Transcribe" to turn it off. It takes a little practice to speak in clear sentences for it to recognise what you are saying. Our intention was for teachers reading out books to young students, but it has become much more than that. Note that this live captioning and transcription feature is highly experimental, and it's accuracy will improve over time with new versions released. 

How can I add my own speech commands to my own favourite applications and games?
- Ask-KITA can do this but requires some text editing. Contact us for a guide, and note as said before - a commands app is due this summer (2022) to let users customise words for any apps and games that they currently have.

How can I change the language model in Ask-KITA?
- This requires a bit of technical know-how but it can be done with the VOSK voice models available online. A guide on this will be provided this summer (2022).

What are the facial gestures in the Facial Navigation microbuild application?
- This app will allow a user to navigate with either their nose (NoseNav) or with Eye Gaze. They can either say commands, or use combinations of facial movements to act as mouse presses or keyboard keys.
you can say "hold left" to drag in a direction and hold the left mouse button. Say "release left" to let go.
If you are in NoseNav mode, say "butterfly" when you move your head, to move the area of interest box to where your head is at that point.
The facial gestures in non-speech modes are:
 		Nose					Eyes
Smile		Left click and drag (v3.01+)		on/off navigation
Fish Face (kiss) left click				left click
Eyebrow raised	double click				double click
Mouth Open	right click				right click
 

Here are some of the additional speech commands specific to Facial Navigation 

Saying the "Next Mode" command will rotate through these options:
Nose Navigation (NoseNav) with speech commands
Nose Navigation (NoseNav) with Facial Gestures
Eye Tracking (EyeGaze) with speech commands
Eye Tracking (EyeGaze) with Facial Gestures
Whenever you enter eye gaze you will run through calibration, which has onscreen instructions.
In eye gaze modes, while watching in a direction, say "go" and cursor will start moving in that direction (magnetic drift mode). Say "stop" to stop moving in that direction.

What are the features of the in-air Inking microbuild application?
- The in-air inking application is an early technical Windows 10/11 demonstrator to show native windows inking in the air, with depth calculations for ink thickness. It works with apps like Krita, OneNote, Fresh Paint, Photoshop and others. Speech is now available as commands and transcribe. You can hold a pen in your fingertips, or you can draw without any pen. The left hand will be used for erasing.

Once unzipped or installed, run the application. A calibration window will open. Follow the on-screen instructions to calibrate depth values. Once calibration is done, the main program will automatically start. Open any Windows Inking application such as Paint 3D, OneNote or FreshPaint. Move your hand forwards or back to control depth (ink thickness).Activate the eraser by doing an index-thumb pinch with your non-dominant hand. Calibration runs by default but can be turned off by simply editing the JSON file in data/config.json. To do this, open the config.json file, find the run_calibration variable, and set it to false. Save the file and restart the program.


What are the features of the Retro Gamepad with Walking on the Spot application?
- This highly experimental microbuild application is a technical Windows 10/11 demonstrator that enables you to play retro games with usual ABXY gamepad buttons, by hitting in the air. It has 3 modes to play; mode 1 - tap d-pad to set walking direction, mode 2 - hold d-pad to walk, and mode 3 - walking is always forward (W key). In order to walk in the game, you have to walk on the spot to enable the directional D-pad buttons. Here are some examples: https://youtu.be/JBsDv12SgBE

Speech with Ask-KITA is included. Say Transcribe and Stop Transcribe to start and stop general text, otherwise it is in commands mode. Future game keyboard shortcuts will be worked on - please drop us a line on your favourite games!

Use at your own risk and make sure you have enough space and clearance when playing!


What are the features of the FPS with Walking on the Spot application?
- This highly experimental microbuild application is a technical Windows 10/11 demonstrator that enables you to play First Person and 3rd Person 3D Games while walking. In order to walk in the game, you have to walk on the spot to enable the directional D-pad buttons. The dominant hand controls the mouse cursor (normally the viewpoint in those games). Here is an example of playing MineCraft: https://www.youtube.com/watch?v=upnEGFUTYo0&

Use at your own risk and make sure you have enough space and clearance when playing!


What are the features of the In-Air Joypad application?
- This is a hands based Joypad mode (requires Vigembus installed). The in-air Joypad works with any compatible Windows PC game that uses a standard controller joypad.  
In-Air Joypad gestures and their assignments:

To press start - pinch together the right thumb and the right little pinky finger
Rotating left wrist - analogue left d-circle
Rotating right wrist - analogue right d-circle
Right hand fingers down - ABXY buttons
Left hand fingers down - Up-Down-Left-Right d-pad buttons
Right handed to left handed switching - edit the data\events.json file where it has args for right hand
Speech is enabled via Ask-KITA so if there is a chat window, enable with "transcribe" and then you can speak. "Stop transcribe" to stop.

Example here: https://www.youtube.com/watch?v=FLLviyhn808&

--------

Errors:

It didn't run, what can I do?
- This Technical Preview version is experimental and we, with zero-funds on this academic project, did not have a wide network for testing beyond our students and staff. We would love for community testers to come onboard so please contact us. Meantime, you can check the following:
If you have multiple cameras on your PC/laptop you can try to disconnect them. Some laptops with "hello" recognition built in have had errors. In the folder "data", you will see config.json. Try changing the value of camera_nr from 0 to 1, 2 or 3. We are working on detecting this currently.
If you have an antivirus it may say warnings (or stop the app completely) from accessing the camera and microphone. The software requires those things to run, so you may have to alter settings in your antivirus and restart your machine.
Some versions (especially with the joypad and the gaming components) will not run unless you install ViGemBus driver as that is needed for the in-air gamepad. We are working on fixes to detect this. Vigembus is available here: https://github.com/ViGEm/ViGEmBus/releases
Keep your Windows 10 and Windows 11 installations up to date, the oldest versions of Windows 10 have reported errors.
You are welcome to contact us with any other thoughts and requests on compatibility issues you have faced.

For the full GUI version, it asks what dotnet version to install, which do I choose?
- For this v3.0, it requires Microsoft Dotnet 3.1 (x64 Desktop specifically) - you will need admin rights to install that from Microsoft. For scenarios where you may not have such rights, try the microbuilds without the GUI.

Why does it say this application is unsafe when I first try to run it?
- As of June 2022 we are in the process of code signing the applications. Therefore please use at your own risk, this a technical preview and try it only on machines that are safe to do so.

--------

Non-technical points of interest:

This is very much in the realm of sci-fi! Can I do gestures like in <insert hollywood film here> ?
-Reach out to us and your suggestions, and lets see what we can do! Feedback form: https://forms.office.com/r/P3a3dsH8Gx or by email on our touchlesscomputing.org website.


What motivated you to build this?
-Covid-19 affected the world, and for a while before the vaccines, as well as the public getting sick, hospital staff were getting severely ill. To keep shared computers clean and germ free comes at a cost to various economies around the world. We saw a critical need to develop cheap/free software to help in healthcare, improve the way in which we work and so we examined many different methods for touchless computing. Along the journey, several major tech firms had made significant jumps in Machine Learning (ML) and Computer Vision, and our UCL IXN programme was well suited to getting them working together with students and academics. Some of the tech firms also had let go of past products which would have been useful if they were still in production, but the learning from them was still there. At the same time, we also realised that childhood obesity and general population health was deteriorating during lockdowns. So we developed several project packages to specifically look at how to get people moving more, with tuning of accuracy for various needs. Especially in healthcare, education and in industrial sectors, we looked at specific forms of systems inputs and patterns of human movements, to develop a robust engine that could scale with future applications of use. The Touchless Computing team at UCLCS have a key aim of equitable computing for all, without requiring further redevelopment of existing and established software products in use today.

Whats next?
-We have a lot of great plans and the tech firm companies on board are excited to see what our academics and students will come up with next. Keep in touch with us, send us your touchless computing requests and especially if it can help people, we want to know and open collaborations.
Feedback form: 
https://forms.office.com/r/P3a3dsH8Gx or by email on our touchlesscomputing.org website.