UCL MotionInput 3 FAQ - www.touchlesscomputing.org Current version 3.2 ------------------ Can I see some videos on what exactly this is? - our videos playlist is here: https://www.youtube.com/playlist?list=PLItOc9xhnCnidFTWh95oh2fTpbjz64Kub What hardware do I need to run this? - A Windows 10/11 based PC or laptop, with a webcam! Ideally an Intel-based PC, 7th Gen and above, and 4GB RAM. An SSD is highly recommended. The more CPU cores the merrier! Running parallel ML and CV is highly compute intensive, even with our extensive optimisations. Doing simple mouse clicks in a web browser, should be much less intensive. What platforms does this run on? - Windows 10 or 11. If you would like to help to test this with us, please contact us: https://forms.office.com/r/P3a3dsH8Gx or by email on our touchlesscomputing.org website. How do I run the software? - From Windows Store, this should download to your start menu - From the downloaded application (non-Windows Store version), double click the installer and it will be installed to your Windows Start Menu. Alternatively, download the zip file version and unzip to your PC, and run the MFC (GUI) executable file. On some computers we have found that the Microsoft Visual C++ Redistributable is not installed. If no window appears, try installing this for X64 - https://docs.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170 Follow the instructions on the download link. What are the hand gestures in the Multitouch application? - Raising your right hand, you will be able to do the following: >Index Pinch (Index Finger and Thumb)- native Windows Touchpoints Press (should appear as a round translucent circle) >Middle Pinch (Middle Finger and Thumb) - Left Mouse Click (or just say "click") >Ring Pinch (Ring Finger and Thumb) - Double Left Mouse Click (or just say "double click") >Pinky Pinch (Pinky Little Finger and Thumb) - Right Mouse Click (or just say "right click") - With both the right and left hand raised, you will be able to do the following: >Both right and left hands doing Index Pinch, moving in and out, starting with right hand - Touchpoints Drag (zooming in a browser) You can also select and drag in-air by saying "hold left" (meaning, hold the left mouse button), and "release left" (to release the left mouse button). >Double Pinch - Pinch both the index and middle finger with the thumb (with all other fingers raised) on the LEFT (non-dominant) hand - Left Mouse Press (holds left mouse button until the gesture is released) Performing the gesture with the left hand activates left mouse press, with the right/dominant hand still being responsible for mouse movement. Move your right hand whilst holding this gesture with the left hand to drag across the screen. >Ring Pinch (Ring Finger and Thumb) on the LEFT (non-dominant) hand - Right Mouse Press - holds right mouse button until the gesture is released. Move the right/dominant hand whilst holding this gesture on the left to right click drag across the screen. >Rabbit Pinch - Pinch the middle and ring finger with the thumb (with all other fingers raised) on the LEFT (non-dominant) hand - Monitor Switch - perform this gesture once to switch to the next screen/monitor (if connected). Repeatedly perform this gesture to cycle through multiple monitors. >Pinky Pinch (Pinky Little Finger and Thumb) on the LEFT (non-dominant) hand - activates 3 finger swipe mode - performing this gesture simulates pressing 3 fingers side by side on the screen. Move your RIGHT hand up/down/sideways whilst holding this gesture on the LEFT (non-dominant) hand to perform 3-finger swipe gestures (supported on Windows 11). Here are some of the actions you can perform whilst holding this gesture with your left hand: - Move your RIGHT hand upwards and then release the gesture on the LEFT (non-dominant) hand to show all open windows. - Move your RIGHT hand downwards and then release the gesture on the LEFT (non-dominant) hand to show the desktop. - Move your RIGHT hand left/right to switch to the last open app. You can find the full list of supported gestures on: https://support.microsoft.com/en-us/windows/touch-gestures-for-windows-a9d28305-4818-a5df-4e2b-e5590f850741#WindowsVersion=Windows_11 --- The dominant hand is set to right by default. By default hand gestures are set to right handed but this can be switched. In the full GUI build this is easily set as an option. However, in the Multitouch app replace the data\mode_controller.json file with mode_controller_left_(rename).json (rename it to mode_controller.json and backup that file). To manually change the dominant hand from right to left, open data\mode_controller.json and rename the current mode from "touchpoint_right_hand_speech" to "touchpoint_left_hand_speech". Save the files and restart the program. We will make this easier in an upcoming build of the Multitouch app. If you would like to manually change the current mode to classic mouse, open data/mode_controller.json and change the value next to the current_mode flag from "touchpoints_right_hand_speech" to "basic_right_hand_speech" (replace "right" with "left" if required). --- What are the hand gestures in the classic mouse mode in the Multitouch app? - Raising your right hand, you will be able to do the following: >Index Pinch (Index Finger and Thumb)- Left Mouse Press (or just say "hold left" and then "release left" when desired) >Middle Pinch (Middle Finger and Thumb) - Right Mouse Press (or just say "hold right" and then "release right" when desired) >Make a fist with your hand - Double Mouse Click (or just say "double click") >Index pinch on both left and right hand and then move hands closer/further apart - Zoom in/out on a browser >Press the index and middle finger against each other (side by side) and then tilt hand upwards/downwards to scroll up/down accordingly >Pinky Pinch (Pinky Little Finger and Thumb) - Monitor Switch - perform this gesture once to switch to the next screen/monitor (if connected). Repeatedly perform this gesture to cycle through multiple monitors. If I have multiple cameras on my device (e.g. a tablet computer with front and back) how can I get to set the right camera? From 3.11 this is set from the MFC graphical user interface. Previously in v3.03+ if you have multiple cameras connected to your PC, to switch between them, press full stop "." and then a number between 0-9 to switch to a camera corresponding to that number. If you have 2 cameras, try pressing 1 to switch to your second camera. If there is no camera available corresponding to the number you have pressed, a black screen will appear. Press full stop to bring back the camera switch mode and try again with a different number, or press 0 to go back to the default camera. How can I make it easier to see where I am pressing on Multitouch? On Windows 10: Open the Settings app, and navigate to Ease of Access and then click on "Mouse pointer" on the left panel under the Vision. Make sure "Show visual feedback when I touch the screen" is enabled and then check "Make visual feedback for touchpoints darker and larger". On Windows 11: Open the Settings app, navigate to Accessibility and then click on "Mouse pointer and touch". Make sure "Touch indicator" is enabled and then check "Make the circle darker and larger". What is Ask-KITA and what speech functions are there? -Ask-KITA is our speech engine, much like Alexa and Siri, and is built on open source VOSK engine. It is intended for teaching, office work and clinical/industrial specific speech recognition. KITA stands for Know-IT-All, and itself is a standalone product as well as embedded in our touchless computing applications. It is a combinatorial speech engine that will mix well with motion gesture technologies and gives three key levels of speech processing - commands (such as turning phrases into keyboard shortcuts and mouse events), localised and offline live caption dictation without user training, delivering recognised words into text-based programs, and gesture combined exploration of spoken phrases. What interface commands can Ask-KITA currently do? - The following speech commands as phrases are integrated already. Mouse Events: click, double click, right click, hold left (for selecting a region), release left Windows-level events: minimize, maximize, maximize left, maximize right, files (file explorer), start menu, windows key, windows run, screenshot, volume up, volume down Keyboard shortcuts: up, down, left, right (these are the arrow keys), page up, page down, help, switch (alt+tab), save, print, cut, copy, paste, undo, space PowerPoint specific keywords: show full screen, next, previous, escape. How do I turn on live dictation (transcribe mode) in any text editor/browser/application/game? - Say "Transcribe" and it should appear in the camera view as a message. Say "Stop Transcribe" to turn it off. It takes a little practice to speak in clear sentences for it to recognise what you are saying. Our intention was for teachers reading out books to young students, but it has become much more than that. Note that this live captioning and transcription feature is highly experimental, and it's accuracy will improve over time with new versions released. How can I add my own speech commands to my own favourite applications and games? - Ask-KITA can do this but requires some text editing. Contact us for more information. How can I change the language model in Ask-KITA? - This requires a bit of technical know-how but it can be done with the VOSK voice models available online. What are the facial gestures in the Facial Navigation app application? - This app will allow a user to navigate with either their nose (NoseNav) or with Eye Gaze. They can either say commands, or use combinations of facial movements to act as mouse presses or keyboard keys. you can say "hold left" to drag in a direction and hold the left mouse button. Say "release left" to let go. If you are in NoseNav mode, say "butterfly" when you move your head, to move the area of interest box to where your head is at that point. The facial gestures in non-speech modes are: Nose Eyes Smile Left click and drag (v3.01+) on/off navigation Fish Face (kiss) left click left click Eyebrow raised double click double click Mouth Open right click right click Here are some of the modes specific to Facial Navigation Nose Navigation (NoseNav) with speech commands Nose Navigation (NoseNav) with Facial Gestures Eye Tracking (EyeGaze) with speech commands Eye Tracking (EyeGaze) with Facial Gestures Whenever you enter eye gaze you will run through calibration, which has onscreen instructions. In eye gaze modes, while watching in a direction, say "go" and cursor will start moving in that direction (magnetic drift mode). Say "stop" to stop moving in that direction. Use at your own risk and make sure you have enough space and clearance when playing! -------- Errors: It didn't run, what can I do? - This Technical Preview version is experimental and we, with zero-funds on this academic project, did not have a wide network for testing beyond our students and staff. We would love for community testers to come onboard so please contact us. Meantime, you can check the following: If you have multiple cameras on your PC/laptop you can try to disconnect them. Some laptops with "hello" recognition built in have had errors. In the folder "data", you will see config.json. Try changing the value of camera_nr from 0 to 1, 2 or 3. We are working on detecting this currently. If you have an antivirus it may say warnings (or stop the app completely) from accessing the camera and microphone. The software requires those things to run, so you may have to alter settings in your antivirus and restart your machine. Keep your Windows 10 and Windows 11 installations up to date, the oldest versions of Windows 10 have reported errors. You are welcome to contact us with any other thoughts and requests on compatibility issues you have faced. Why does it say this application is unsafe when I first try to run it? - As of June 2022 we are in the process of code signing the applications. Therefore please use at your own risk, this a technical preview and try it only on machines that are safe to do so. -------- Non-technical points of interest: This is very much in the realm of sci-fi! Can I do gestures like in ? -Reach out to us and your suggestions, and lets see what we can do! Feedback form: https://forms.office.com/r/P3a3dsH8Gx or by email on our touchlesscomputing.org website. What motivated you to build this? -Covid-19 affected the world, and for a while before the vaccines, as well as the public getting sick, hospital staff were getting severely ill. To keep shared computers clean and germ free comes at a cost to various economies around the world. We saw a critical need to develop cheap/free software to help in healthcare, improve the way in which we work and so we examined many different methods for touchless computing. Along the journey, several major tech firms had made significant jumps in Machine Learning (ML) and Computer Vision, and our UCL IXN programme was well suited to getting them working together with students and academics. Some of the tech firms also had let go of past products which would have been useful if they were still in production, but the learning from them was still there. At the same time, we also realised that childhood obesity and general population health was deteriorating during lockdowns. So we developed several project packages to specifically look at how to get people moving more, with tuning of accuracy for various needs. Especially in healthcare, education and in industrial sectors, we looked at specific forms of systems inputs and patterns of human movements, to develop a robust engine that could scale with future applications of use. The Touchless Computing team at UCLCS have a key aim of equitable computing for all, without requiring further redevelopment of existing and established software products in use today. Whats next? -We have a lot of great plans and the tech firm companies on board are excited to see what our academics and students will come up with next. Keep in touch with us, send us your touchless computing requests and especially if it can help people, we want to know and open collaborations. Feedback form: https://forms.office.com/r/P3a3dsH8Gx or by email on our touchlesscomputing.org website.