Technology Review

Depending on our needs, we have reviewed the following technologies:

Machine Learning

For extending MotionInput v3 face and hand navigation modules, since our main goal was to enable navigation from an angle, we looked at a range of Python computer vision libraries, as well as a few other machine learning libraries:



• Facial navigation

  There are a number of facial recognition packages available in the field. In particular, we investigated the most popular ones: Dlib, MediaPipe, and similar models provided by OpenVINO’s Open Model Zoo.
  
  MediaPipe provides two separate solutions for facial recognition: MediaPipe Face Mesh and Face Detection. Both solutions are proven to have sub-millisecond inference speed on mobile GPUs [3], which is essential for running on light-weight, fan-less devices. However, MediaPipe Face Mesh provides a much more accurate facial recognition, as it detects 468 landmarks [4], whereas MediaPipe Face Detect only detects only 6 [5].
  
  Dlib was the initial choice made by MotionInput teams for facial recognition [6], with the reason that MediaPipe Face solutions were too slow. However this is not the case, as demonstrated by BlazeFace [3], as well as our testing cases. We asked Dlib and MediaPipe to detect face and display the nose-tip landmark on the screen, frame by frame. We discovered that both MediaPipe Face Mesh and Face Detect runs twice as fast as Dlib. Moreover, we discovered that Dlib is less accurate than both MediaPipe solutions.
  
  OpenVINO is a toolkit that aims to optimise inference speed of machine learning models on different devices, by maximising usage of hardware resources [7]. It has supported a wide range of machine learning models, and we looked at those for facial recognition, as well as for head pose calculation. Specifically, we looked at Intel’s pre-trained models for facial recognition [8]. Although the models can detect landmarks for a fairly high accuracy, the output is in 2D coordinates.
  
  




• Hand navigation

  For hand detection, we looked at MediaPipe Hand module, which was the initial choice made by MotionInput teams for hand landmark detection [6], and is one of the most widely used models for hand tracking. It is able to detect 21 landmarks in 3D coordinates. The additional z-value provides the depth of the landmarks, which is essential to determining the movement of user’s hand (i.e. is it moving away or towards the camera).
  
  




• Other Machine Learning Tool(s)

  Additionally, we looked at the sklearn library, as we wanted to explore the potential of training a user-specific navigation model during the calibration stage of MotionInput. Sklearn provides a range of simple yet powerful models [9] which might be ideal for running MotionInput on small, fan-less computers.