Body Detection

Work has been done in the field of gesture classification on RGB videos ​(Schneider et al., 2019)​. We build upon this work by refining their method.

The first step is to determine the body in the video to isolate the body landmarks. To do this, we use the MediaPipe library, which can take a video feed and return the coordinates of 33 landmarks. It offers two variations of these coordinates:

1. (x, y, z): Normalized to the width and height of the camera dimensions.
2. (x, y, z): Real-world 3D coordinates in meters with the origin at the centre between hips.

Although MediaPipe offers a prediction of depth, we did not find it accurate enough to be dependent on and hence we restrict ourselves to the 2D RGB video.

However, we opt to take the real-world data. Although depth may be inaccurate, the x and y axes are consistent within the same scale. We later perform our own normalization pass, so we prefer to keep the raw data.

An optimisation we make is to ignore the landmarks detected on the face, as these are not significant in the process of a full-body gesture.

Signal Filtering

As suggested by the works of Schneider et al., we apply a moving median filter with a window size of 3 to filter out peaks in the landmark graphs caused by noise or loss in tracking. The code can be seen below:

                def median_filter(tracking_points: np.ndarray): 
                    """Smooth the tracking points using the mean of each point and its neighbors""" 
                    smoothed_points = np.zeros(tracking_points.shape) 
                    smoothed_points[1:-1] = (tracking_points[:-2] + tracking_points[1:-1] + \ 
                    tracking_points[2:]) / 3 
                    smoothed_points[0] = (tracking_points[0] + tracking_points[1]) / 2 
                    smoothed_points[-1] = (tracking_points[-1] + tracking_points[-2]) / 2 
                
                    return smoothed_points 
              

This filter is only applied during the signal filtering step and the median-filtered points are not used later, to preserve peaks in amplitude that may be important for a gesture. We then find the signals with a variance above a certain threshold (we used 0.02 - a value we found experimentally to be suitable) to restrict our tracking to only the relevant landmarks that compose a gesture.

For example, the “punch” gesture has a signal that originally looks like this:

Once we apply feature scaling, it looks like this:

Signal filtering reduces it to two landmarks – the wrist and shoulder:

Reduction

For the final step, we aim to simplify the coordinates to reduce the array sizes needed to store the gesture in the final JSON file. A Savitzky-Golay filter is applied, where moving sections of the graphs are approximated to polynomial equations, giving an accurate representation of the overall signal whilst smoothing and reducing the points.

                def savgol_filter_points(points: np.ndarray, window_length: int, polyorder: int): 
                    """Apply a Savitzky-Golay filter to the given points""" 
                    x_values, y_values = points.T 
                    x_values = savgol_filter(x_values, window_length, polyorder, mode='nearest') 
                    y_values = savgol_filter(y_values, window_length, polyorder, mode='nearest') 
                
                    return np.column_stack((x_values, y_values)) 
              

To reduce the final array sizes, we then use the Ramer-Douglas-Peucker algorithm to reduce the coordinates as much as possible without losing too much data. The algorithm recursively divides the line, finding the midpoints that have the greatest distance from any point, whilst being bound by the end points. If this value is within epsilon, the point is kept.

                def simplify_gesture(points: np.ndarray, tolerance: float, start=None, end=None): 
                    """Simplify the given set of points using the Ramer-Douglas-Peucker algorithm""" 
                    if start is None: 
                        start = 0 
                    if end is None: 
                        end = len(points) - 1 
                
                    if end <= start + 1: 
                        return points[start:end] 
                
                    d_max = 0 
                    index = start 
                    for i in range(start + 1, end): 
                        denom = np.linalg.norm(points[end] - points[start]) 
                        if denom == 0: 
                            d = 0 
                        else: 
                            d = (np.abs(np.cross(points[end] - points[start], points[i] - \
                            points[start])) / denom) 
                
                        if d > d_max: 
                            index = i 
                            d_max = d 
                
                    if d_max > tolerance: 
                        res1 = simplify_gesture(points, tolerance, start, index) 
                        res2 = simplify_gesture(points, tolerance, index, end) 
                        return np.concatenate((res1[:-1], res2)) 
                    else: 
                        return np.array([points[start], points[end]]) 
              

This not only serves as a compression step, but also means that subsequent detections of the gesture can be a bit more lenient, as exact curves do not need to be followed. This decision was made to ensure that, like previously mentioned, the gestures can be triggered by a range of people making similar, but not necessarily exact, replications of the movements.

The final JSON will look something like this:

                { 
                  "16": [ 
                    [ 
                      -0.23593216121935873, 
                      0.4114285525200301 
                    ], 
                    [ 
                      -0.29122764690827413, 
                      0.39050805933070437 
                    ], 
                    [ 
                      -0.1749886776941411, 
                      0.16612659402369012 
                    ], 
                    [ 
                      0.13273842376899053, 
                      -0.23562188590099653 
                    ], 
                    [ 
                      0.19699474576552184, 
                      -0.27565540981220144 
                    ], 
                    [ 
                      0.20972398885153526, 
                      -0.25244847361058836 
                    ], 
                    [ 
                      0.10685148591740964, 
                      -0.12154869941773049 
                    ] 
                  ], 
                  "14": [ 
                    [ 
                      -0.11034779593211581, 
                      0.40496086124592134 
                    ], 
                    [ 
                      -0.21215575408096204, 
                      0.4283078708572851 
                    ], 
                    [ 
                      -0.15002866406749532, 
                      0.11082938179054341 
                    ], 
                    [ 
                      0.12209288057901993, 
                      -0.313689028784506 
                    ], 
                    [ 
                      0.17850962975715, 
                      -0.33288743103839985 
                    ], 
                    [ 
                      0.17895087203347146, 
                      -0.2715065003010426 
                    ], 
                    [ 
                      0.09376114198806573, 
                      -0.1286408402784482 
                    ] 
                  ] 
                } 
              

This reduced array size also means reduced runtime complexity, allowing us to handle real-time gesture detection with minimal latency.

Real-Time Detection

To ensure that gestures can be detected without requiring too large a buffer, we empirically found that a buffer size of around 25-30 frames is sufficient. During runtime, the buffer is filled with the most recent frame fetched from the camera and the last {n} frames are processed using all the above steps.

An optimization step we take is to use FastDTW ​(Salvador & Chan, 2007)​, which approximates DTW in O(n) time complexity. The game profile determines which gestures are loaded for the game currently being played, and the distance between each of these gestures and the buffer motion is calculated. If it is below a certain threshold, it is added to the list of possible gesture options. The one with the lowest distance is returned and the buffer is flushed to allow for another gesture to be detected.

              def detect_gesture(self): 
                  processed = process_landmarks(self.point_history, include_landmarks=FOCUS_POINTS) 
                  best_score = best_gesture = None 
              
                  for gesture in self.gestures: 
                      distances = [] 
                      num_points = 0 
                      not_visible = False 
                      for landmark_id, points in gesture.ratios.items(): 
                          count = 0 
                          for coord in self.point_history[landmark_id]: 
                              if coord[0] == 0 and coord[1] == 0: 
                                  count += 1 
                                  if count > BUFFER_SIZE // 2: 
                                      not_visible = True 
                                      break 
                          else: 
                              num_points += len(points) 
                              distance, _ = fastdtw(processed[landmark_id], points, dist=euclidean) 
                              distances.append(distance) 
              
                      if not distances or not_visible: 
                          continue 
              
                      mean = sum(distances) / len(distances) 
                      threshold = 0.8 + 0.02 * num_points 
                      if mean < threshold: 
                          if best_score is None or mean < best_score: 
                              best_score = mean 
                              best_gesture = gesture 
              
                  if best_gesture: 
                      best_gesture.activate() 
                      self.clear_history()