Welcome to the MotionInput API User Manual

Here, we provide detailed descriptions of the poses within MotionInput API, enabling you to integrate them into your custom modes.

Mode Creation

To create a new mode, begin by adding a JSON file in the data/modes directory. This file should be named after your mode and structured as follows, using the class names and parameters described in this documentation:


{
"poses": [{
    "file": "pose_name_1.py",
    "arg1": value1,
    "arg2": value2
},
{
    "file": "pose_name_2.py",
    "arg1": value1,
    "arg2": value2
},
...
]
}

Execution

To utilize your newly created mode, navigate to the data folder and modify the config.json file to select your mode. You can then launch the application using the provided executable file (.exe).

Documentation

body_head_turn

This pose performs an action when the distance between specified body points (head, hands, body) exceeds a specified threshold. These body points can be defined in mode. This pose is used in body_points mode.

Name	Data Type	Description
left	dict[str, str]	The left action, will have three elements in it: method, args, flag.
method	str	The method of the action.
args	list[str]	The arguments of the action.
flag	str	The flag of the action
right	dict[str, str]	The right action, will have three elements in it: method, args, flag.
method	str	The method of the action.
args	list[str]	The arguments of the action.
flag	str	The flag of the action.

body_mouse

This Pose is used to checks if the landmark is detected. If it is, it calculates the scale factor and moves the mouse to the landmark position. In default, the landmark is the right wrist.

Name	Data Type	Description
landmark	str	The landmark, should be one of the body part.

body_points

Name	Data Type	Description
point1	str	The name of the first body point.
point2	str	The name of the second body point.
threshold_distance	float	The threshold distance for checking the distance
interval_angle	list[int, int]	The interval angle for checking the angle
method	str	The method of the action.
args	list[str]	The arguments of the action.

brick_ball

This pose is used to represent the BrickBall pose, which is used to hold left click action once index finger is pinched.

Name	Data Type	Description
hand	Literal['left', 'right']	The side of the hand (LEFT or RIGHT).

drive

Control of a virtual gamepad in a driving simulation environment using gestures. Gestures and their respective landmarks used to activate the event can be defined in the modes. It is used in driving and rocket_league modes.

Name	Data Type	Description
left	dict[str, str]	The left action, will have two elements in it: module, landmark.
right	dict[str, str]	The right action, will have two elements in it: module, landmark.
controller	object	The mode of the action, either in GamepadController mode or KeyboardController mode.

fish_face

This pose is used to detect when a person makes a fish face.

Name	Data Type	Description

forcefield_event

This pose is used to represents a forcefield gesture based on hand depth with level boundaries, triggering different keypresses.

Name	Data Type	Description
hand	Literal['left', 'right']	The side of the hand (LEFT or RIGHT).
buttons	list[str]	A list of buttons to be triggered.
depth_buffer_size	int	The size of the depth buffer, need to be bigger than 0. The default value is 10.

gun_move

A pose representing a driving action based on head movements. If the head is tilted left while having fish face, 'a' is pressed on keyboard, on right head tilt while fish face, 'd' is pressed. The default key press is 'w'. This pose is used in head_drive mode.

Name	Data Type	Description
hand	Literal['left', 'right']	The side of the hand (LEFT or RIGHT).
multiplier	float	The multiplier to scale the cursor movement, need to be bigger than 0.

handMousePen

This pose is used to detect when a person pinches their hand and moves the mouse cursor. pen is used to draw on the screen

Name	Data Type	Description

hand_double_pinch

This pose is used to represents a double pinch gesture, which is a specific hand pose.

Name	Data Type	Description
hand	Literal['left', 'right']	The side of the hand (LEFT or RIGHT).
key_hold	bool	Whether the key is held down.
num_frames	int	The number of frames to check for.
num_repetitions	int	The number of repetitions to trigger an action.

hand_left_click

This pose is used to detect when a person pinches their left hand.

Name	Data Type	Description

hand_mouse

hand_mouse is a pose used to move the cursor using the hand. It is used desktop, mouse_options, navigation, presentation, and scroll_navigation modes.

Name	Data Type	Description

hand_pinch

This pose is used to represents a pose for pinching with the left and right hands.

Name	Data Type	Description
hand	Literal['left', 'right']	The side of the hand (LEFT or RIGHT).
key_hold	bool	Whether the key is held down.
args	list[str]	The arguments of the action.

head_drive

Name	Data Type	Description
method	str	The method used for driving, either "tilt" or "turn".
show_face	bool	Whether to show the face element during the driving action.

head_suika_controller

In the game Suika, if the head is turned left or right, then the mouse moves left/right respectively. If a fish face is made then a mouse lrft click is done.

Name	Data Type	Description
method	str	The method used to detect triggers. Can be "tilt" or "turn".
show_face	bool	Flag indicating whether to show the face element.

head_tilt

This pose is used to detect when a person tilts their head to the left or right.

Name	Data Type	Description
left_flag	str	The flag to set when the head is tilted to the left.
right_flag	str	The flag to set when the head is tilted to the right.

head_trigger

An event is triggered on detection of head gesture. This head gesture can be defined in mode (e.g., 'smiling', or 'fish_face'). This pose is used in head_trigger mode.

Name	Data Type	Description
method	str	The method used for driving, either "tilt" or "turn".
args	list[str]	The arguments of the action, such as "left" or "right".
gesture	str	The gesture to be checked, such as "tilted_left" or "tilted_right".

head_turn

This pose is used to detect when a person turn their head to the left or right.

Name	Data Type	Description
left_flag	str	The flag to set when the head is tilted to the left.
right_flag	str	The flag to set when the head is tilted to the right.

hit_trigger

Defaults at 'left wrist' but if a defined body landmark is detected then an event is triggered. This event can be defined in mode. This pose is used in rocket_league and suika_hand modes.

Name	Data Type	Description
method	str	The method of the action.
args	list[str]	The arguments of the action.
pos	tuple[int, int]	The position of the trigger.
landmark	str	The landmark, should be one of the body part.
radius	int	The radius of the trigger.
color	tuple[int, int, int]	The color of the trigger.
skin	str	The skin of the trigger, which is the path of the image file.
text	str	The text to display in the center of the trigger.
dwell	float	The default dwell time of the trigger.
animation	int	The animation of the trigger.

joystick_press

Event (usually gamepad) is triggered on hand gesture. The hand gesture to trigger, event and the side of the hand can be defined in mode. It is used in joystick mode.

Name	Data Type	Description
hand	Literal['left', 'right']	The side of the hand (left or right).
gesture	dict[str, str], optional	A dictionary mapping gesture methods to expected values. Defaults to None.

joystick_wrist

Joystick movement based on wrist movement. Moving wrist left/right enables the left/ right trigger respectively. This pose is used in joystick mode.

Name	Data Type	Description
hand	Literal['left', 'right']	The side of the hand (LEFT or RIGHT).
analog_name	str	The name of the analog input.
switch	str	The switch mode ("on" or "off").
mode	str	The mode of operation ("default" or "swap")

move_joystick

This pose is used to represents a joystick that can be controlled by a person's body.

Name	Data Type	Description
method	str	The method of the action.
args	list[str]	The arguments of the action.
keys	dict[str, str]	The keys to use for the joystick.
pos	tuple[int, int]	The position of the joystick.
landmark	str	The landmark to use for the joystick.
color	tuple[int, int, int]	The color of the joystick.
skin	str	The skin of the joystick.

mr_swipe

This pose is used to detect the swipe gesture and perform the corresponding action.

Name	Data Type	Description
hand	Literal['left', 'right']	The side of the hand (LEFT or RIGHT).
sensitivity	int	The sensitivity value, need to be bigger than 0.
threshold_min	float	The minimum threshold of the gesture, the value should be in range of 0 to 1.
threshold_max	float	The maximum threshold of the gesture, the value should be in range of 0 to 1.
horizontal	bool	Whether the swipe in horzontal or not.

nose_scroll

This pose is used to represents a pose for scrolling based on the movement of the nose.

Name	Data Type	Description
sensitivity	float	The sensitivity of the gesture, need to be bigger than 0.
scale_factor	float	Scale factor for determining the scroll amount based on

nose_tracking_event

This is the mode of the Nose Navigation. It tracks the position of the nose of the user and moves the mouse or presses keyboard keys based on its relative position of a NoseBox.

Name	Data Type	Description
nose_tracking_mode	str	The mode of the nose tracking. Can be ARROW_KEYS_HOLD to hold keyboard keys down, ARROW_KEYS_PRESS to press keyboard keys once or MOUSE to move the cursor.
reset_nose_box	bool	Whether to reset the nose box to the current nose position.
show_face	bool	Whether to show the facial overlay element on the view.

open_mouth

This pose is used to detect when a person opens their mouth.

Name	Data Type	Description

pinch_drive

If right hand index finger pinched then a keyboard 'a' is pressed, else if a left hand index finger is pinched then a 'd' is pressed. This pose is used in pinch_driving mode.

Name	Data Type	Description
hand	Literal['left', 'right']	The side of the hand (LEFT or RIGHT).
method	str	The method of the action.
args	list[str]	The arguments of the action.

raise_eyebrows

This pose is used to detect when a person raises their eyebrows.

Name	Data Type	Description

raise_hand_transcribe

This pose is used to detect when a person raises their hand and transcribe what they say.

Name	Data Type	Description
hand	str	The hand to use for the gesture.
method	str	The method of the action.
args	list[str]	The arguments of the action.

samurai_swipe_event

This pose is used to represent user's hand is in a position similar to a samurai sword swipe.Class method to create a SamuraiSwipeEvent instance from keyword arguments.

Name	Data Type	Description
dom_hand	Literal['left' or 'right']	dominant hand side.
off_hand	Literal['left' or 'right']	off hand side.
sensitivity	float	The sensitivity of the gesture, need to be bigger than 0.

smile

This pose is used to detect when a person is smiling.

Name	Data Type	Description

trigger_dial

This pose is used to create a trigger dial that can be used to select a specific action.

Name	Data Type	Description
dial_name	str	The name of the dial.
base_offset	float	The offset for dial. 0 = half-circle, 1 = full circle.
element_offset	float	The element offset, determines how much the distance affects the angle step.
landmark	str	The landmark, should be one of the body part.
anchor	str	The anchor, should be one of the body part that connect with landmark.
text	str	The text to display in the center of the dial.
elements	dict[str, dict[str]]	The dictionary that contains multiple element dictionaries that can be selected.
class	str	the class of the action
method	str	the method of the action
args	list[str]	the arguments of the action
dwell	float	The default dwell time of the dial elements.

zoom_event

This pose is used to represent zooming by holding the index pinch gesture on both hands. Zooming direction and speed according tho the change in the index finger tips distance.

Name	Data Type	Description
dom_hand	Literal['left' or 'right']	dominant hand side
off_hand	Literal['left' or 'right']	off hand side