Beyond 2 displays

Going beyond 2 displays

To make our algorithm work with more than two displays/cameras, we had to find a way to know which camera corresponds to which display. For example, the first camera can be linked to the second display and so on.

But we first had to figure out which other camera should be activated when switching between more than two cameras. This means we can no longer simply switch to the next camera/display. We introduced a camera sampling to check which camera has the best angle with the hand. So when the algorithm notices that the user wants to switch to a different camera - we sample all other cameras and choose the one with the lowest angle to the hand.

To improve the app’s speed, we also added camera sampling when the hand is out of frame. This way, when the hand is out of one camera’s frame, we sample the other cameras and determine which camera captures the hand best. This allows us to speed up the process of switching between different displays as it eliminates the need to drag an object between all the monitors in-between.

In this video, we have two cameras set up next to the laptop running our project, giving a total of three to change between. By tracking the angle of the hand and sampling between cameras when it is out of frame, we were able to quickly and intuitively switch between each of the three cameras quickly and intuitively, just using hand movements. It is possible to change from the camera on the right of the setup directly to the one on the left without necessarily going through the middle and vice versa. This presents a useful feature in moving between larger numbers of cameras or displays, as current methods, such as moving a mouse, require a user to move all the way across multiple screens to do this.

By finding out the camera-display pairs by trial and error and having the camera sampling system, we can easily use and switch between 3 or more displays. We made sure to make the system easily scalable to support the number of displays limited only by the hardware. In addition to this, in the future, we can experiment with more complex setups where more than one camera can correspond to a display.

In this technical demo, we once again have a setup with three cameras, but here we also have three displays to change between. Here you can see a window being dragged from the rightmost display to the leftmost one and then to the middle, demonstrating the system’s speed in changing between the displays. Because of the held click gesture used, a user can click on a window, then turn or even walk to a different display, and move the window they’ve selected to that display simply by bringing their hand into the frame of the camera associated with it, making the system scaleable with both larger numbers of displays, but also displays that are at a distance from, or angled away from one another.