On the potential benefits of hand gesture controls

As I've mentioned, I'm working on hand gesture controls for icosi-do with Tilt Five glasses, which includes quite a bit of testing. To some degree, I was surprised how good these controls feel (as long as they work as intended). That's not to say that my implementation works sufficiently well already, but I hope that it will once I can access the wide-angle head tracking camera of the Tilt Five glasses.

Now I'm wondering: what makes hand gesture controls feel better than hand-held controllers? Before answering that question, it's useful to consider what is required to make hand gesture controls feel at least as good as hand-held controllers.

The Power of Controlling

Of course, the detection of specific hand gestures has to work sufficiently well. On top of that, many hand gestures will include some form of motion tracking, which has to be sufficiently accurate and fast to facilitate comfortable, interactive controls. While these requirements are not trivial, they are only the most basic ones in a longer list.

The Power of Resting

Long-term use of input devices such as keyboards, computer mice, touch pads, etc. is only comfortable if hands may rest on their palms while using these devices. Similarly, lower arms should be allowed to rest while using hand gesture controls. For Tilt Five, it might sometimes be possible to rest one's elbows on the same surface that the game board is placed on. Armrest pillows or adjustable armrests might be even more comfortable as demonstrated by the armrests of the surgeon's console of da Vinci Surgical Systems.

The Power of Touching

Another challenge for hand gesture controls is the lack of touch. Part of the problem is that touching is inherently pleasurable to most humans - even very young humans: some start thumb sucking even before birth. But there is more to it: holding a button down and keeping it in exactly that position is a lot easier than keeping a trigger or joystick in a specific position midway between its extreme positions. That's because we can hold a button down by applying a varying force as long as that force is always great enough to keep the button down; thus, there is no need for continuously correcting the applied force based on sensory feedback to keep its position, which means that the required cognitive load is smaller. One problem with mid-air hand gestures is that you are usually not able to push against other objects to reduce the required cognitive load. Thus, if a hand gesture requires users to keep their fingers in specific positions mid-air, holding that gesture can feel quite demanding and uncomfortable.

A possible design solution to this challenge might be two-fold: one part is to include touch by requiring two fingers to touch each other, e.g., as part of a OK or ring hand gesture. The other part is:

The Power of Clutching

With "clutch", I mean a way for users to temporarily stop controlling a user interface. Lifting a computer mouse up from a surface or lifting a finger up from a touch surface or muting a microphone are clutches in this sense. Often these clutches only exist in software: a microphone "muted in software" might technically still record sounds, but that doesn't matter if the software discards all recordings.

There are multiple benefits and uses of clutches. One of them is that users may relax and don't have to worry that their actions may have unintended consequences. Thus, if hovering with a mouse (without pressing a mouse button) doesn't have any consequences (other than changing the position of a mouse pointer), then mouse buttons become a clutch, too.

As mentioned above, holding your fingers or hand in a specific, controlled position may feel quite demanding and uncomfortable, thus, it is useful to offer users a clutch that allows them to temporarily exit this requirement. A good candidate for such a clutch is the touching of two fingers, i.e., as long as a user's fingers don't touch each other, no actions are triggered - similarly to hovering with a mouse.

Less is More

Assuming that all above issues have been solved: why would hand gesture controls feel better than hand-held controllers? A useful comparison might be to the relation between finger touch and stylus touch. Nowadays, finger touch is considerably more popular than stylus touch for a couple of reasons. For example:

• Finger touch does not require buying, storing, finding, handling, and/or charging another device, which improves convenience and lowers friction.

• Finger touch provides more direct interaction with a screen without the need for an intermediate device, which potentially improves immersion.

In my opinion, both points are relevant for the comparison between hand gesture controls and the Tilt Five wand: finding the wand, putting batteries into it, and turning it on does not take long, but it adds to the friction of using the system.

While pointing with the wand is fine, the long stick at its front means that the user's hands are usually more than 15 cm away from the point of interaction. Compare this with finger touch or with hand-held VR controllers, which are sometimes required to be moved directly next to or even into virtual objects to interact with them.

The Magic of Hand Gestures

While the comparison to finger touch and stylus touch is useful, in my opinion it doesn't fully explain how good hand gestures may feel. The missing element might be the magical feeling of using hand gestures to interact with objects at a distance without touching them. In fact, magicians often distract the attention of their audience with dramatic hand gestures that pretend to affect objects at a distance. Similarly, telekinetic abilities in films are often visually communicated with hand gestures, for example, the use of the "Force" in the Star Wars universe. The VR gaming industry has long recognised grabbing at a distance as a more convenient alternative to grabbing only those virtual objects that are very close to the virtual representations of hand-held controllers. One example of distance grabbing is provided by the "gravity gloves" in "Half-Life: Alyx" (which was predated by the "gravity gun" in "Half-Life 2").

Remote controls, laser pointers, and VR controllers have familiarised many people with the idea of interacting with objects at a distance. However, the possibility of interacting with distant objects by means of hand gestures without holding any technical device might cause a renewed feeling of performing magic when using hand gesture controls.

TL;DR

The feeling of magic might be an important benefit of hand gesture controls. But make no mistake: all the other issues mentioned above have to be solved first before one can hope to reap this prize. The good news is that reproducing the experience of grabbing or touching virtual objects is not necessary - and probably not even desirable.