Since the advent of hand tracking for the Oculus Quest researchers of artificial intelligence, Facebook has made some progress. In a research project they show how well the current hand tracking.
For example, Quest Facebook has shown that a reliable hand tracking from offline points of virtual reality possible with the help of artificial intelligence. Hand tracking is a technical problem, particularly for those VR points as Quest and Quest 2: they have special cameras of depth, a comparatively small computational power, we have to cope with different lighting conditions and each user has different hands.
The first version of the hand tracking appeared in Oculus Quest in December 2019. After six months, the feature is out of beta testing (may 28, 2020). But the researchers of artificial intelligence Facebook continued to work on this technology: the current scientific publication shows how they can improve the hand tracking.
MEgATrack!
Good for hand tracking requires low latency, a small tremor (shaking) and reliable tracking, even in the case of concealment or interaction of the hands.
Facebook has not solved all the problems of the current hand tracking using a new approach MEgATrack (Monochrome egocentric tracking of an articulated hand). Twisted fingers or interaction with objects still lead to failures in tracking. But thanks to a new method of data collection and a kind of memory hand the researchers were able to further improve the quality of tracking compared to the old method.
How AI helps to track hand?
Tracking Facebook mainly occurs in four stages: first, a neural network (DetNet) identificeret the user’s hand with the four monochrome camera images and crop them using the so-called boundary boxes. The following neural network (KeyNet) then imposes the so-called “key points”, a kind of artificial skeleton that represents the fingers, fingertips, knuckles and palm with 21 point.
Hand tracking in Facebook: create a bounding box, the appointment of key points and creation of a virtual model of the hand. Hand model supports prediction of the first two systems. | Image: Facebook
On the basis of this information, then Facebook rekonstruiruet artificial hand, which is superimposed on the model key points and, therefore, on the real hand in the virtual space. Facebook then uses the last two images of virtual hands for support DetNet: network image analysis only uses images from one of the four chambers and the tracking system compares its forecast of the bounding box with the forecast of DetNet. This saves computing power. If there are imbalances, KeyNet starts to work, otherwise DetNet is executed again for all four records cameras.
Robust hand tracking in nearly any lighting conditions
Forecasting based on previous positions of the hand, is the main innovation of the new system and has three main advantages: the tracking system requires less computing power, the jitter prediction the key points is significantly reduced, and if the finger is hidden for a short time, the tracking system may nevertheless correctly predict its position.
To tracking worked under various lighting conditions, Facebook has developed a special data collection method for training AI: researchers have created a kind of exoskeleton of the camera, the subjects can attach to his back. The rod is attached to six monochrome cameras and a depth camera. After that, the subject can freely move his hands. This allows you to record from different angles, with different backgrounds and in different lighting conditions. It is this variance needed Facebook data to improve learning artificial intelligence, which then leads to more robust tracking performance.
Tracking system Facebook AI still has problems
Improved AI learning a new system, Facebook can accurately track even the smallest movements of the hands, for example, touching the palm of second hand index finger or the raising of the object is well visible by hand. According to the researchers, this is the best mobile tracking system owner on the market because of its accuracy, low energy consumption and a rate of 30 Hz.
But as soon as the individual fingers are hidden a little longer, two hands come into direct contact or close objects hand tracking on Facebook stops working. According to the researchers, the rejection of cooperation hand and hand with the subject shows the limitations of the design of the tracking and the fundamental complexity of these tasks.
The resolution must address both problems: the interaction between hand and hand and object. Both are “critical immersion” and, therefore, are an important area of research for future work.
Source
