Researchers at UW–Madison and Cornell University have developed FingerTrak, a wrist-mounted camera system that captures the position of the human hand. Such a device could be used for sign language translation, virtual reality, and health diagnostics.
Past wrist-mounted cameras have been considered too bulky and obtrusive for everyday use, and most could reconstruct only a few discrete hand gestures. Conventional devices have used cameras to capture finger positions.
The FingerTrak device is a lightweight bracelet, allowing for free movement. It uses a combination of thermal imaging and machine learning to virtually reconstruct the hand. Four miniature, thermal cameras – each about the size of a pea – snap multiple silhouette images to form an outline of the hand.
A deep neural network then stitches these silhouette images together and reconstructs the virtual hand in 3D. Through this method, researchers are able to capture the entire hand pose, even when the hand is holding an object.
Zhang said the most promising application is in sign language translation.
“Current sign language translation technology requires the user to either wear a glove or have a camera in the environment, both of which are cumbersome,” he said. “This could really push the current technology into new areas.”
Li suggests that the device could also be of use for health care applications, specifically in monitoring disorders that affect fine-motor skills.
“How we move our hands and fingers often tells about our health condition,” Li said. “A device like this might be used to better understand how the elderly use their hands in daily life, helping to detect early signs of diseases like Parkinson’s and Alzheimer’s.”
In addition to Zhang and Li, the FingerTrak team includes three collaborators who were visiting undergraduate students to Cornell’s SciFi Lab last fall: first author Fang Hu of Shanghai Jiao Tong University; Peng He of Hangzhou Dianzi University; and Songlin Xu of the University of Science and Technology of China.